Machine translation of Chinese classical poetry: a comparison among ChatGPT, Google Translate, and DeepL Translator (2024)

Machine translation, which involves the automatic translation of text from one language to another, has become increasingly important in the modern world. Rapid globalization and the need for effective communication across language barriers have made machine translation an essential tool for individuals, businesses, and governments alike (Hutchins, 2007). Machine translation has been applied in various domains, including international trade, diplomacy, education, and research, facilitating the exchange of information and ideas among people from different linguistic backgrounds (Koehn, 2009). The quest for machines capable of translating between languages has been a longstanding goal. In recent decades, advancements in technology, especially large language models (LLMs), have propelled machine translation forward; yet, challenges remain, particularly in the translation of literary works such as poetry. In the current paper, we compare the machine translation of Chinese classical poetry by a dominant LLM, ChatGPT, in comparison to traditional machine translators Google Translate and DeepL Translator.

Machine translation of literary texts

Literary translation typically demands high standards, often considered the domain of human expertise, with human translations serving as benchmarks in the machine translation process. In recent years, there has been a growing interest in the application of artificial intelligence (AI) in translation tasks. AI-based machine translation systems have shown remarkable improvements in translation quality, due to the advancements in deep learning and neural networks (Wang et al., 2022; Wu et al., 2016). Neural machine translation models, which learn from vast amounts of parallel corpora, have become the dominant approach in machine translation (Bahdanau et al., 2014). These models can capture the semantic and syntactic features of languages, enabling them to generate more fluent and accurate translations compared to traditional rule-based or statistical machine translation systems (Cho et al., 2014).

One of the most prominent AI-based language models that has emerged in recent times is ChatGPT, developed by OpenAI. ChatGPT is a large-scale language model based on the generative pre-trained transformer (GPT) architecture (Radford et al., 2019). It has been trained on an enormous corpus of text data, allowing it to generate human-like responses to a wide range of prompts. ChatGPT’s ability to understand and generate natural language has attracted significant attention from researchers and industry practitioners alike (Brown et al., 2020). ChatGPT’s emergence as a powerful language model has opened up new possibilities for machine translation. Unlike conventional machine translation systems that are designed specifically for translation tasks, ChatGPT’s general language understanding capabilities make it potentially suitable for various language-related applications, including translation (Lee, 2023). Its ability to comprehend context, handle ambiguity, and generate coherent and fluent text has raised expectations for its performance in machine translation (Sutskever et al., 2014).

However, despite the growing interest in ChatGPT’s potential for machine translation, there has been limited research on its application in literary translation, particularly in the context of Chinese classical poetry. Chinese classical poetry, with its unique linguistic features, artistic expressions, and cultural nuances, poses significant challenges for human translators (Xia and Jing, 2018) and machine translation systems (He et al., 2012). The effectiveness of ChatGPT in translating Chinese classical poetry and its comparison with other machine translation systems remain largely unexplored.

Historically, scholars have been skeptical about the applicability of computer translation to literary or culturally rich texts (Hutchins and Somers, 1992). Machine-translated literary texts have frequently faced criticism from critics and translators for their quality or reliability (Chan, 2018; Robinson, 2019). Drawing upon corpus statistics, Weng and Wang (2020) have highlighted the need for machine translation to improve on capturing verb richness, syntactic diversity, complexity, and the diversity of linking patterns within discourse, in order to match the vividness and authenticity of human translation. Hadley et al. (2022), in an analysis of the translation of Jackie Polzin’s novel Brood by Google Translate and human translators, further brought attention to the intrinsic limitations of machine translation, particularly in handling ambiguity and the reliance on contextual or prior knowledge.

Machine translation of Chinese classical poetry

Despite significant advancements in machine translation, the potential of ChatGPT in the domain of literary translation remains underexplored, especially in the translation of Chinese classical poetry. Chinese classical poetry, characterized by its concise and ambiguous language, rich allusions, and deep cultural references (Cai, 2008; Cheng, 2017), poses a formidable challenge to machine translation systems. These systems often struggle to capture the subtle nuances and artistic expressions inherent in the original texts. While there have been attempts to use recent LLMs like ChatGPT for translating Chinese classical poems into modern Chinese (Jin et al., 2021), translating them into English has not yet been attempted in research. In addition, machine translation has frequently been regarded as inferior to human translation, particularly in the realm of literary texts where human translators are considered the gold standard (Chan, 2018; Robinson, 2019). Instead of drawing direct comparisons between human and machine translation capabilities, our focus is to evaluate the performance of ChatGPT against traditional translators in translating Chinese classical poetry. This approach allows us to explore how the advent of LLMs like ChatGPT can introduce new possibilities for machine translation in literary contexts.

ChatGPT, with its inherent attributes (Bang et al., 2023; Belouadi and Eger, 2022; Brown et al., 2020; Schulman et al., 2022), exhibits unique promise within the domain of literary translation. One of the most significant advantages is ChatGPT’s capacity to use human-like language (Cai et al., 2023), which allows it to engage in conversational interactions, allowing it to respond to follow-up questions, acknowledge errors, challenge incorrect assumptions, and decline unsuitable requests (Schulman et al., 2022). ChatGPT also undergoes fine-tuning through reinforcement learning with a human feedback approach, an endeavor designed to align ChatGPT more closely with human preferences (Christiano et al., 2017). LLMs and ChatGPT chain-of-thought prompting can effectively elicit the model’s ability to conduct complex tasks, especially those involving reasoning, and can significantly improve performance (Kojima et al., 2022). Moreover, ChatGPT’s performance stands to benefit from in-context learning, involving the provision of labeled examples (prompts) within the input context (Brown et al., 2020; Moslem et al., 2023). It has demonstrated proficiency in generating creative content, including poetry and stories (Belouadi and Eger, 2022), thereby extending the potential for literary translation to encompass creative interpretation of meaning. Finally, for LLMs such as ChatGPT, prompts play a crucial role in eliciting the desired output, as they guide language modeling and stimulate model capabilities. Effective, prompt engineering is, therefore, essential to fully harness the potential of these models (Bang et al., 2023). In light of this, our research also investigates the effect of prompts on translation performance.

Given ChatGPT’s impressive language understanding and generation capabilities (Radford et al., 2019) and in particular its capacity for machine translation (Hendy et al., 2023; Jiao et al., 2023; Bawden and Yvon, 2023), it is essential to investigate its potential in translating Chinese classical poetry and compare its performance with other widely used machine translation systems, such as Google Translate and DeepL Translator. To the best of our knowledge, no study has systematically compared ChatGPT with other machine translation systems in the context of Chinese classical poetry translation. Such a comparative analysis is crucial for several reasons. First, it can provide a comprehensive understanding of ChatGPT’s capabilities and limitations in handling the complexities of literary translation. Second, it can shed light on the relative performance of different machine translation systems, enabling researchers and translators to make informed decisions when choosing tools for their translation tasks. Finally, it can identify areas for improvement and guide the development of more advanced and specialized machine translation models for literary translation.

Traditional translators such as Google Translate and DeepL Translator are widely adopted commercial translation systems that predominantly rely on the conventional neural machine translation paradigm. These commercial translation systems are underpinned by the transformer architecture from various aspects, with encoder–decoder structures wherein the encoder encodes the source sentence, and the decoder generates the target sentence based on preceding outputs (Hendy et al., 2023; Vaswani et al., 2017). The training of neural models relies heavily on meticulously curated parallel data. In contrast, GPT models assume a decoder-only architecture, simultaneously processing context and source information to produce the next output (Radford et al., 2018). These distinctive characteristics yield varied advantages and performance characteristics between the two translation paradigms. Neural machine translation systems exhibit sensitivity to the quality of typically noisy parallel data, while GPT models can integrate contextual cues during information processing. Hendy et al. (2023) compared the translation quality of GPT models in comparison with commercial systems, and they observed that GPT models consistently outperformed Microsoft Translator in terms of fluency. However, this fluency advantage was juxtaposed with an increased usage of punctuation marks and unaligned source words, which may diverge from a faithful representation of the original input. Despite the inherent challenges and limitations associated with GPT models, they exhibit good translation capabilities. Notably, in comparative evaluations, Jiao et al. (2023) conducted evaluations on various benchmark test sets among ChatGPT, Google Translate, and DeepL Translator. They underscored ChatGPT’s competitive performance in high-resource European languages but identified its limitations in translating low-resource or distant languages.

The present study

Therefore, in the present study, we compared ChatGPT with Google Translate and DeepL Translator in the translation of Chinese classical poetry; we additionally examined whether prompts contribute to ChatGPT’s translation performance. To do this, we evaluated these translations from various perspectives, including fidelity, fluency, language style, machine translation style, and overall performance. In evaluating machine translation, fidelity, and fluency are well-established metrics (Ali, 2020; Doherty, 2016; White, 1995). Fidelity is typically defined as the degree to which the translation accurately conveys the meaning and intent of the original text, without omitting or distorting important information. Fluency is generally characterized by the extent to which the translation follows the rules and norms of the target language regardless of the source text (Moorkens et al., 2018), focusing on the smoothness, coherence, and naturalness of the translated text in the target language. In the literary realm, language style is a crucial metric. As suggested by Hutchins and Somers (1992), translators are equally invested in the ‘stylistic’ quality as they are in accuracy and intelligibility. Language style pertains to the form of language used, such as the degree of formality, and is distinct from fidelity, which concerns how faithfully the message is conveyed (e.g., a Chinese classical poem could be translated into either literary or plain English, with both versions maintaining the same underlying meaning but differing in stylistic expression). Therefore, we incorporate language style in the evaluation framework to measure the extent to which a translation employs appropriate language to effectively convey the original message, a particularly important attribute in the translation of literary texts. This dimension evaluates the appropriateness and adherence to the language style of the original text, including mood, tone, rhythm, poetic language and format, and cultural nuances. Another critical concern with machine translation is its tendency towards roboticness, leading to a lack of naturalness and fluidity in language. Thus, in our research, we also include machine translation style as a criterion to evaluate the degree of naturalness in literary translation. This metric is particularly relevant given the advancements in machine translation systems that aim to emulate human-like language (Lambert et al., 2006). Within this framework, a higher machine translation style score indicates a translation that appears more machine-like and less human-like and, thus less desirable in the context of literary translation. The current study thus addresses the following questions:

1. (1)

   Does ChatGPT represent an advancement over Google Translate and DeepL Translator in translating Chinese classical poetry? If so, in what aspects?

2. (2)

   In comparison to Google Translate and DeepL Translator, what advantages and limitations does ChatGPT exhibit in the field of poetry translation?

3. (3)

   Can varying prompts significantly enhance ChatGPT’s performance in poetry translation? Is it feasible to mitigate machine translation style through the use of prompts?

Ethical statement

The research was ethically approved by an institutional ethical board. Participants gave their written consent before taking part in the survey.

Materials

The poems used in the translation tasks were selected from the China Poetry Network (https://www.zgshige.com) and sponsored by the authoritative Chinese poetry publication “Poetry Magazine” (see the Appendix for the poems and translations). The website collects poems from writers and publishes a weekly poetry selection based on expert reviews and popular voting. To ensure that the tested materials have not been trained for ChatGPT in advance and to ensure the randomness of material selection, we used poems from the “Poetry and Verse Weekly Selection” in issues 14 and 16, released on April 10, 2023, and April 24, 2023, respectively. In total, 21 poems were included, consisting of 12 quatrains (绝句; a poetic form consisting of four lines, with rhymes on the second and fourth line) and 9 Lǜshī (律诗; a poetic form consisting of eight lines, with rhymes on the second, fourth, sixth, and eighth line), which are the two most popular types in Chinese classical poetry. Examples of quatrain and Lǜshī are displayed in Table 1.

Translation systems and prompts

We compared ChatGPT with two popular commercial translation products: Google Translate and DeepL Translator (the translations are available in the Appendix).

Google Translate (https://translate.google.com/) is a web-based machine translation system developed by Google; it is known for its extensive language support, encompassing over 130 languages, making it one of the largest machine translation providers. DeepL Translator (https://www.deepl.com/translator), launched in August 2017, has garnered popularity as a freely accessible translation service that supports 31 languages. It has emerged as one of the most widely used translation systems, serving a user base of over half a billion people. ChatGPT (https://chat.openai.com/), released in 2022 by OpenAI, is a publicly accessible advanced conversational AI model designed to generate coherent and context-aware responses. For our study, we use GPT-3.5 for conducting the translations.

Wtih ChatGPT, we further compared the following two prompts:

Prompt 1: Please provide the English translation for the following material.

Prompt 2: The following are Chinese classical poems, please interpret their meaning first and translate them into English poems with rhymes.

Prompt 1, used by Jiao et al. (2023), consistently achieved the highest translation performance score among the three tested prompts. Prompt 2 was devised with insights drawn from the concepts of task-specific and domain-specific prompting proposed by Peng et al. (2023). It encompasses domain information (Chinese poem to English poem), task information (translation), and specific requirements (interpret the meaning first; rhyme). Additionally, to enhance ChatGPT’s comprehension of Chinese classical poetry while preserving fidelity, we applied an approach inspired by in-context learning and the chain of thought. As a result, Prompt 2 required ChatGPT to carry out interpretation prior to commencing the translation process.

Translation quality evaluation

All translations were evaluated by nine native Chinese speakers, each holding a master’s or PhD degree in English translation from universities in China. They rated the translations on five defined aspects using a 5-point Likert scale, as shown in Table 2. The evaluation criteria employed in this study encompassed fidelity, fluency, language style, machine translation style, and overall performance. The definitions of these criteria were provided in the previous section. The overall score reflects a comprehensive evaluation based on the listed criteria and the experts’ judgments.

The evaluation of translation quality was conducted using Qualtrics, an online survey platform (https://cuhk.qualtrics.com). Participants provided their consent to participate in the survey and were then given detailed instructions on five rating criteria. They were tasked with assessing four different translations of 21 Chinese poems, evaluating fidelity, fluency, language style, machine translation style, and overall performance (refer to Table 2). Each poem was displayed on a single page alongside a translation from Google Translate, DeepL Translator, or ChatGPT (Prompt 1 or Prompt 2). Raters were required to score each criterion from 1 to 5 for each translation. The survey displayed the four translations of each poem consecutively without disclosing the sources to the raters. The sequence of translations and the order of the 21 poems were randomized for each rater. Upon completion of the scoring process, participants were compensated with 80 yuan for their participation.

Comparisons among DeepL Translator, Google Translate, and ChatGPT

The experimental data and analytical scripts are publicly available at Open Science Framework (https://osf.io/34cgd/). We compared the scores of DeepL Translator, Google Translate, and ChatGPT (with Prompt 1) on the five criteria mentioned above. Since DeepL Translator and Google Translate cannot be provided with instructions as detailed as those in ChatGPT’s Prompt 2, we only compare the translation performance of these two systems with ChatGPT using Prompt 1. This approach ensures that the comparison among the three translation tools is fair and valid. Differences in performance among the three machine translation systems were examined by a series of linear mixed-effects models.

To determine whether there are differences in the translation performance among Google Translate, DeepL Translator, and ChatGPT, we analyzed the data using linear mixed-effects regressions in the lme4 package (Bates et al. 2015) in R (R Core Team, 2014), with translator (Google Translate/ DeepL Translator/ ChatGPT) as the fixed effect, while raters and poems were treated as random effects. We conducted an analysis of each of the five criteria as a dependent variable. Significance was calculated using the lmerTest package (Kuznetsova et al., 2017). We used a forward algorithm to determine the maximal random effect structure based on the data, using the alpha level of 0.2 instead of 0.05 (Matuschek et al., 2017). For the analysis, the “Translator” variable was coded using sum contrasts, with ChatGPT as the reference level.

For fidelity (see Fig. 1), ChatGPT had a higher score than DeepL Translator (3.83 vs. 2.97, β = 0.85, SE = 0.25, t = 3.38, df = 10.27, p = 0.017) and Google Translate (3.83 vs. 2.75, β = 1.07, SE = 0.22, t = 4.92, df = 9.24, p = 0.002). There was no significant difference between Google Translate and DeepL Translator (2.75 vs. 2.97, β = 0.22, SE = 0.13, t = 1.78, df = 14.22, p = 0.21).

For fluency, ChatGPT had a higher score than DeepL Translator (3.82 vs. 2.47, β = 1.35, SE = 0.14, t = 9.69, df = 8, p < 0.001) and Google Translate (3.82 vs. 2.44, β = 1.38, SE = 0.12, t = 11.30, df = 8, p < 0.001). There was no significant difference between Google Translate and DeepL Translator (2.44 vs. 2.47, β = 0.03, SE = 0.09, t = 0.35, df = 8, p = 0.93).

For language style, ChatGPT had a higher score DeepL Translator (3.36 vs. 2.06, β = 1.30, SE = 0.13, t = 10.02, df = 8, p < 0.001) and Google Translate (3.36 vs. 2.05, β = 1.31, SE = 0.11, t = 12.27, df = 8, p < 0.001). There was no significant difference between Google Translate and DeepL Translator (2.05 vs. 2.06, β = 0.0053, SE = 0.09, t = 0.06, df = 8, p = 0.998).

For machine translation style, ChatGPT was less robotic than DeepL Translator (1.69 vs. 3.91, β = −2.22, SE = 0.28, t = −7.93, df = 8, p < 0.001) and Google Translate (1.69 vs. 3.97, β = 2.28, SE = 0.17, t = 8.00, df = 13.23, p < 0.001). There was no significant difference between DeepL Translator and Google Translate (3.91 vs. 3.97, β = −0.058, SE = 0.21, t = −0.28, df = 8, p = 0.96).

For the overall score, ChatGPT outperformed DeepL Translator (3.61 vs. 2.29, β = 1.33, SE = 0.17, t = 7.62, df = 11.7, p < 0.001) and Google Translate (3.61 vs. 2.13, β = 1.48, SE = 0.14, t = 10.53, df = 10.3, p < 0.0001. No significant difference was found between DeepL Translator and Google Translate (2.29 vs. 2.13, β = 0.15, SE = 0.10, t = 1.52, df = 10.6, p = 0.32).

Comparisons between Prompt 1 and Prompt 2 of ChatGPT

We then compared the scores of Prompt 1 and Prompt 2 of ChatGPT in terms of the five criteria. As shown in Fig. 2, our analysis revealed no significant difference between Prompt 1 and Prompt 2 in terms of fidelity (3.83 vs. 3.58, β = −0.25, SE = 0.20, t = −1.22, df = 8, p = 0.26) or fluency (3.82 vs. 3.98, β = 0.16, SE = 0.14, t = 1.16, df = 8, p = 0.28). ChatGPT performed better translation with Prompt 2 than with Prompt 1 in terms of language style (4.30 vs. 3.36, β = 0.94, SE = 0.15, t = 6.32, df = 8, p < 0.001), machine translation style (a lower score indicating less robotic; 1.22 vs. 1.69, β = −0.47, SE = 0.057, t = −8.24, df = 348, p < 0.001), and the overall performance (4.01 vs. 3.61, β = 0.40, SE = 0.16, t = 2.56, df = 8, p = 0.034).

Our statistical analysis revealed that ChatGPT performed significantly better than DeepL Translator and Google Translate in the translation of Chinese classical poetry, and no significant disparity was observed between DeepL Translator and Google Translate in each criterion. This underscored the great potential of ChatGPT in the domain of literary translation where machine translation has historically faced skepticism (Bellos, 2012; Chan, 2018; Robinson, 2019).

ChatGPT’s exceptional proficiency in poetry translation can be attributed to its advanced capabilities in understanding nuanced human language (Bang et al., 2023) and in creative writing (Belouadi and Eger, 2022). This finding substantiated ChatGPT’s capability to generate natural languages and incorporate contextual information, as suggested by previous research (Brown et al., 2020; Moslem et al., 2023). It has been argued that traditional machine translation systems are deficient in these capabilities, hence their inability to match humans in literary translation (Hadley et al., 2022). Our results suggested that ChatGPT is able to leverage its strong ability in text understanding and creative writing to literary translation and surpass traditional machine translation models significantly in all evaluation criteria, particularly with regard to ensuring fidelity, enhancing fluency, and mitigating machine translation style.

Fidelity in poetry translation by ChatGPT

ChatGPT attained a superior fidelity score relative to other machine translators in the translation of Chinese classical poems. Further in-text analysis (detailed below) reveals that ChatGPT’s superior fidelity performance is due to its proficiency in interpreting rhetoric.

Chinese classical poetry typically employs symbolic imagery, allusions, metaphors, and metonymy to convey meaning. Unlike Google Translate and DeepL Translator, ChatGPT consistently recognizes and accurately translates commonly used imagery in Chinese classical poetry, whereas Google Translate and DeepL Translator often opt for literal transliterations. This is demonstrated in Example 1:

Example 1

Source text: 蟾辉轻抚土门楼, 似水柔情似水流。

Gloss: The moonlight gently caresses the earthen gate tower. The water-like, tender affection is like the flowing water.

DeepL Translator: The toad glow gently caresses the Tumen Tower, and the water-like tenderness flows like water.

Google Translate: Toad Hui caressed the Tumenlou lightly, like water and tenderness like water flowing.

ChatGPT (Prompt 1): The moonlight gently touches the earthen gate tower, as if caressing it. Like water, its tenderness flows.

ChatGPT (Prompt 2): The moonlight caresses the earthen gate tower with grace, soft and gentle, like flowing water’s tender embrace.

In Chinese classical poetry, “蟾辉“ (chán huī) is a term used to convey the concept of “moonlight”. This symbolism stems from a legend in ancient China that suggests the presence of a toad residing on the moon, thus “蟾“ (toad) is used metaphorically to refer to the moon, and “蟾辉“ represents the moon’s glow. While DeepL Translator rendered it as “toad glow,” and Google Translate as “Toad Hui,” ChatGPT (Prompts 1 and 2) accurately interpreted and translated it as “moonlight”.

A similar instance can be found with the term “杜宇“ (dù yǔ), which is often used to symbolize a “cuckoo” in Chinese classical poetry, stemming from the legend of a person named Dù Yǔ, who is said to transform into a cuckoo after death. DeepL Translator translated it as “doohickeys” and Google Translate directly transliterated it as “Du Yu”, whereas ChatGPT (prompts 1 and 2) adeptly grasped the intended meaning and translated it as ‘cuckoo’.

Furthermore, “芳菲“ (fāng fēi) is another commonly employed image in classical poetry, signifying fragrant and beautiful flowers and grass. In the original poem, this term is juxtaposed with ‘rain’ to depict the imagery of fragrant flowers and petals falling like rain. DeepL Translator missed these nuances in its translation, and Google Translate resorted to using the Pinyin “Fang Feiyu”; ChatGPT (Prompt 1), however, successfully captured the original meaning and translated it as “fragrant rain.” This exemplified ChatGPT’s proficiency in comprehending the contextual significance of these terms in Chinese classical poetry, thus enhancing the overall fidelity of the translation.

Fluency in poetry translation by ChatGPT

In terms of fluency, ChatGPT demonstrated superior performance compared to other machine translators, attributable to its robust reasoning and comprehension abilities, including the insertion of appropriate and consistent subjects, maintaining inter-sentence transitional relationships, and incorporating other inferred semantic components derived from the original text.

Chinese classical poetry often portrays a character or protagonist who can either be a reflection of the poet or a character created to express the poet’s emotions. In Chinese classical poetry, the character’s self-designation and the address to the character are often omitted, resulting in a vacant subject within the sentence. However, English does not permit for empty subjects (pro-drop). Therefore, when translating Chinese poetry into English, it becomes imperative to correctly identify these omitted subjects and replace them with appropriate pronouns. This is illustrated in Example 2:

Example 2

Source text: 垂钓午时倦, 贪杯老酒磋

Gloss: (The fisherman) feels sleepy while fishing at noon, (he) indulges in a cup of aged wine.

DeepL Translator: The fishing is tired at noon, and the greedy old wine is negotiated.

Google Translate: Tired of fishing at noon, greedy for old wine.

ChatGPT (Prompt 1): Feeling tired from fishing at noon, I indulge in a cup of aged wine.

ChatGPT (Prompt 2): Fatigued from fishing at noon, he takes a rest. Indulging in wine, his heart feels blessed.

The quatrain in Example 2 combines scenic depictions with lyrical expressions, seamlessly shifting the focus between the scenery and the lyrical character. ChatGPT’s exceptional comprehension and contextual integration capabilities allow it to identify and supplement subjects adeptly, enabling the expression of complete meanings with a natural and fluent style. This is achieved through the skillful manipulation of sentence structures and the addition of transitional elements. In contrast, DeepL Translator erroneously misidentified the subject as The fishing, leading to a translation that deviates significantly from the original text. Google Translate omitted the subject, leading to an incomplete translation that impedes full comprehension of the poem. Conversely, ChatGPT recognized the lyrical character as a person and the logical relationship between the two sentences by analyzing contextual meaning and semantic relationships. As a result, ChatGPT accurately supplied the correct subjects and transformed the initial sentence into a gerund phrase, thereby establishing a logical and seamless transition.

Language style and translation style in poetry translation by ChatGPT

Our results also showed a significant distinction between Prompt 1 and Prompt 2 regarding both language style and machine translation style, with Prompt 2 outperforming Prompt 1. This underscores the efficacy of harnessing prompts to enhance translation quality, consistent with previous research regarding optimizing machine translation through prompt engineering (e.g. Gao et al., 2023; Yamada, 2023). Notably, with the application of Prompt 2, ChatGPT reduced the machine translation style to a very low level, suggesting its ability to achieve more precise and customized translations through prompt engineering and human engagement. However, with respect to fidelity and fluency, there is no significant difference between the two prompts. This indicated that ChatGPT exhibits stable performance in these two dimensions, reaffirming its robust comprehension capabilities and its capacity to generate fluent and natural language.

Esthetic and rhythmic qualities of language are prominent features of poetry language (Greene et al., 2012; Masters, 1915). Wang and Li (2012) posited that language style constitutes a crucial facet of translation. The translator needs to consider how to meet the diverse stylistic demands of the original texts and their translations. In this regard, ChatGPT demonstrates a substantial advantage over traditional machine translators. Our results indicated that ChatGPT was able to translate Chinese classical poetry into English with rhythm and rhyme, effectively preserving the beauty of poetic language, especially when provided with more detailed prompt instructions.

In English poetry, various forms of rhyme, such as alliteration, assonance, and end rhyme, are employed based on the repetition of internal phonemes within words (Harmon, 1987). In Chinese poetry, the most prevalent form of rhyme is end rhyme, where rhyming characters share the same ending vowel or ending vowel plus an ending consonant. In a Chinese quatrain or Lǜshī, the rhyming scheme dictates that the ending characters of the even-numbered lines should rhyme.

By using Prompt 2, which incorporates domain information (Chinese poems to English poems) and specific requirements related to rhyme, ChatGPT can translate Chinese poetry into English using stylistics that adhere to end rhymes and maintain fixed syllable counts per line. This rhyme scheme is relatively stable, predominantly following an AABB pattern, with only a few poems using an ABAB pattern. This contrasts with the findings of Belouadi and Eger (2022), who showed that ChatGPT generated rhymes at arbitrary positions rather than adhering to prescribed rhyme schemes. The discrepancy may be due to the translation task and the consideration of the rhyming scheme of quatrain and Lǜshī, which consistently uses end rhyme and the AABB rhyming pattern.

Remaining challenges

Although ChatGPT demonstrates superior performance over traditional translators in all criteria, it is important to note that the average fidelity score fell within the range between “moderate” and “good,” even when utilizing Prompt 2. This indicates that fidelity issues still exist when ChatGPT translates Chinese classical poetry. These issues primarily revolve around the ineffective implementation of knowledge and the inclusion of fabricated content.

ChatGPT exhibits a proficient understanding of specific phrases that require expertise in ancient Chinese when queried independently. However, during actual translations, it fails to apply this knowledge effectively and often resorts to providing literal interpretations. For example, when queried separately about the meaning of this phrase, ChatGPT offered an accurate interpretation:

“In the poem, ‘郭墙‘ refers to the outer wall or fortified wall surrounding a city or a pass. It signifies a protective barrier or boundary. In the context of the poem, ‘郭墙空锁榆关在‘ conveys the image of the locked gate at the Yuguan Pass with the deserted outer wall. It symbolizes the desolation and abandonment of a once-important stronghold or border defense.”

ChatGPT correctly comprehended the meaning of ‘郭墙‘ within the poem’s context. However, in its actual translation for the poem sentence “郭墙空锁榆关在“ (The outer wall of the city is locked in vain, the Yuguan Pass still stands there), both Prompt 1 and Prompt 2 translated “郭墙” as “The Guo wall”. It failed to apply its knowledge of ‘郭墙‘, supplying a transliteration rather than a meaningful translation.

In the process of enhancing translation fluency and coherence, ChatGPT sometimes adds elements not explicitly found in the source text. These include the addition of omitted subjects, transitions between sentences, and other semantic inferences derived from the original content. However, there are instances where ChatGPT introduces entirely fabricated content, deviating significantly from the source material. Such deviations, often referred to as “hallucinations” (Ji et al., 2023), have been a major point of criticism. Despite these issues, our research findings demonstrate that ChatGPT generally adhered to the translation tasks as specified in our prompts. Nevertheless, we did observe occurrences of incorrect insertions and hallucinated words in the translations. This issue was particularly pronounced with Prompt 2, as illustrated in Example 3.

Example 3

Source text: 桥上几多车马客, 清波唯照此身闲

Gloss: There are so many people who ride a horse or take a carriage passing by on the bridge. The clear waves only reflect my leisurely form.

ChatGPT (Prompt 1): There are so many passers-by on the bridge, but only the clear waves reflect my leisurely form.

ChatGPT (Prompt 2): Upon the bridge, carriages pass with grace, yet clear waters reflect my leisurely pace. \

In Example 3, the original text does not refer to “with grace” and “pace” but ChatGPT added these expressions to maintain the rhyme scheme. While the inclusion of “leisurely pace” successfully preserved the protagonist’s leisurely mood and posture, “with grace” diverged from the intended meaning and the overall language style of the original poem, resulting in a deviation from its original composition. These instances underscore the importance of human post-editing despite ChatGPT’s impressive performance in translation.

Potential of ChatGPT in poetry translation

Despite the challenges discussed, ChatGPT, along with the underlying LLMs, has exhibited a robust capability and substantial potential in poetry translation compared to traditional neural machine translation models like Google Translate and DeepL Translator. The advent of these LLMs, which have been trained to understand patterns, grammar, and semantic relationships through extensive data, paves the way for machine translation to shift from a reliance on parallel corpora for token-to-token correspondence to a more human-like approach.

The integration of ChatGPT into the realm of Chinese poetry translation, as demonstrated in this study, demonstrated the immense potential of LLMs. In particular, ChatGPT exemplifies the capability of these models in comprehending and translating poetry. Furthermore, by fostering the advancement of machine translation in the domain of literary works, we open the door to the appreciation of exceptional poems that have not yet undergone manual translation. This can facilitate cultural exchange, bridging language barriers, and promoting mutual understanding and appreciation among diverse readers (e.g., Matusov, 2019). In addition, the ability of AI models like ChatGPT to translate poetry suggests a deeper grasp of creative language, which could have implications for other creative fields such as literature, music, and art (e.g., Franceschelli and Musolesi, 2023).

The increasing sophistication of AI translation models like ChatGPT could pose a threat to professional translators. However, while AI can automate certain aspects of translation in literary translation, human translators are still essential to offer expertise, cultural understanding, and nuanced interpretations that AI may lack and leverage their experience and esthetics to ensure the translation captures the essence and beauty of the original work (e.g., Benmansour and Hdouch, 2023). In the future, professional translators may need to emphasize their unique value proposition, focusing on quality, cultural sensitivity, and specialized knowledge to differentiate themselves from AI-generated translations. Rather than viewing AI as a threat, professional translators could explore opportunities for collaboration with AI models. By leveraging AI tools to enhance their productivity and efficiency, translators can focus on higher-level tasks such as editing, quality assurance, and creative adaptation, ultimately delivering more value to target readers (e.g., Vieira, 2020). They may also train the models with knowledge in a specific field or with their writing pieces to enable the model to perform better in a specific translation field or conduct translation closer to their translation style.

Despite its potential, the translation of poetry by AI also raises challenges and ethical considerations. There are concerns about the loss of cultural nuances and artistic subtleties in machine translations, as well as questions about the authenticity and originality of AI-generated creative works (Vieira and Alonso, 2020). AI translation of poetry could lead to the appropriation of cultural works without proper understanding or respect for their cultural context. This raises concerns about the ethical use of AI in translating culturally significant poetry, especially if the translations are inaccurate or insensitive. There may also be questions about the ownership and attribution of AI-generated translations. Who owns the rights to the translated work, and how should credit be given to the AI model, its developers, and users? These questions have implications for intellectual property and the recognition of creative contributions (Lacruz Mantecón, 2023). As AI continues to advance in creative tasks, it will be important to address these challenges and ensure responsible use of AI technologies in creative endeavors.

The present study compared the performances of ChatGPT with Google Translate and DeepL Translator in translating Chinese classical poetry in terms of fidelity, fluency, language style, and machine translation style. The results revealed that ChatGPT outperformed Google Translate and DeepL Translator in all evaluation criteria, and no significant difference was found between DeepL Translator and Google Translate, corroborating its powerful ability to generate natural and human-like language. However, the overall average score still did not reach the “Good Level”, indicating that there is still significant room for improvement in ChatGPT’s understanding and translation of Chinese classical poetry. Moreover, by using prompt instructions to let ChatGPT interpret meaning first and then translate with rhymes, the translation quality in terms of language style and machine translation style was significantly improved.

Further in-text examinations revealed that ChatGPT demonstrated the ability to understand and translate some common symbols and imagery in Chinese classical poetry. It can also fill in the appropriate and consistent subjects, inter-sentence transitional relationships, and other semantic components inferred from the original text, ensuring complete and smooth translations and effectively improving fluency while mitigating machine translation style. Although ChatGPT still faces challenges in the accurate interpretation and translation of certain phrases and sentences, our findings showed a great potential of ChatGPT in literary translation and creative writing, shedding light on the evolving role of technology in translation, creativity, and cultural exchange. Future avenues of research are needed to fully leverage ChatGPT’s potential in translating Chinese classical poetry by refining and fine-tuning the model through continued training, incorporating specialized domain knowledge and corpora, such as ancient literary works and cultural annotations, as well as investigating human–AI collaboration in literary translation.

All the experimental stimuli and data are publicly available at Open Science Framework (https://osf.io/34cgd/?view_only=565a2575ebaf47e4990a4fd190a328cc).

Author notes

1. These authors contributed equally: Ruiyao Gao, Yumeng Lin.

Authors and Affiliations

1. Department of Linguistics and Modern Languages, The Chinese University of Hong KongKong, Shatin, Hong Kong SAR

   Ruiyao Gao,Yumeng Lin&Zhenguang G. Cai

2. Department of Translation, Interpreting and Intercultural Studies, Hong Kong Baptist University, Kowloon, Hong Kong SAR

   Nan Zhao

3. Brain and Mind Institute, The Chinese University of Hong Kong, Shatin, Hong Kong SAR

   Zhenguang G. Cai

Contributions

R. Gao: Conceptualization; data curation; formal analysis; investigation; methodology; writing— original draft; writing—review & editing. Y. Lin: Data curation; formal analysis; visualization; writing—original draft; writing—review & editing. N. Zhao: Writing—review & editing. Z.G. Cai: Conceptualization; methodology; supervision; writing—review & editing.

Corresponding author

Correspondence to Zhenguang G. Cai.

Competing interests

The authors declare no competing interests.

Ethical approval

Ethical approval was obtained from The Chinese University of Hong Kong (SBRE-21-0445). It was confirmed that the research complied with ethical standards and was performed in accordance with relevant guidelines/regulations.

Informed consent

Informed consent was gained from all participants. The participants were all adults, and their participation was entirely voluntary. The personal data from all participants has been anonymised.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions