All articles
5 min readai / productivity

26% to 17%: the cost of making a model think in your language

In the trainings I run, one question lands in every room: write the prompt in French, or in English? The 2022 benchmarks gave a blunt answer, translate to English. Google reopened the best known of those benchmarks in 2026 and found most of the measured gap came from errors in the test set. The rule that survives is narrower and it points at the reasoning trace rather than at the prompt.

Three results that pointed at English.

Freda Shi and co-authors built MGSM by having 250 grade school math problems from GSM8K translated by hand into ten languages. On PaLM-540B, reasoning in the language of the question averaged 48.1%. Keeping the question in its language while forcing the chain of thought into English reached 51.3%. Translating the question into English first reached 55.0% 1. Each step toward English bought a few points.

Julen Etxaniz and colleagues removed the external translator from that comparison. Their self-translate setup asks the model to translate its own input into English, then answer. Across five tasks, self-translate beat direct inference in the original language 2. The model could answer in English what it missed in the source language, with no extra translation system involved.

Lisa Schut, Yarin Gal and Sebastian Farquhar looked inside the weights. Using a logit lens on sentences in French, German, Dutch and Mandarin, they found that semantically loaded words pass through representations close to English before being rendered in the output language. Activation steering also worked better when the steering vectors were computed in English than in the language of the input 3.

Google rechecked the test set and the gap mostly vanished.

Jan-Thorsten Peter, David Vilar and colleagues at Google went back to MGSM. Two error sources dominated the apparent language gap: questions whose human translation had shifted the meaning, and answer extraction that failed on non-English output. They fixed both, released a corrected benchmark as MGSM-Rev2, and the cross-language gap mostly disappeared for the stronger models 4.

That changes where to look first. A French prompt scoring below its English twin in your own eval is as likely to come from a translation slip in your test items, or a regex that misses an answer written as "42 euros", as from the model.

The reasoning trace still has a price.

Jirui Qi and co-authors built XReasoning, hard math and science questions in several languages, to watch which language reasoning models think in. Distilled-R1 and Skywork-OR1 drifted back to English or produced fragmented reasoning in the target language. Prompt instructions that force the trace into the user's language raised the language match rate from 46% to 98% and dropped accuracy from 26% to 17% 5. Post-training on 100 to 250 examples improved the match further and still cost accuracy.

Readable reasoning for a French reviewer cost about 9 points of accuracy on that benchmark. Pay it when a human audits the chain. Skip it when only the final answer ships.

Two costs that have nothing to do with quality.

Orevaoghene Ahia and co-authors priced 22 typologically diverse languages through OpenAI's API. Telugu needs around five times more tokens than English to carry the same content, so its speakers pay more per request and score worse on the same benchmarks 6. Aleksandar Petrov and colleagues measured differences in tokenized length of up to 15 times for the same text across languages 7. Your context budget, your latency and your per-user bill all move with the language your users write in.

Safety behavior does not travel either. Zheng-Xin Yong, Cristina Menghini and Stephen Bach translated AdvBench prompts into Zulu, Scots Gaelic and other low-resource languages. The share of attempts where GPT-4 returned actionable harmful content went from under 1% in English to 79% 8. Rates stayed far lower for high- and mid-resource languages, so a refusal tested in French tells you nothing about one tested in Zulu.

The protocol.

  1. Write the instructions in English. Leave the source material in its original language and name the output language on the last line: "Answer in French." One prompt, two languages, no translation round trip.
  2. Let the model reason in English on hard tasks. Force the trace into the user's language only when a person reads it, and budget for the loss: 26% to 17% on XReasoning 5.
  3. Before you blame the language, reread 20 items of your translated eval against the English source and run your answer parser on non-English formats. Google's two corrections erased most of the MGSM gap 4.
  4. Build a 30-item eval per language, same prompt skeleton, same grader, results in one table. Ship the English-scaffold version only if it wins by more than 2 items.
  5. Run the same paragraph through the tokenizer in each language you serve before you set a context limit or a price. Telugu took about five times the tokens of English in the OpenAI study, and across tokenizers the spread reaches 15 times 6, 7.
  6. Filter inputs and outputs yourself in every language you serve, and treat refusals as untested outside English 8.
  7. On a hard task in a poorly supported language, add a first step: have the model translate the input to English, then answer, then translate the answer back 2.

Sources.

  1. Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., Das, D., Wei, J. (2023). Language Models are Multilingual Chain-of-Thought Reasoners. ICLR 2023. arXiv:2210.03057
  2. Etxaniz, J., Azkune, G., Soroa, A., Lopez de Lacalle, O., Artetxe, M. (2024). Do Multilingual Language Models Think Better in English? NAACL 2024. aclanthology.org/2024.naacl-short.46
  3. Schut, L., Gal, Y., Farquhar, S. (2025). Do Multilingual LLMs Think In English? ICLR 2025 workshop on Building Trust in LLMs. arXiv:2502.15603
  4. Peter, J.-T., Vilar, D., Domhan, T., Malkin, D., Freitag, M. (2026). Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM). aclanthology.org/2026.gem-main.22
  5. Qi, J., Chen, S., Xiong, Z., Fernandez, R., Bitterman, D. S., Bisazza, A. (2025). When Models Reason in Your Language: Controlling Thinking Trace Language Comes at the Cost of Accuracy. Findings of EMNLP 2025. aclanthology.org/2025.findings-emnlp.1103
  6. Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., Tsvetkov, Y. (2023). Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. EMNLP 2023. aclanthology.org/2023.emnlp-main.614
  7. Petrov, A., La Malfa, E., Torr, P. H. S., Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023. proceedings.neurips.cc
  8. Yong, Z.-X., Menghini, C., Bach, S. H. (2024). Low-Resource Languages Jailbreak GPT-4. SoLaR workshop best paper, NeurIPS 2023. arXiv:2310.02446

Keep reading