Study: AI prompts cost up to 17x more in non-English languages

AI pricing is per-token, and tokens are not language-neutral. A study published last month by the Coalition for Language Equity in AI Research, reported by Rest of World, found that among the 20 most-used internet languages, the most expensive one consumed 1.5 to 5 times more tokens than the cheapest. Extended across 200 languages, the gap reached 17 times in one AI model.

The scripts paying most are Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki, non-Latin writing systems that tokenizers trained mostly on English text break into far more pieces. "A startup in [Bengaluru] may pay a surcharge on every prompt and completion while its competitor in Austin does not," the study said.

The hit is not only cash. The authors describe higher token counts as translating into "higher latency and reduced effective context capacity," calling the inequity "a constraint on who can participate in the AI economy." Pratyush Choudhury, co-founder of India's Activate AI, framed the quality cost through teaching: "Think about a tutor: Retaining the student's earlier mistakes and explanations is part of delivering a good lesson. If that history consumes the context window much faster in one language, the disadvantage reaches into the quality of the service."

Dheemanth Reddy, CEO of Maya Research, offered partial reassurance that falling token prices will soften the direct cost impact, "but actually fixing the tokenization gap will still require labs to deliberately design future tokenizers to handle these languages and scripts better." Choudhury's close is less generous: "Perfect equality may be unrealistic. Large, persistent disparities should still require an explanation and an improvement plan."