Rest of World: Non-English Prompts Can Cost 17x More Tokens

A new study puts the pricing gap in a sentence: 'A startup in [Bengaluru] may pay a surcharge on every prompt and completion while its competitor in Austin does not.'

Rest of World reports on findings from the Coalition for Language Equity in AI Research, which measured how many tokens leading AI models burn through for the same work in different languages. Across the top 20 internet languages, the most expensive used between 1.5 and 5 times more tokens than the cheapest. Stretched to 200 languages, that gap grew to 17 times in one model.

The pattern tracks writing systems. Asturian, which has relatively few speakers and no Common Crawl data, 'benefits from the Latin-script advantage, as well as the representation from other related Romance languages like Italian.' Languages written in scripts the tokenizer rarely saw, including Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki, 'were at the greatest disadvantage consistently across nearly all models in the analysis.'

The hit lands past the invoice, too. 'Think about a tutor: Retaining the student's earlier mistakes and explanations is part of delivering a good lesson,' Pratyush Choudhury, co-founder of Activate AI, told the outlet. 'If that history consumes the context window much faster in one language, the disadvantage reaches into the quality of the service.' His ask of frontier labs is a 'language-by-language scorecard showing what it costs to complete comparable tasks successfully, and commit to measurable improvements for its worst-served languages.'

Not everyone treats the gap as sticky. Dheemanth Reddy, CEO of Maya Research, said 'token prices are falling very quickly, so the direct cost impact should become less severe over time,' though he added that fixing tokenization itself 'will still require labs to deliberately design future tokenizers to handle these languages and scripts better.'

Rest of World does not say which specific models the Coalition tested.