Study Finds Lesser-Used Languages Face Higher AI Costs

A study by an AI language fairness coalition found that AI models use very different amounts of tokens for different languages. People who speak languages with less web content often pay more. Researchers call this a hidden "tokenizer tax".

Among the 20 most-used languages online, token use differs by 1.5 to 5 times. In one model, the gap reached 17 times. More tokens mean slower AI replies and less context, hitting areas with limited computing resources the hardest.

Pratyush Chowdhury, an Indian investor, urged major AI labs to publish language test results and treat language efficiency as a basic requirement. Better data is one fix, like the RESPIN project with recordings in 38 dialects.

Maya Research CEO Dimant Reddy said falling token prices will ease the cost gap, but labs still need to improve next-generation tokenizers to truly close it.