Multilingual large language models do not comprehend all natural languages to equal degrees

Abstract

Abstract Large Language Models (LLMs) play a critical role in how humans access information. While their core functionality relies on the ability to comprehend written requests, our understanding of this ability is currently constrained, because most benchmarks evaluate LLMs in high-resource languages that are predominantly spoken by Western, Educated, Industrialized, Rich, and Democratic (WEIRD) communities. The default assumption is that English is the best-performing language for LLMs, while smaller, low-resource languages are linked to less natural or reliable outputs, even in multilingual, state-of-the-art models. To track fluctuations in the comprehension abilities of LLMs, we prompt 3 popular models across 12 languages, representing the Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, and Japonic language families. Our results suggest that the models exhibit remarkable accuracy across typologically diverse languages, yet they may fall behind human baselines, albeit to different degrees. Contrary to what was expected, English is not the best-performing language, as it was systematically outperformed by several Romance languages, even lower-resource ones. We frame the results by highlighting the role of factors that drive LLM performance, such as tokenization, language distance from Spanish and English, size of training data, and data origin in high- vs. low-resource languages and WEIRD vs. non-WEIRD communities.