Skip to content
Open access

Lost in Dialect: The Annotation Gap in Multilingual LLM Safety

2026 · Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026) · pp. 1-13 · 0 citations · 47 references

TL;DR

The MSAG is formalized as a framework for analyzing systematic weaknesses in multilingual safety annotation pipelines, identifying four sources of bias: language coverage gaps, dialect representation gaps, cultural semantic gaps, and annotation guideline gaps.

Abstract

Large Language Models are increasingly used as safety infrastructure for detecting harmful online content and moderating social media across multiple languages. Yet their effectiveness remains uneven across linguistic communities. This disparity reflects not only disparities in training data availability but also structural problems in annotation design. We argue that a central source of multilingual safety failure lies in the annotation gap underlying existing hate speech datasets. Most annotation guidelines and safety benchmarks are developed for English and standard language varieties, overlooking dialectal variation and culturally embedded forms of hostility. Using Arabic dialectal discourse as a case study, we show how harmful speech expressed through dialects, sarcasm, code-switching, and culturally specific expressions often remains undetected by current annotation schemes. We introduce the concept of the Multilingual Safety Annotation Gap (MSAG), identifying four sources of bias: language coverage gaps, dialect representation gaps, cultural semantic gaps, and annotation guideline gaps. We discuss implications for LLM safety alignment and outline directions for culturally grounded multilingual annotation. This paper is primarily a conceptual and methodological position paper; rather than introducing a new benchmark or empirical evaluation, we aim to formalize the MSAG as a framework for analyzing systematic weaknesses in multilingual safety annotation pipelines.

Read PDF

Similar papers

#artificial intelligence Preprint Jul 2026

Safety Alignment Illusion: The Cross-Lingual Safety Gap in LLMs

Current safety alignment training for Large Language Models (LLMs) are heavily English-centric. When such safety filters fail for non-English languages, the consequences are immediate and user-facing: voice assistants and spoken dialogue systems may produce stereotype-reinforcing outputs, bypassing the standard English...

Namya Bhatnagar · 0 citations
Preprint Aug 2026

The Illusion of Cross-Lingual Safety in Low-Resource Languages

This work investigates cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts to demonstrate superficial safety alignment.

Abigail Oppong, P SAM SAHIL, Tadesse Destaw Belay et al. · 1 citation
Preprint Aug 2026

SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs

SurakshaEval is introduced, a novel safety benchmark composed of human-written prompts spanning real-world scenarios, explicitly designed for ten major Indian languages - Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu - along with English.

Debopriyo Banerjee, K. R. Kavitha, Angana Borah et al. · 0 citations
#artificial intelligence Preprint Aug 2026

Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark

This work introduces Pak3H1, the first human-validated, culturally contextualized Urdu benchmark suite for 3H alignment, comprising PakAlpaca (helpfulness), PakBeaverTails (harmlessness), and PakTruthfulQA (honesty), and underscores the necessity of human-guided localization for equitable multilingual evaluation.

Abdullah Hashmat, Usman Naseem, Agha Ali Raza · 0 citations
Jul 2026

Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context

Inspect India Evals is an open-source framework built on top of UK AISI's Inspect AI platform that has six benchmarks: Multilingual MMLU across sixteen Indian languages, BharatBBQ, a safety evaluation for Digital Public Infrastructure, a multilingual safety test using harmful prompts in Indian languages, a multi-turn j...

Abhishek Kumar Singh, S. Nag, Sachita et al. · 0 citations
Preprint Aug 2026

Redteaming Leading Arabic LLMs with ASAS

The Arabic Safety Index (ASAS) is introduced, the first fully human-curated Arabic benchmark for redteaming LLMs and provides a culturally grounded benchmark and redteaming protocol to drive progress in Arabic LLM safety.

F. Abed, Haidar Khan, M Saiful Bari et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.