Oct 2026· Transactions of the Association for Computational Linguistics· 0 citations· 21 references
Computer Science
TL;DR
This work uses an expanded corpus of 722 official UN Security Council records to measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels and proposing an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline.
Abstract
While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+-where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.
In recent years, artificial intelligence (AI) systems have become increasingly integrated into recruitment processes, particularly in early-stage candidate screening based on semantic matching between CVs and job descriptions. While embedding-based models promise efficiency and scalability, they also raise concerns reg...
Xhoana Shkajoti, Martina Ullasci, Marco Rondina et al.· Proceedings of the 2026 Inte...· 0 citations
A unified latent variable framework is proposed that jointly models pairwise and ordinal data while explicitly correcting for confounders, recovering reliable rankings from substantially fewer comparisons, and is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.
H. Katoch, D. A. Selby, Gerrit Großmann et al.· 1 citation
In Chinese content governance, large language models are increasingly used to identify toxic and discriminatory expressions, yet both task operationalization and ironic expression can alter bias measurement readings. We conducted a preregistered multi-task behavioral audit of multiple Chinese large language models. In...
Open-weight large language models are rapidly entering hiring pipelines, yet their discriminatory failure modes—and the regulatory exposure these create under the EU AI Act high-risk classification (Annex III) and U.S. EEOC adverse-impact analysis—remain poorly understood. We present the first systematic, multi-model a...
Kosuke Kitahara, Nobuhiro Yamaguchi· Proceedings of the AAAI/ACM...· 1 citation· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.