Skip to content
Open access

Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines

Oct 2026 · Transactions of the Association for Computational Linguistics · 0 citations · 21 references
Computer Science

TL;DR

This work uses an expanded corpus of 722 official UN Security Council records to measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels and proposing an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline.

Abstract

While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+-where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.

Read PDF

Similar papers

Book Open access Sep 2026

Auditing Bias in AI-Based Hiring Systems: A Fairness Analysis of Nationality and Gender Discrimination

In recent years, artificial intelligence (AI) systems have become increasingly integrated into recruitment processes, particularly in early-stage candidate screening based on semantic matching between CVs and job descriptions. While embedding-based models promise efficiency and scalability, they also raise concerns reg...

Xhoana Shkajoti, Martina Ullasci, Marco Rondina et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Accounting for Bias Enables Sustainable LLM Evaluation

A unified latent variable framework is proposed that jointly models pairwise and ordinal data while explicitly correcting for confounders, recovering reliable rankings from substantially fewer comparisons, and is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.

H. Katoch, D. A. Selby, Gerrit Großmann et al. · 1 citation
#natural language process... Conference Open access Oct 2026

The “10th Juror”: Open-Set Standpoint Screening for Bureaucratic Bias Detection

MARS-Gov, a standpoint-aware multi-agent framework that combines legal retrieval, open-set target screening, specialized jurors, conservative routing, and rewrite verification is proposed, a standpoint-aware multi-agent framework that combines legal retrieval, open-set target screening, specialized jurors, conservative...

Yu-Chen Miao, Zi-Jun Wang, Chang Han et al. · 0 citations
#natural language process... Preprint Oct 2026

Evaluating Bias in Chinese Large Language Models across Tasks and Registers: Task Operationalization Effects and Irony Classification Risks

In Chinese content governance, large language models are increasingly used to identify toxic and discriminatory expressions, yet both task operationalization and ironic expression can alter bias measurement readings. We conducted a preregistered multi-task behavioral audit of multiple Chinese large language models. In...

Mo Zhou · 0 citations
#artificial intelligence Conference Open access Sep 2026

Linguistic Triggers of Gender and Racial Bias in Open-Weight LLMs Applied to Recruitment

Open-weight large language models are rapidly entering hiring pipelines, yet their discriminatory failure modes—and the regulatory exposure these create under the EU AI Act high-risk classification (Annex III) and U.S. EEOC adverse-impact analysis—remain poorly understood. We present the first systematic, multi-model a...

Kosuke Kitahara, Nobuhiro Yamaguchi · 1 citation · ⚡1

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.