Shallow Lexical, Deep Source: A Hierarchical Examination of LLM Bias in News Evaluation

Abstract

Large language models (LLMs) increasingly mediate how people read the news, but how deeply their evaluations are shaped by textual cues remains unclear. This study manipulated news coverage of the Columbia Gaza campus protests at two levels: a shallow, lexical level (adjective framing: left, right, neutral, original) and a deeper, source level (labels: none, The Guardian, Fox News), each crossed with defensive prompts (none, ignore adjectives, ignore source). The full design (92 articles × 4 versions × 3 defenses × 3 sources × 2 methods × 2 models = 13,248 responses) was scored by MPQA adjective sentiment and GloVe-based fill-in-the-blank framing on DeepSeek and ChatGPT. Bias proved hierarchical. At the lexical level it was shallow and correctable: defensive prompting narrowed the left-right gap by up to 85%. At the source level it was deeper and resistant: the Fox News label elicited more negative evaluations of protesters even under instructions to ignore the source. Bias was also task-dependent: adjective scoring detected large fluctuations, while fill-in-the-blank outputs converged on left-leaning concepts (justice, rights) regardless of manipulation. Shallow lexical bias may yield to prompt-level fixes; source-level bias likely requires intervention at training or alignment.