Should I Be Polite to My LLM Relevance Judge? Tone as a Severity Operating-Point Shift
TL;DR
The account reconciles prior contradictory findings and identifies prompt tone as a potential validity threat when absolute relevance labels matter.
Abstract
Large language models are increasingly used as relevance judges, yet their labels can shift with prompt surface form. We study one such feature—tone—on 3, 498 TREC DL19/DL20 query–passage pairs, across eight judge models, five classifier-calibrated politeness levels, and three paraphrases per level. Effects are strongly model-dependent: one judge shows a structured U-shaped response, whereas most show only small changes. Where tone changes agreement, the results are more consistent with a shift in the judge’s severity operating point—its overall scoring leniency—than with improved judgment. Agreement rises or falls as this shift moves the judge toward or away from human annotators’ strictness. A query-disjoint cross-fit retains the expected association (Spearman ρ = −0.683; exact model-block permutation p = 0.019). Tone affects calibration-based agreement more than ranking outcomes: across 32 model–tone contrasts, the largest absolute mean change in NDCG@10 is 0.011, although Kendall’s τ as low as 0.743 shows that reordering is reduced, not absent. The account reconciles prior contradictory findings and identifies prompt tone as a potential validity threat when absolute relevance labels matter.
