LLMJudge: Meta-Evaluating Automatic Relevance Judgments

Abstract

Large language models (LLMs) are increasingly used to produce the relevance labels that underpin the evaluation of information retrieval (IR) and retrieval-augmented generation systems. Yet delegating relevance assessment to an LLM raises a meta-evaluation question that is hard to answer with existing resources: when, and how much, can we trust an automatic judge? We present LLMJudge, a resource for meta-evaluating LLM-generated relevance judgments. LLMJudge collects 42 complete judgment sets over the TREC 2023 Deep Learning track, spanning fine-tuned, proprietary, and open-source models and a wide range of prompting strategies. Each judgment set is paired with official human qrels, enabling two complementary axes of analysis: label agreement with human assessors (e.g., Cohen's κ, Krippendorff's α) and system-ranking fidelity (e.g., Kendall's τ, Spearman's ρ). Beyond the data, we release a standardized format, baselines, and an evaluation harness so that new judges can be plugged in and compared reproducibly. Our analysis shows that high label agreement is neither necessary nor sufficient for faithful system ranking, that open-source judges are markedly more stable than proprietary ones, and that fine-tuning improves correlation at the cost of reliability. LLMJudge is publicly available and provides a reusable testbed for auditing the judges who increasingly judge our systems. Full resources, data, and detailed results are available at https://llm4eval.github.io/LLMJudge-benchmark/.