Reasoning with Large Language Models for Relevance Judgements

Abstract

Relevance judgements are central to information retrieval (IR) evaluation, and there is much interest in using large language models (LLMs) for these judgements. This work presents a novel framework that operationalizes explicit reasoning as a control mechanism for LLM-based relevance judgements and systematically quantifies its effects across evaluation settings. We compare judgements with and without reasoning to assess the impact on accuracy and reliability across multiple datasets and models (e.g., gpt-oss, Gemini 2.5). Our results show that using reasoning often leads to lower-quality judgements and a higher rate of formatting failures, while incurring substantially higher token costs. These findings raise important questions about when and how reasoning should be incorporated into evaluation pipelines, particularly for passage-level relevance assessment, suggesting that its benefits depend on task context and complexity rather than being universally beneficial. This work also contributes a novel dataset of LLM reasoning outputs, augmenting two established test collections to support future research.