Ranking Passages in Relevant Documents Using LLMs
TL;DR
This work presents a study of ranking approaches, including lexical, dense and zero-shot prompted large language models (LLMs), to rank passages in relevant documents based on the presumed fraction of relevant text they contain, and demonstrates the merits of these approaches in utilizing relevance feedback.
Abstract
The ad hoc document retrieval task is to rank documents by their presumed relevance to a query. Most TREC benchmarks provide relevance judgments only at the document level, without indicating which parts of a document are actually relevant. Focused relevance judgments, which highlight relevant text at the character level, are valuable but scarce. In this work, we present a study of ranking approaches, including lexical, dense and zero-shot prompted large language models (LLMs), to rank passages in relevant documents based on the presumed fraction of relevant text they contain. Our analysis shows that LLM-based rankings are highly effective and outperform strong sparse and dense retrieval baselines. We demonstrate the merits of our approaches in utilizing relevance feedback: constructing relevance models from top-ranked passages in relevant documents yields performance that transcends that of relevance models constructed from the entire documents.