Skip to content
Open access

Breaking the Top-k Assumption in Pseudo-Relevance Feedback: An Empirical Analysis of Relevance Distribution in BM25 Rankings

Aug 2026 · Engineering, Technology & Applied Science Research · Vol 16, pp. 37231-37238 · 0 citations · 17 references

TL;DR

It is established that document selection, rather than term weighting, is the primary lever for PRF improvement and the vocabulary gap is identified as the principal target for next-generation methods.

Abstract

Pseudo-Relevance Feedback (PRF) has remained a cornerstone of unsupervised retrieval since Rocchio (1971), yet the foundational assumption that the top-  retrieved documents are the best available feedback has received limited direct empirical scrutiny, despite widespread adoption in both classical and neural approaches. This study analyzed BM25 retrieval on the TREC Deep Learning 2019 and 2020 test collections (97 queries, 20,646 graded relevance judgements) and found that 79% of relevant documents fall outside the top-10, with a mean rank of 37.3. An oracle selection strategy achieved 0.324 higher feedback precision at , defined as the proportion of graded-relevant documents within the  selected for expansion, with a large effect size that is consistent across all tested values of  and both test collections. LLM-based analysis of 61 extreme cases identified vocabulary gap as the dominant failure mode in 96.7% of cases, driven primarily by implicit relevance (35.6%) and hypernym-hyponym mismatch (27.1%). These findings establish that document selection, rather than term weighting, is the primary lever for PRF improvement and identify the vocabulary gap as the principal target for next-generation methods. The results demonstrate that improving feedback-document selection represents a largely unexplored avenue for PRF advancement.

Read PDF

Similar papers

Book Open access Sep 2026

Seek: Self-Evaluative Exploration for Knowledge Retrieval

seek, Self-Evaluative Exploration for Knowledge Retrieval, a training-free framework that addresses this limitation through iterative corpus interaction at test time through iterative corpus interaction at test time.

Amin Bigdeli, Radin Hamidi Rad, Negar Arabzadeh et al. · 1 citation
#small language model Book Open access Nov 2026

Effective Re-Ranking with DIMEs using Few Noisy LLM-Generated Relevance Judgments

New approaches based on the recently proposed Dimension IMportance Estimators (DIMEs) are introduced, incorporating LLM-generated relevance judgments into the re-ranking process, showing that even a few noisy relevance judgments from small LLMs can provide a powerful signal to improve the effectiveness of IR systems.

Andrea Pasin, Guglielmo Faggioli, Nicola Ferro et al. · 0 citations
Book Open access Nov 2026

LLMJudge: Meta-Evaluating Automatic Relevance Judgments

Large language models (LLMs) are increasingly used to produce the relevance labels that underpin the evaluation of information retrieval (IR) and retrieval-augmented generation systems. Yet delegating relevance assessment to an LLM raises a meta-evaluation question that is hard to answer with existing resources: when,...

Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi et al. · 0 citations
#natural language process... Preprint Oct 2026

Finding the Right Balance: Relevance and Diversity in LLM Retrieval

It is concluded that diversification should be used selectively, based on observable redundancy and evidence requirements, and introduces a query-adaptive rule that diversifies only when the effective number of distinct documents in the nearest-neighbor top-$k$ selection falls below the query's evidence requirement.

Guillaume Brouillette, Faustin Kagabo, Usef Faghihi et al. · 0 citations
Preprint Oct 2026

Beyond the Vote: When Do LLMs Add Value to Retrieval-Based Tabular Classification?

Retrieval-based in-context learning (ICL) enables large language models (LLMs) to predict tabular labels from similar examples, but it remains unclear whether LLM inference improves on direct voting over those same examples. We address this question by comparing four general-purpose LLMs with distance-weighted nearest-...

Zi-Quan Wang, Yu-Yang Nie, Jian Ma et al. · 0 citations
#natural language process... Review Open access Oct 2026

The Fusion Tax: A Diagnostic Analysis of When Hybrid Retrieval Hurts

Hybrid retrieval, which fuses a sparse lexical retriever with a dense neural one, is widely recommended in the software documentation surveyed in September 2026, usually with reciprocal rank fusion (RRF), on the strength of aggregate gains. We ask what fusion costs on individual queries, treating “do not fuse” as a fir...

A. Chauhan, Arun Agarwal · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.