Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools

Abstract

Standard reranking evaluations study how a reranker orders candidates returned by an upstream retriever. This setup couples ranking behavior with retrieval quality, so differences in output cannot be attributed to the ranking policy alone. We introduce a controlled diagnostic for reranking that uses Multi-News clusters as fixed evidence pools. We limit each pool to eight documents and pass identical inputs to all rankers. Within this setup, BM25 and MMR serve as interpretable reference points for lexical matching and diversity optimization. Across 345 clusters, we find that redundancy patterns vary by model: one LLM implicitly diversifies at larger selection budgets, while another increases redundancy. In contrast, LLMs underperform on lexical coverage at small selection budgets. As a result, LLM rankings diverge substantially from both baselines rather than consistently approximating either strategy. By reducing retrieval variance through fixed pools, we interpret these differences more directly as differences in ranking policy. This diagnostic is model agnostic and can be applied to any ranker, including open source systems and proprietary APIs. Our code and processed data for both the Multi-News diagnostic and the complementary TREC-DL evaluation are publicly available at https://github.com/barisarat/llm_reranker_multinews.git.