Jul 2026· International Conference on the Theory of Information Retrieval· pp. 187-198· 0 citations· 40 references
Computer Science
TL;DR
JaReIR is introduced, a Japanese benchmark built from Yahoo! Chiebukuro question-answer pairs and Japanese Wikipedia passages with crowdsourced relevance and answerability judgments and performance generally drops from Easy to Hard across retrieval settings, which shows that Japanese reasoning-intensive retrieval remains challenging.
Abstract
Reasoning-intensive retrieval has become increasingly important for Retrieval-Augmented Generation (RAG), where systems must retrieve evidence that supports an answer rather than rely on surface matching alone. However, existing benchmarks mainly focus on English and leave Japanese retrieval underexplored. We introduce JaReIR, a Japanese benchmark built from Yahoo! Chiebukuro question-answer pairs and Japanese Wikipedia passages with crowdsourced relevance and answerability judgments. JaReIR contains 153 queries split into Easy queries, which have at least one Direct answer passage, and Hard queries, whose positive passages all require reasoning. We benchmark sparse and dense retrievers on JaReIR and analyze the effects of query expansion and reranking. Performance generally drops from Easy to Hard across retrieval settings, which shows that Japanese reasoning-intensive retrieval remains challenging. Passage-level analyses and retrieved-title dispersion statistics show that Hard queries yield more ambiguous top-ranked results. Document-level retrieval reduces, but does not eliminate, the Easy-Hard gap, suggesting that passage localization explains part of the difficulty. Evaluating a translated Japanese version of an existing English benchmark under the same passage-level setup further suggests that translation noise and linguistic differences contribute to lower performance.
This work introduces AnchorQE, a training-free method that separately encodes the original query and its expansion before interpolating them, and shows that AnchorQE improves retrieval effectiveness by up to 12.89% when compared to widely-used expansion-only or text-level concatenation baselines across TREC-DL, LoTTE, and BEIR.
BeyondUncertainty first elicits a structured provisional answer and confidence estimate, then applies a model-specific threshold selected on held-out validation data and frozen before test evaluation, revealing a trade-off between more selective evidence acquisition and end-to-end token efficiency.
Chandan Kumar Sah, Xiaoli Lian, Li Zhang· arXiv.org· 0 citations
Evaluation on a multi-page technical PDF document shows that the hybrid retrieval and re-ranking stages together raise retrieval precision and reduce irrelevant or unsupported answers compared with retrieval limited to a single method, supporting the use of this approach for reliable, document-grounded question answering.
Vishwa K Dave, K. Pallavi· International Research Journ...· 0 citations
Reasoning-intensive temporal retrieval requires matching a query to documents whose relevance depends on shared temporal reasoning rather than lexical overlap. Expanding a query into several reformulations that make its temporal intent explicit, and retrieving with each, supplies this reasoning, but fusing the resulting rankings with equal weights wastes accuracy: for any single query, only some reformulations are reliable. We propose query-difficulty-gated fusion of reasoning views. From each view we read an eight-dimensional signature of its score distribution, built from query-performance-prediction quantities such as softmax entropy, score gaps, and dispersion, and a gate of roughly one thousand parameters maps these signatures to per-query view weights. The fused ranking uses no relevance labels at inference, no re-ranking, and no fine-tuning of the retriever; the gate is trained leave-one-task-out. On the \textsc{Tempo} benchmark, the method improves all six retrievers we evaluate, from BERT encoders to 7B decoder retrievers, with the largest gains on the weaker backbones. The strongest retrievers reach $0.297$ and $0.303$ nDCG@10, and the per-query gain over the original query is significant under a paired bootstrap ($p<0.001$). A per-query oracle reaches $0.364$ against our realized $0.297$, exposing headroom that identifies per-query view selection as a concrete next step.
J. Holdcroft, Abdelrahman Abdallah, Adam Jatowt· 0 citations
These findings establish learned sparse retrieval as a highly impactful design choice in automated fact-checking, with retrieval quality serving as a critical determinant of end-to-end performance in the studied systems.
Ritvik Setty, Vinay Setty· International Conference on...· 0 citations
Results show that observed evidence can guide graph retrieval toward the part of a supporting chain left underspecified by the original question, and introduce EviReform, which separates revising the retrieval request from aggregating evidence in the graph.
Xinlong Xu, Yoshua Y. Li· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
