Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

Bowen Qin Yi Xie

Jul 2026 · arXiv.org · Vol abs/2607.24882 · 0 citations · 22 references

Computer Science

TL;DR

A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.

Abstract

Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task. We introduce Agent Retrieval Bench, a file-level benchmark for this upstream retrieval problem. Samples are built from real coding-workflow signals and evaluated against frozen base-commit repositories, with relevance defined by what an agent needs next rather than direct query-file semantic similarity. The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple; a fifth subset evaluates selective retrieval using natural evidence-backed no-gold cases and counterfactual wrong-repository controls. Agent Retrieval Bench contains 427 samples across 25 repositories: 345 positive examples, 50 natural no-gold examples, and 32 counterfactual controls. The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. We evaluate lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates: Qwen3-Embedding-4B has the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B the best Recall@20, and RepoMap the best budgeted context yield at 8K tokens, with task-level winners differing substantially. Selective thresholds calibrated with counterfactual controls do not improve selective success on natural no-gold cases, revealing a calibration gap. Logged trajectories also miss every gold file on 27-35 percent of samples. A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.

DOI arXiv Semantic Scholar

View source

Similar papers

Open access Jul 2026

From noisy feedback to evidence-aware issue specifications: an agent-governed retrieval-augmented generation approach

Results show that AGR achieves strong decision accuracy in triage and evidence verification, and produces more actionable and engineering-useful issue specifications than both raw feedback and a strong LLM baseline, while reducing unsupported details.

Zhiyao Wang, Jialong Li, Xiujing Guo et al. · International Conference on... · 0 citations

Preprint Aug 2026

Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study

The results show that the protection deep agentic search offers may not be free, and that for read-only questions over a repository that can be indexed, retrieval was the stronger and cheaper option.

Amirkia Rafiei Oskooei, Bora Ilci, Alperen Kayim et al. · 0 citations

Jul 2026

VecTree-RAG: An Agentic Retrieval-Augmented Generation Framework Combining Vector and Tree Retrieval for Efficiency and Accuracy

Although multi-turn inference remains more expensive than single-call retrieval, VecTree-RAG provides a structure-aware and traceable architecture for scientific literature question answering.

Xinyan Zhong, Yu-Wei Shi, Yu-Qi Wei et al. · arXiv.org · 0 citations

Jul 2026

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

It is shown that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents'informative bands.

Prakhar Khatri · arXiv.org · 2 citations

Jul 2026

Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation

Repository-level code generation relies on heterogeneous evidence whose relevance, compatibility, and completeness are inherently uncertain. Similar-code examples, repository context, and project-specific APIs may provide complementary information, but can also introduce noisy, redundant, or conflicting signals. Existing retrieval-augmented approaches primarily optimize retrieval relevance without explicitly modeling how uncertainty in retrieved evidence affects downstream generation. We introduce OpenCoder, an uncertainty-aware framework that estimates source-specific uncertainty, uses it to filter and rank heterogeneous evidence, and guides generation, verification, and repair. A factorial analysis over API knowledge, repository context, and similar-code evidence reveals no universal additive source ranking; instead, significant cross-source interactions depend on the accompanying evidence and LLM backend. On an expanded 32-task RepoExec-inline evaluation, OpenCoder improves GPT selected-output correctness over Baseline RAG from 56.25\% to 78.13\%. However, it matches a verification-and-repair control, and the corresponding Gemini improvement is not statistically supported, indicating backend-dependent benefits. Target-aware API refinement also substantially improves API-set retrieval. These findings support treating uncertainty as an actionable control signal for repository-level retrieval, verification, and repair.

Chandan Kumar Sah, Xiaoli Lian, Li Zhang · arXiv.org · 1 citation

Jul 2026

Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs

BeyondUncertainty first elicits a structured provisional answer and confidence estimate, then applies a model-specific threshold selected on held-out validation data and frozen before test evaluation, revealing a trade-off between more selective evidence acquisition and end-to-end token efficiency.

Chandan Kumar Sah, Xiaoli Lian, Li Zhang · arXiv.org · 0 citations