Skip to content

Evaluating Semantic and Quality-Aware Retrieval for Source Code Repositories

Jul 2026 · arXiv.org · Vol abs/2607.09161 · 1 citation · 21 references
Computer Science

TL;DR

Within the reported query set, semantic retrieval was the strongest overall mode, while explicit quality metadata was most useful for explicitly quality-oriented queries.

Abstract

Keyword-based retrieval is limited for source-code repositories when queries are expressed in natural language or concern implementation intent and code quality rather than exact tokens. This study evaluates a prototype retrieval system that combines function-level fragmentation, text-and-code embeddings, ChromaDB vector storage, LLM-derived quality metadata, and four retrieval modes: semantic, quality-filtered, hybrid, and automatic routing. The concrete evaluation uses an educational C-code corpus. The full corpus contains 563 anonymized programmer identifiers and 8,951 C files; a reproducible 10% indexed sample contains 56 programmer identifiers, 847 files, and 3,839 fragments. Across 15 manually judged queries, semantic retrieval achieved nDCG@5 of 0.820, Success@5 of 0.800, and MRR of 0.644. The automatic router selected the expected mode for all 15 queries. In a small manual audit, LLM-derived quality scores were within one point of the manual assessment for 9 of 12 fragments. Within the reported query set, semantic retrieval was the strongest overall mode, while explicit quality metadata was most useful for explicitly quality-oriented queries.

View source

Similar papers

Jul 2026

Retrieval-Oriented Code Representations in Agentic Bug Localization

LLM-based agents are increasingly being used to support software development, yet their performance in repository-level tasks depends on retrieving the right code context. Existing studies have explored file-level localization using traditional information retrieval over file paths and raw source code. However, the role of textual code representations in retrieval and localization remains underexplored. We study file-level bug localization as a representation-driven retrieval problem. Across the Long Code Arena (LCA) and SWE-bench Verified (SWE) datasets, we compare five code representations: file paths, raw source code, and three LLM-generated textual representations. Our experiments include lexical, semantic, and LLM-based retrieval, followed by LLM-based post-retrieval ranking. We quantify the cost incurred by a representation through the representation footprint. We find that the choice of code representation affects both localization effectiveness and cost. Role-aware summaries outperform file-path representations by up to 40% Hit@5 while requiring a representation footprint 10.4 to 20.9x smaller than raw source code. Combining complementary representation results and ranking retrieved candidates with an LLM provides further gains of up to 31.9% and 42.0%, respectively. Overall, role-aware summaries provide the best cost-effectiveness trade-off, while raw source code offers effectiveness in some settings at a significantly higher cost. A case study with Agentless reveals the utility of our techniques within a well-known pipeline, reaching 94% Hit@6 on file localization (+4.7% against the baseline). Our findings suggest that code representation should be treated as a first-class design choice in agentic localization pipelines, guided by pipeline stage and cost-accuracy requirements.

Genevieve Caumartin, T. Chen, Diego Elias Costa · 0 citations
Jul 2026

OwlPath: Lossless Knowledge Compression for LLM Bug Repair

LLM-based software engineering agents are constrained by limited context windows: roughly 100K tokens must store structurally relevant code subsets to resolve bugs. Standard retrieval models treat code as plain text, forcing agents to resolve multi-hop dependencies including subclass chains, transitive callers and interface implementations through slow trial and error. We tackle this limitation with lossless knowledge compression, encoding source code into an OWL2 ontology to answer structural queries using minimal relevant code fragments. We present OwlPath, an OWL2 reasoning layer atop CodeGraph, a widely used code intelligence platform with 500K+ GitHub stars, offering a unified CLI for structural code retrieval. Powered by tree-sitter parsing, OwlPath supports multi-language repositories (Python, JavaScript, TypeScript, Go, etc.) and encodes language-specific semantics into a unified OWL2 ontology. It adopts two complementary modules. First, a transitive-closure engine fetches all structurally linked symbols via single SPARQL property-path queries, capturing multi-hop relations missed by string matching. Second, the OWL Software Knowledge Map (OWL-SKM) precomputes a compact 3KB summary with module trees, core APIs and issue-related symbols, directing agents to target modules in the first query. Evaluated on 18 SWE-bench Pro instances, OwlPath obtains a 68.4% strict-apply rate versus 66.7% for the CodeGraph baseline, cutting token usage by 28.8% and runtime by 39.5%. In offline retrieval tests over 67 instances, OwlPath improves recall 2.06 times (0.464 vs 0.226) and reaches 88.1% hit rate compared to CodeGraph's 59.7%. On a 37-question structural retrieval benchmark, recall rises from 4.4% to 28.8%, with 69-80% accuracy on transitive caller and interface tasks.

Bo Zhang, Renke Pan, Huan Chen et al. · 0 citations
Book Open access Jul 2026

CODENS: Transforming Code Changes into Living, Accessible, and Queryable Documentation

Results show that CODENS produces highly relevant and well-grounded answers, while qualitative feedback highlights a remaining challenge in concise, documentation-oriented synthesis.

Abdelhak Kelious, Chyrine Tahri, E. Bardet · 0 citations
Jul 2026

ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation

Repository-level code generation requires implementing target functions while accounting for complex cross-file dependencies and project-specific conventions. Existing retrieval methods predominantly rely on lexical, structural, or semantic similarity, often overlooking repository functions that implement similar procedural logic despite differing in identifiers or application domains. We propose ProjAgent, a repository-level code generation system that introduces procedural similarity as an explicit retrieval signal. ProjAgent decomposes the target function into intermediate reasoning steps and employs an agentic workflow to retrieve repository functions that exhibit similar procedural behavior at each step. The retrieved procedural context is integrated with conventional semantic retrieval to construct a richer repository context for code generation. ProjAgent further incorporates a conservative static-analysis feedback loop that iteratively repairs generated code using compiler and static-analysis feedback. Evaluated on REPOCOD, ProjAgent achieves 41.14% Pass@1, outperforming existing retrieval-based baselines. These results demonstrate that procedural similarity is an effective and previously unexplored retrieval dimension for repository-level code generation.

Qihong Chen, Aaron Imani, Iftekhar Ahmed · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.