Diagnosing and Repairing Factual Errors in RAG under Budget Constraints

Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness

Large Language Models (LLMs) have shown strong potential for regression-oracle completion, where a test prefix is given and the current program version is treated as expected behavior. Recent approaches increasingly rely on iterative self-repair and execution feedback, but optimizing execution success does not necessarily yield strong fault-revealing oracles. This objective, widely adopted in repair-based methods, serves only as a proxy and may be misaligned with the true goal of oracle generation. Such misalignment biases the repair process, giving rise to a feedback-driven degeneration that we term the Self-Repair Trap, where iterative repair progressively drives models toward assertions that are easier to satisfy but less effective at detecting faults. To address this issue, we propose DCAware, a computationally efficient, non-iterative framework that prioritizes high signal-to-noise contextual grounding over multi-round repair. DCAware integrates structured static context with selectively retrieved dynamic states, enabling precise and robust oracle generation without iterative feedback loops. Extensive experiments based on execution and mutation testing show that DCAware consistently improves fault-revealing effectiveness while maintaining high execution success, outperforming prior methods with substantially lower computational cost. Our results suggest that improving contextual quality is more effective than adding iterative repair complexity in the studied regression-oracle setting.

TraceStructRepair: Effective Diagnostic Representation for Context-Limited Automated Program Repair

Self-supervised automated program repair (APR) leverages project-specific perturbations to generate training data and uses test execution diagnostics to guide patch generation. In practice, however, diagnostics are heterogeneous (e.g., exception messages, stack traces, assertion diffs, and dynamic execution signals) and must fit within a strict context budget. Naive concatenation either truncates critical evidence or amplifies noisy artifacts, especially for deep bugs where the failure symptom is far from the root cause. We present TraceStructRepair, a diagnostic structuring and budgeting approach for execution-aware self-supervised APR. TraceStructRepair (1) extracts a compact set of execution diagnostics from a single failing test, including exception type and message, stack trace frames, assertion diffs, and optionally dynamically loaded classes; (2) normalizes and ranks diagnostic elements to reduce redundancy and framework noise; and (3) assembles a field-aware representation under a fixed token budget with per-field caps and lexicographic priority rules. We implement TraceStructRepair on top of the SelfAPR pipeline and evaluate it on Defects4J using a project-wise heldout protocol. Beyond end-to-end repair outcomes, we analyze robustness under noisy fault localization and component ablations. We release artifacts to facilitate replication and future work on execution-aware, budget-constrained APR.

Att2RAG: A Double-Condition Framework for Knowledge Poisoning Attacks on RAG Systems

Modern retrieval-augmented generation (RAG) and memoryaugmented LLM applications are widely deployed in knowledge-intensive settings. These systems ground model outputs on external knowledge stores and may persist interaction traces in vector memory. If the underlying store is compromised, poisoned content can be retrieved repeatedly and thereby shape downstream responses, yielding confident yet harmful outputs supported by seemingly plausible evidence. Recent studies have shown that RAG pipelines and LLM agents are vulnerable to knowledge poisoning and prompt injection, but many formulations treat attack success as a single end-to-end outcome and do not separate retrieval and generation failure modes in a retrieval–generation aligned manner. Moreover, although long-horizon memory writes can introduce cumulative risks, these effects are not empirically evaluated under our benchmark setting. We present Att2RAG, a double-condition framework for knowledge poisoning attacks on RAG systems. Att2RAG decomposes a successful poisoning event into a retrieval condition and a generation condition, and casts poisoning as maximizing attack success subject to satisfying both conditions. We instantiate the framework with (i) a white-box variant that applies projected gradient descent (PGD) in embedding space, serving as an approximate upper bound under strong attacker assumptions, and (ii) a black-box attack based on a Q ⊕ I construction that combines problem self-similarity with adversarial instruction injection using only query access. We evaluate Att2RAG on representative QA benchmarks under several common RAG configurations and consider practical defenses such as paraphrasing and duplicate filtering. Attack success is measured by ASR and Target-F1 relative to an attacker-specified target response. Across the evaluated configurations, Att2RAG attains attack success rates in the mid-90% range without defenses; paraphrasing plus duplicate filtering reduce ASR only modestly, and many attacks remain successful. These results highlight limitations of semantic-similarity-driven retrieval and suggest that strengthening RAG systems requires defenses beyond surface-form rewriting and naive duplicate removal.

TraceGate: Policy-Based Disclosure of Structured Crash Evidence for LLM-Assisted Debugging

Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests. We propose Q-CARE, a query-agnostic and fully reference-free framework that enables fine-grained assessment by decomposing queries into sub-queries and answers into atomic claims. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework. Code and data are publicly available at https://github.com/DISL-Lab/Q-CaRE-COLM-26.

When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutations

Retrieval-Augmented Generation (RAG)-based LLM systems rely on external document corpora that can evolve and change over time. However, current evaluation methodologies (e.g., RAGAS) assess correctness against static snapshots, failing to detect faults when routine updates, factual changes, or noise alter the underlying data. We introduce a metamorphic testing framework that evaluates the consistency of RAG systems under corpus evolution. We formalise a fault taxonomy and 11 mutation operators that systematically perturb the system at both the pre-chunk (retrieval index) and post-chunk (retrieved context) levels. An empirical evaluation across five datasets and over 28k mutants reveals metamorphic violation rates of 4.9-10.2%. In a meta-evaluation against ground truth, our metamorphic oracle achieves F1 scores of 0.927-1.000, while the best RAGAS metric reaches only 0.570. Finally, we provide actionable insights into mitigating these faults through retrieval re-configuration, generator upgrades, and LLM-based reranking.