Jul 2026· International Conference on Artificial Intelligence Testing· pp. 249-260· 0 citations· 26 references
Abstract
AI-enabled service-oriented systems change through code, data, prompts, service contracts, retrieval indices, and deployment workflows, which makes regression impact difficult to localize with code-centric evidence only. Existing regression test selection methods provide strong code-, configuration-, and service-level signals, but they provide limited guidance on how to reconcile structural, document, semantic, retrieval, and risk evidence when these signals disagree. This paper presents a multi-agent orchestration framework for change impact analysis and budget-aware regression test selection, where specialized agents score various evidence, a coordinator applies a fixed CI/CD budget, and a graph-backed variant records arbitration traces. We evaluate the framework on three case-study systems: an anonymized industrial wellness retrieval-augmented generation (RAG) platform with 300 tests, a microservice application with 150 tests, and a CI/CD pipeline with 60 tests. Each system has six snapshots, producing five evaluated change transitions, with two replicates per transition and the same budget rule across systems. The results are mixed and informative: the base multi-agent configuration is competitive with monolithic fusion, the graph-backed configuration recovers the strongest CI/CD score, and single-signal baselines remain strongest when one impact mechanism dominates. These findings position the multi-agent test selection approach as an observable, configurable decision framework for cross-domain impact analysis rather than a universally superior predictor.
As LLM coding agents increasingly perform end-to-end engineering work, we lack empirical characterization of how they behave on systems-level requirements: schema design, async orchestration, configuration correctness, and retrieval-filtering trade-offs. We present a case study of one such agent implementing a multi-component data system against a detailed pre-existing specification. Storage technologies, schema, entity-resolution algorithm, and retrieval-filtering strategy were fixed in advance; the agent autonomy was in the implementation, in diagnosing and fixing defects it introduced, and in interaction-design choices left open. Over a single session, we catalog five such defects, categorized by constraint violated and detection method. We further evaluate, on the public HotpotQA benchmark, the one retrieval trade-off specified in that architecture: restricting candidates to a graph-identified entity set before ranking versus unfiltered search. We substitute the benchmark gold evidence labels for entity identification, since we lacked LLM access to run that stage, and report standard recall rather than the benchmark own accuracy metrics. Across retrieval budgets from 1 to 10 and 100 questions against a pooled corpus of 2994 paragraphs, filtered recall reaches its ceiling by a budget of 3, expected once candidates are restricted to the gold paragraphs themselves, while unfiltered search recovers all required evidence only 69 percent of the time even at a budget of 10, a gap that holds at every budget tested, with sign test p less than 0.0001. We close with a discussion of where the agent autonomy succeeded versus required correction, including one instance where a claimed performance fix was never re-measured on the regression that motivated it.
A multi-agent testing framework in which requirement analysis, test-case generation, sandboxed execution, and defect detection are delegated to four distinct agents, and in which detection rests on a composite rule combining the execution signal with the semantic judgment of a dedicated diagnostic agent.
Yuxuan Li· Transactions on Computing Sc...· 0 citations
Maintaining open-source repositories demands continuous attention to issue triage, code review, build monitoring, and community support—tasks that overwhelm individual maintainers when existing automation tools operate in isolation without shared context. Static analyzers check code quality, CI/CD systems run automated tests, and simple bots handle basic labeling, yet none of these tools share information with each other, leaving maintainers as the sole integration point between disconnected systems. We present Repogent, a multi-agent system where four specialized AI agents—Issue Manager, Pull Request Reviewer, CI/CD Maintainer, and Community Assistant—collaborate through event-driven coordination and persistent cross-task memory. Our GitHub webhook-driven architecture processes events through a priority queue, routes them to specialized agents via intelligent orchestration, and stores decisions in a persistent context layer that enables cross-task reasoning. Events are assigned CRITICAL, HIGH, or NORMAL priority, while a semantic memory module based on sentence-transformer embeddings supports code-level search, and all agents share a common LLM service using Qwen 3 32B accessed through the Groq API. Evaluation on a dataset of 150 issues, 100 pull requests, 80 CI/CD workflow executions, and 120 community queries collected from five active open-source Python repositories demonstrates that Repogent achieves 93.3% accuracy in issue classification, 69.0% combined review coverage for pull requests, 86.7% accuracy in CI/CD failure categorization with a macro F1-score of 0.884, and an average quality score of 3.96 out of 5 for community responses. These results establish a foundation for sustainable open-source maintenance through context-aware, multi-agent workflow automation. Furthermore, the proposed framework demonstrates that multi-agent orchestration with persistent shared memory can serve as an effective and scalable solution for automating repository maintenance activities, improving coordination across heterogeneous development workflows, reducing maintainer overhead, and enabling more consistent decision-making across repository events.
Venkata Satya Santhi Somisetti, Vijaya Bhaskar Santhuluri, Sai Teja Pathivada et al.· International Journal of Lat...· 0 citations
DSA is presented, an evidence-aware orchestration framework for multi-market stock research with large language model (LLM) agents that establishes implementation conformance for the tested software contracts, not superior report quality, forecasting accuracy, or investment returns.
DDBench is introduced, a code-repair benchmark of 60 historical bugs mined from 13 open-source distributed systems, partitioned into three difficulty tiers, isolating the effect of debugging context from model capability.
Yibo Yan, Huijuan Wang, Junzhou He et al.· 0 citations
A contract-bounded runtime architecture, a source-preserving data substrate, and a falsifiable measurement protocol are contributed, which proposes a cluster-period randomized crossover experiment with a four-state verdict: supported, falsified, conditional-engineering, or inconclusive.
Ya-Xiao Liu, Peng Liu, Yi-Wen Liu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
