A configurable multi-agent framework for benchmarking LLMs in Prolog code generation that combines a Code Generator Agent, a deterministic execution layer using SWI-Prolog, and an evaluator based on the LLM-as-a-Judge paradigm that supports model-agnostic experimentation and evaluates outputs across functional correctness, efficiency, structural properties and evaluator reliability.
A multi-agent testing framework in which requirement analysis, test-case generation, sandboxed execution, and defect detection are delegated to four distinct agents, and in which detection rests on a composite rule combining the execution signal with the semantic judgment of a dedicated diagnostic agent.
Yuxuan Li· Transactions on Computing Sc...· 0 citations
A comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering and a set of lessons learned and challenges that can help researchers and practitioners to select a suitable MAS framework according to their needs are provided.
Maria Sâmyla Serafim de Oliveira, M. Ibiyo, Marco Gianrusso et al.· 1 citation
Multi-agent systems built on large language models (LLMs) are increasingly deployed for complex tasks requiring autonomous planning, tool use, and inter-agent coordination. However, the non-deterministic nature of LLM outputs and the emergent behavior arising from agent interactions render traditional test oracles inef...
Gopalakrishnan Marimuthu· International Conference on...· 0 citations
An empirical study of whether complete software artifacts generated by LLM coding agents can be executed in a clean environment using only the code, dependency specifications, and instructions the agent provides suggests that coding-agent evaluation should treat clean-environment executability as a first-class metric a...
The AgentCodeReview system is presented, a multi-agent system that is able to conduct explainable code review and automated bug repair by leveraging software engineering agents with different code review tasks and its utility and extensibility to the field of explainable AI in software quality assurance are demonstrate...
B. N, T. Manasa· International journal of com...· 0 citations
This paper proposes the Evaluation Context Protocol (ECP), an early-stage, vendor-neutral framework intended to act as a portable evaluation contract layer for agentic systems and describes an open-source reference implementation that includes adapters for LangChain, LlamaIndex, CrewAI, and PydanticAI.
