A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.
Abstract
The rapid development of Large Language Models (LLMs) accelerates the adoption of LLM-based agents in software engineering, data analysis, and automated decision-making. To support agent development, various open-source frameworks provide modular abstractions for planning, tool invocation, and memory management, lowering the engineering barrier. Although widely used, research mainly emphasizes capability modeling and architectural innovation, while systematic analysis of engineering practices and real-world usage remains limited. This paper presents a large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective. We first identify the ten most popular frameworks in the open-source community and examine their popularity and language support. By analyzing official documentation, API interfaces, and source-level implementations, we extract common design principles and distill five best practices. We then formalize corresponding defect patterns and perform stratified proportional sampling on 73,014 downstream projects to evaluate how these practices are implemented. The results reveal recurring issues, including incomplete agent description, improper memory Strategy, insufficient monitoring and error handling, and suboptimal tool design. These defects appear across frameworks, indicating structural gaps between recommended abstractions and practical adoption. Overall, this study provides empirical evidence linking framework design choices to engineering risks and highlights the need for stronger guidance and support in agent framework development.
The rise of Software Engineering (SE) agents, i.e., LLM-based agents that can understand large codebases and carry out engineering tasks with limited human intervention, has been marked by rapid advances and adoption, but little is known about how developers build these systems in practice: existing studies mine repositories or examine deployment, but few investigate how SE agents are constructed. Through semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, this paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face. We find that as implementation becomes cheaper, bottlenecks shift rather than disappear: long-standing work in requirements, coordination, and deployment becomes more visible, while reviewing generated code and evaluating agent behavior become new and increasingly central forms of work. We characterize a seven-stage workflow and five process shifts, including a move toward evaluation-driven development, in which evaluation is increasingly defined early and steers iteration, and the emergence of specifications as first-class artifacts that teams test and version alongside code. We further identify six challenges that teams face, together with 12 corresponding practices they use or propose to address them, including unreliable evaluation signals, comprehension debt as code outpaces understanding, and behavioral changes introduced by provider-side model updates.
Yunbo Lyu, David Williams, Jieke Shi et al.· 0 citations
Context. Autonomous coding agents are increasingly used in software development, shifting parts of the engineering process to AI assistance. While this automation brings clear benefits, it introduces challenges in governance, traceability, and control over agent behavior. Agent Context Files (ACFs) have emerged as a practical mechanism to guide agents through structured instructions, yet little is known about how these artifacts are maintained and how their evolution relates to code development. Objective. This paper plans to investigate the evolution of ACFs and their role in agent-driven development. Specifically, we (1) classify ACF changes through a taxonomy grounded in software maintenance theory, (2) analyze how different types of changes are associated with code quality outcomes, and (3) examine their temporal patterns across the development lifecycle. Method. We conduct a large-scale mining study combining repositories with ACFs and agent-generated commits. We reconstruct ACF evolution at the commit level, classify changes using a qualitative approach, and analyze their association with code quality metrics. Statistical analyses and hypotheses are used to evaluate differences across maintenance categories, to inform future design of ACFs for governing autonomous coding agents.
Gianmario Voria, Alfonso Cannavale, A. Lucia et al.· arXiv.org· 0 citations
UML class diagram modeling is a fundamental task in software engineering, yet the application of large language models (LLMs) to this domain remains underexplored. Existing studies predominantly focus on single closed-source models with simple prompting strategies, lacking systematic comparisons across model types, prompt engineering techniques, and iterative refinement approaches. In this paper, we construct a difficulty-stratified dataset of 30 UML class diagram exercises and propose an automated weighted evaluation metric over generated PlantUML code—both of which are rarely constructed and systematically applied in existing LLM-driven UML modeling research. We present a preliminary empirical evaluation comparing open-source and closed-source LLMs across multiple scales and types, diverse prompting strategies, and varying requirement complexity levels. Beyond the single-round static paradigm of prior work, we further introduce and evaluate iterative prompting schemes that continuously improve model outputs through structured feedback. Our findings reveal that chain-of-thought prompting has different effects on improving the quality of different models, that relationship modeling is the persistent bottleneck under increasing complexity, and attribute extraction remains a largely unsolved technical challenge across all tested LLMs. Further, automated feedback-driven iterative refinement yields varied improvements: it brings notable performance gains for reasoning-oriented thinking models while delivering only marginal promotion for high-performance general chat models. These results provide actionable guidance for practitioners and researchers applying LLMs to UML modeling tasks.
Yong Cheng, You Huang, Shixin Yao et al.· Applied Sciences· 0 citations
User experience (UX) evaluation is central to software quality, yet it remains difficult to integrate into software engineering workflows in a systematic, explainable, and early-stage manner. This paper presents TraceUX, a framework for operationalizing UX heuristics and design guidance into a rule-based static evaluation pipeline that combines machine-interpretable formalization, executability-aware assessment, context-sensitive scoring, and actionable reporting. The framework is instantiated using Apple Human Interface Guidelines, Swift abstract syntax trees, and mobile games, and implemented in a proof-of-concept tool named TraceHIG. Evaluation was conducted in four layers: analysis of the full rule repository, controlled synthetic validation with injected violations, baseline assessment of 12 public Swift game projects, and a case study on one project. The full repository contained 206 rules; after excluding non-iOS yet platform-specific rules, 193 rules were retained for the downstream experiments. In controlled validation, 216 injected violations yielded 99.2% precision, 61.6% recall, and an F1-score of 0.760. In baseline analysis, overall project scores ranged from 41.6 to 88.0, reflecting rule-conformance spread under the instantiated rule base rather than direct measures of UX quality. The case study demonstrated that profile-aware scoring can yield materially different UX assessments for the same codebase under different game configurations, highlighting the importance of app profiling in static UX evaluation. These findings show that a meaningful subset of UX knowledge can be operationalized into explainable, context-aware static analysis that provides structured and actionable decision support while complementing, rather than replacing, manual and empirical UX evaluation.
Agents built on large language models (LLMs) are increasingly used to build applications that perform complex, multi-step tasks involving reasoning, tool use, and interaction with external environments. Despite rapid progress in benchmarking LLM-based agents, very few studies have attempted to understand how such systems are tested in practice. In particular, testing levels, objectives, data patterns, test complexity, and validation strategies for agent applications remain underexplored. In this paper, we present an empirical study of testing practices in LLM-based agent applications using a large corpus of mined open-source projects. We construct a large-scale dataset of agent applications, tools, and tests, and manually label 2,572 test methods from 240 modules. From this analysis, we derive a taxonomy of 23 testing patterns across test fixtures, data, objectives, and assertions, and characterize tests by level (unit, module, integration). We complement this with structured interviews of 10 senior industry practitioners building agentic systems. Our results show that testing of LLM-based agent applications is dominated by narrowly scoped unit tests, with limited coverage of complex interactions, realistic scenarios, and non-functional requirements. Tests frequently rely on simplistic inputs, heavy mocking, and shallow validation, and agent-related tests exhibit low structural complexity. While industry practice places greater emphasis on non-functional testing than open-source projects, both reveal common gaps, including the lack of formal testing foundations, unclear test objectives, and challenges in generating high-quality test data. Based on these findings, we outline research directions toward more systematic and rigorous testing of agent applications, including foundations for agent testability, formalized test objectives, and fault-based testing techniques.
Rangeet Pan, Tyler Stennett, Divya Sankar et al.· 1 citation
Enterprise software requires specification governance to transform probabilistic AI generation into deterministic, auditable engineering, and the SGRM framework is introduced, which defines four-component specification contracts, constrains stochastic generation via deterministic validation, and integrates generation, verification, and governance into a closed-loop architecture.