A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.
Abstract
The rapid development of Large Language Models (LLMs) accelerates the adoption of LLM-based agents in software engineering, data analysis, and automated decision-making. To support agent development, various open-source frameworks provide modular abstractions for planning, tool invocation, and memory management, lowering the engineering barrier. Although widely used, research mainly emphasizes capability modeling and architectural innovation, while systematic analysis of engineering practices and real-world usage remains limited. This paper presents a large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective. We first identify the ten most popular frameworks in the open-source community and examine their popularity and language support. By analyzing official documentation, API interfaces, and source-level implementations, we extract common design principles and distill five best practices. We then formalize corresponding defect patterns and perform stratified proportional sampling on 73,014 downstream projects to evaluate how these practices are implemented. The results reveal recurring issues, including incomplete agent description, improper memory Strategy, insufficient monitoring and error handling, and suboptimal tool design. These defects appear across frameworks, indicating structural gaps between recommended abstractions and practical adoption. Overall, this study provides empirical evidence linking framework design choices to engineering risks and highlights the need for stronger guidance and support in agent framework development.
This paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face, and describes a seven-stage workflow and five process shifts, including a move toward evaluation-driven development.
Yunbo Lyu, David Williams, Jieke Shi et al.· 0 citations
A large-scale mining study combining repositories with ACFs and agent-generated commits is conducted, reconstructing ACF evolution at the commit level, classify changes using a qualitative approach, and analyze their association with code quality metrics.
Gianmario Voria, Alfonso Cannavale, A. Lucia et al.· arXiv.org· 0 citations
UML class diagram modeling is a fundamental task in software engineering, yet the application of large language models (LLMs) to this domain remains underexplored. Existing studies predominantly focus on single closed-source models with simple prompting strategies, lacking systematic comparisons across model types, prompt engineering techniques, and iterative refinement approaches. In this paper, we construct a difficulty-stratified dataset of 30 UML class diagram exercises and propose an automated weighted evaluation metric over generated PlantUML code—both of which are rarely constructed and systematically applied in existing LLM-driven UML modeling research. We present a preliminary empirical evaluation comparing open-source and closed-source LLMs across multiple scales and types, diverse prompting strategies, and varying requirement complexity levels. Beyond the single-round static paradigm of prior work, we further introduce and evaluate iterative prompting schemes that continuously improve model outputs through structured feedback. Our findings reveal that chain-of-thought prompting has different effects on improving the quality of different models, that relationship modeling is the persistent bottleneck under increasing complexity, and attribute extraction remains a largely unsolved technical challenge across all tested LLMs. Further, automated feedback-driven iterative refinement yields varied improvements: it brings notable performance gains for reasoning-oriented thinking models while delivering only marginal promotion for high-performance general chat models. These results provide actionable guidance for practitioners and researchers applying LLMs to UML modeling tasks.
Yong Cheng, You Huang, Shixin Yao et al.· Applied Sciences· 0 citations
User experience (UX) evaluation is central to software quality, yet it remains difficult to integrate into software engineering workflows in a systematic, explainable, and early-stage manner. This paper presents TraceUX, a framework for operationalizing UX heuristics and design guidance into a rule-based static evaluation pipeline that combines machine-interpretable formalization, executability-aware assessment, context-sensitive scoring, and actionable reporting. The framework is instantiated using Apple Human Interface Guidelines, Swift abstract syntax trees, and mobile games, and implemented in a proof-of-concept tool named TraceHIG. Evaluation was conducted in four layers: analysis of the full rule repository, controlled synthetic validation with injected violations, baseline assessment of 12 public Swift game projects, and a case study on one project. The full repository contained 206 rules; after excluding non-iOS yet platform-specific rules, 193 rules were retained for the downstream experiments. In controlled validation, 216 injected violations yielded 99.2% precision, 61.6% recall, and an F1-score of 0.760. In baseline analysis, overall project scores ranged from 41.6 to 88.0, reflecting rule-conformance spread under the instantiated rule base rather than direct measures of UX quality. The case study demonstrated that profile-aware scoring can yield materially different UX assessments for the same codebase under different game configurations, highlighting the importance of app profiling in static UX evaluation. These findings show that a meaningful subset of UX knowledge can be operationalized into explainable, context-aware static analysis that provides structured and actionable decision support while complementing, rather than replacing, manual and empirical UX evaluation.
This paper constructs a large-scale dataset of agent applications, tools, and tests, and manually label 2,572 test methods from 240 modules, and derives a taxonomy of 23 testing patterns across test fixtures, data, objectives, and assertions, and characterize tests by level.
Rangeet Pan, Tyler Stennett, Divya Sankar et al.· 1 citation
Enterprise software requires specification governance to transform probabilistic AI generation into deterministic, auditable engineering, and the SGRM framework is introduced, which defines four-component specification contracts, constrains stochastic generation via deterministic validation, and integrates generation, verification, and governance into a closed-loop architecture.