Engineering Practices in Open-Source Agent Frameworks: A Large-Scale Empirical Study
TL;DR
A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.
Abstract
The rapid development of Large Language Models (LLMs) accelerates the adoption of LLM-based agents in software engineering, data analysis, and automated decision-making. To support agent development, various open-source frameworks provide modular abstractions for planning, tool invocation, and memory management, lowering the engineering barrier. Although widely used, research mainly emphasizes capability modeling and architectural innovation, while systematic analysis of engineering practices and real-world usage remains limited. This paper presents a large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective. We first identify the ten most popular frameworks in the open-source community and examine their popularity and language support. By analyzing official documentation, API interfaces, and source-level implementations, we extract common design principles and distill five best practices. We then formalize corresponding defect patterns and perform stratified proportional sampling on 73,014 downstream projects to evaluate how these practices are implemented. The results reveal recurring issues, including incomplete agent description, improper memory Strategy, insufficient monitoring and error handling, and suboptimal tool design. These defects appear across frameworks, indicating structural gaps between recommended abstractions and practical adoption. Overall, this study provides empirical evidence linking framework design choices to engineering risks and highlights the need for stronger guidance and support in agent framework development.