This paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face, and describes a seven-stage workflow and five process shifts, including a move toward evaluation-driven development.
Abstract
The rise of Software Engineering (SE) agents, i.e., LLM-based agents that can understand large codebases and carry out engineering tasks with limited human intervention, has been marked by rapid advances and adoption, but little is known about how developers build these systems in practice: existing studies mine repositories or examine deployment, but few investigate how SE agents are constructed. Through semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, this paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face. We find that as implementation becomes cheaper, bottlenecks shift rather than disappear: long-standing work in requirements, coordination, and deployment becomes more visible, while reviewing generated code and evaluating agent behavior become new and increasingly central forms of work. We characterize a seven-stage workflow and five process shifts, including a move toward evaluation-driven development, in which evaluation is increasingly defined early and steers iteration, and the emergence of specifications as first-class artifacts that teams test and version alongside code. We further identify six challenges that teams face, together with 12 corresponding practices they use or propose to address them, including unreliable evaluation signals, comprehension debt as code outpaces understanding, and behavioral changes introduced by provider-side model updates.
A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.
Yibo Zhai, Junjun Si, Yan Wang et al.· SIGSOFT FSE Companion· 0 citations
AI agents are becoming a fundamental part of modern software creation, helping developers in generating code, debugging, designing systems, etc. But there is a clear difference between how beginners and experienced software engineers get benefits from these tools. Newbies usually depend on agents for one-time prompts and quick answers, whereas mature users utilize them through well-defined, repeated workflows that raise productivity and consistency. In this article, we discuss this difference and emphasize that getting the full potential does not merely depend on better prompts but on workflows driven by instructions developers create clear and reusable instruction files to direct agent behavior across tasks. When developers stop seeing agents only as chat interfaces but as programmable collaborators, they can produce more reliable and high-quality outputs. We offer in our paper methods like designing modular instructions, narrowing down the context, and iterative refinement loops, as well as a case study illustrating how a team made a code review more efficient and minimized the rework by making agent instructions standard. The results stress that structured forms of interaction rather than sporadic use are the main ways to tap into advanced features. Our paper provides a conceptual model for agent usage at large scale, hands-on advice for the implementation of instruction files in actual settings, and validation that skillful developers can far exceed basic usage by adopting orderly, system-like approaches to agent collaboration.
Madhurima Kommuru, Srujana Pulipaka· International Journal of Mod...· 0 citations
The use of large language models (LLMs) is being introduced into requirements, code generation, testing, maintenance, and documentation processes, but most IT organizations have yet to establish a practical and evidence-based methodology regarding when these tools are value added, when they become risky, and how to regulate their usage. The article is a synthesis of recent empirical research, surveys of developers, and guidance on the use of LLMs in software engineering and translates that information into a playbook of guidance that can be applied by practitioners. The primary contribution of the article is a staged adoption framework, which includes explore, pilot, and scale, supported with lightweight survey templates, small-task assessment designs, and accept/edit/reject logging practices that organizations can adopt to produce their own context-specific evidence. The objective is to facilitate disciplined, open-minded adoption of LLMs in actual software engineering environments.
Large language models (LLMs) are rapidly reshaping software development, but their impact across the full software development lifecycle is underexplored. Existing work tends to focus on isolated activities such as code generation or testing, leaving open questions about how LLMs affect developers, processes, products, and the broader software ecosystem. We address this gap through semi-structured interviews with sixteen early-adopter software professionals who integrated LLM-based tools into their day-to-day work in early to mid-2023. We treat these interviews as early empirical evidence and compare participants’ accounts with recent work on LLMs in software engineering, noting which early patterns persist or shift. Using thematic analysis, we organize our findings around four dimensions: people, process, product, and society. Developers reported substantial productivity gains from reducing mundane tasks, streamlining search, and accelerating debugging, but also described a productivity-quality paradox: they frequently discarded generated code and shifted effort from writing code to critically evaluating and integrating it. LLM use was highly phase-dependent, with strong uptake in implementation and debugging but limited influence on requirements gathering and collaborative work. Participants developed new competencies to use LLMs effectively, including prompt engineering strategies, multi-layered verification, and security-conscious integration to protect proprietary data. They also anticipated changes in hiring expectations, team practices, and computing education, while emphasizing that human judgment and foundational software engineering skills remain essential. Our findings, consistent with evidence from large-scale studies, offer actionable implications for developers, organizations, educators, and tool designers seeking to integrate LLMs responsibly into professional software practice.
Benyamin T. Tabarsi, Heidi Reichert, Sam Gilson et al.· Empirical Software Engineeri...· 0 citations
Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.
Aymen Masmoudi, Belhassen Khefacha, Andrew Haralambous et al.· Annual International Compute...· 0 citations
Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.
Aymen Masmoudi, Belhassen Khefacha, Andrew Haralambous et al.· Annual International Compute...· 0 citations