How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests

Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality

This paper empirically study the development and release evolution of five major open-source agent harnesses, revealing extreme release velocities exceeding two releases per day and thousands of issues within months, and performs the first controlled longitudinal study that isolates the agent harness contribution.

Towards Autonomous Software Development

When Code Authors Are Agents: A Large-Scale Study of Human–Agent Collaboration in Pull Requests

Autonomous coding agents are increasingly participating in collaborative software development by generating repository-level pull requests (PRs) that must be reviewed and integrated by human teams. While prior work has examined the technical characteristics of agent-generated patches, less is known about how autonomous authorship reshapes human collaboration dynamics in real-world workflows. In this paper, we investigate large-scale human–agent collaboration by comparing 40,214 pull requests across 2,807 GitHub repositories, including 33,596 agent-authored PRs from five autonomous coding agents and 6,618 human-authored PRs. We examine differences across three dimensions: integration outcomes, structural characteristics, and collaboration signals. Our findings reveal a socio-technical trade-off. Agent-authored PRs are integrated significantly faster, yet exhibit lower merge rates overall. Task type moderates this effect: agents outperform humans in documentation tasks but underperform in behavior-changing contributions. Beyond outcomes, collaboration patterns differ systematically. Agent-authored PRs attract proportionally more bot-generated comments and elicit more analytic, less socially oriented review communication. In contrast, human-authored PRs receive more elaborative and socially engaged feedback. Incorporating psycholinguistic features into predictive models significantly improves merge outcome prediction, demonstrating that communication style carries explanatory power beyond structural code characteristics. These results suggest that autonomous agents do not merely introduce technical artifacts into repositories, but actively reshape review interaction patterns and evaluative behavior. The impact of coding agents is therefore fundamentally socio-technical, highlighting the importance of studying AI systems within authentic human–AI collaborative environments.

A Preliminary Study on How AI Coding Agents Describe Refactoring Activities

Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.

A Preliminary Study on How AI Coding Agents Describe Refactoring Activities

Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.

Using Process Mining to Generate AI Agents from Software Engineering Process Records

Integrating AI agents into Software Engineering (SE) raises an important challenge: how can we specify and realize AI agents that work effectively alongside humans in hybrid SE teams? Determining the right granularity and separation of concerns for such agents is non-trivial. Coarse-grained agents may introduce unmanageable complexity, whereas micro-agents may create severe coordination overhead. Moreover, existing multi-agent SE frameworks typically rely on predefined role structures and do not account for project-specific characteristics or process adaptations. We address this by combining object-centric, imperative, and declarative process mining. Using event logs extracted from software repositories, our approach discovers project-specific agent roles using a predefined SE role vocabulary grounded in repository behavior and generates matching agent specifications and implementations. As proof-of-concept, we applied our approach to a well-established open-source project. We performed functional tests and an exploratory user study to determine how well the generated AI agent specifications are aligned with human expectations.