This vision reframes AI code review from automated commenting to human-AI sensemaking before integration, and outlines a research agenda for studying review conversations, designing conversational AI review capabilities, and evaluating their impact on software evolution and maintenance.
Abstract
AI-based code review tools increasingly promise to help developers inspect pull requests, identify defects, and improve code quality. Yet most current approaches frame code review as a one-shot commenting task: given a diff, the system produces warnings or suggestions. This framing overlooks a central property of modern code review: review is a conversation. Human reviewers do not merely comment on code; they ask questions, explain expectations, negotiate design trade-offs, request evidence, transfer project knowledge, document rationale, and collectively decide whether a change is good enough to integrate. In this vision paper, we argue for conversational AI review assistants: systems that participate in code review as interactive partners rather than static comment generators. Such assistants should identify when conversation is needed, ask grounded questions, respond to developer explanations, summarize unresolved issues, help capture rationale, and know when to abstain or escalate to human reviewers. Such a paradigm shift requires novel evaluation methodologies as well. We outline a research agenda for studying review conversations, designing conversational AI review capabilities, and evaluating their impact on software evolution and maintenance. Our vision reframes AI code review from automated commenting to human-AI sensemaking before integration.
AI coding agents are generating code at volumes that exceed the capacity of traditional peer review. At the same time, existing AI code review tools over-index on low-value suggestions such as style and best practices while under-indexing on the concerns human reviewers prioritize most: correctness, security, and performance. We present ARCTIC, an AI-powered Code Critique system that reframes code review around three capabilities: intent prediction, which infers why a change was made from conversation logs and metadata; drift detection, which measures divergence between the developer's intent and the agent's output via backtranslation; and code spotlight, which ranks the regions of a diff most warranting human scrutiny. We ground these capabilities in a six-theme taxonomy derived from 18,000 code reviews. Offline evaluation shows that intent prediction achieves 0.86 F1, drift detection reaches near-perfect ordinal agreement with human annotators (QWK = 0.907), and spotlight outperforms the baseline AI reviewer by 2.4x on quality estimation at 5x fewer tokens. In the experimental rollout, the drift scores reduces code misalignment by an additional 5.76 points (p = 0.026), intent prediction receives 90.2% approval, and zero defects have been attributed to self-reviewed diffs since launch.
C. Maddila, Mashrur Rashik, E. Khan et al.· arXiv.org· 0 citations
A large-scale empirical characterization of AI-assisted OSS contribution is provided and practical insights for designing and governing responsible vibe-coding practices in open-source development are offered.
Code review is a critical quality assurance practice in software engineering development, and AI coding agents are increasingly generating review comments on pull requests. However, little is known about how developers actually respond to such agent-generated feedback. In this paper, we present the first large-scale empirical study on the resolution of agent-generated code review comments. We analyze $54{,}791$ comments generated by five widely used coding agents (i.e., Copilot, Cursor, Codex, Devin, and Claude) across $342$ Python repositories on GitHub. We examine (1) resolution rates across agents and comment types, (2) the role of developer experience, and (3) characteristics that influence comment usefulness. Our results show that resolution rate varies considerably across agents, with Copilot accounting for the majority of resolved comments (72.9\%). Core developers resolve the majority of agent-generated feedback, particularly for \textit{design} and \textit{evolvability}-related comments, while peripheral developers are more involved in resolving \textit{functional defect} comments. Through open card sorting of 470 unresolved comment discussions, we identify \textit{ten} discussion patterns explaining why comments remain unresolved, with \textit{incorrect suggestions} and \textit{intentional design decisions} being the most prevalent. Finally, our analysis reveals that the presence of an inline \textit{code suggestion} is the strongest predictor of comment resolution, while lengthy and complex comments are less likely to be acted upon. Our findings provide insights for improving AI-generated code review feedback and its integration into development workflows.
Shamse Tasnim Cynthia, Ratnadira Widyasari, Banani Roy et al.· 0 citations
Background: Large language models (LLMs) are increasingly used to automate code review, but the reasoning behind their decisions remains hard to understand. Developers struggle to assess the validity of LLM-generated reviews, making it difficult to gauge how much trust to place in them. The role of Explainable AI (XAI) in code review and its impact on trust remain underexplored. Objective: We study the influence of XAI on developer trust in AI-assisted code reviews. Method: We conducted a within-subjects user study with 34 participants, comparing three LLM-based code review systems with varying levels of XAI support: Condition A (detailed explanation and review feedback), Condition B (review feedback only), and Condition C (no explanations). Participants reviewed real-world code change requests alongside the AI-generated reviews. We measured trust perceptions, agreement with the AI recommendation, the reasoning given for each decision, and the time taken. Results: The level of explanation significantly influences both trust and agreement with AI recommendations, but in different ways. Full explanations (A) yield the highest perceived trust (M = 3.99/5) but not the highest agreement, whereas moderate explanations (B) achieve the highest agreement (89.22%). This could suggest that more explanation prompts developers to question AI recommendations more frequently. No explanations (C) results in the lowest trust and agreement. Explanation level did not significantly affect review time. The most commonly cited reasons for decisions were code readability and correctness. Conclusion: Incorporating XAI into code review significantly changes trust perceptions and agreement with AI recommendations. These results inform the design and evaluation of trustworthy AI-based code review systems, as well as studies on the human factors of AI-assisted software development.
Zhenhan Gao, Marvin Muñoz Barón, Umm E. Habiba et al.· arXiv.org· 0 citations
A vision for inclusive, human-centered AI-assisted code review systems that adapt feedback to developers' problem-solving preferences is outlined, highlighting the need to balance personalization with clarity and trust.
Shamse Tasnim Cynthia, Ratnadira Widyasari, Banani Roy et al.· 0 citations
The results show that agent-involved collaboration patterns, especially reviews initiated by AI agents or involving multiple AI agents, are associated with faster review decisions under Gradual AI Adoption and Rapid AI Agent Adoption, but these efficiency gains do not translate into better review quality.
Suzhen Zhong, Shayan Noei, Bram Adams et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
