A method called Personalized Probing Puzzles to evaluate students' understanding of their own code, and a pilot study shows that it can help identify potential gaps in students' understanding of their own code.
Abstract
The rapid development and popularization of AI-enabled coding agents have meant software engineering students and professionals cannot be assumed to understand their own code, which risks academic integrity and professional accountability. We developed a method called Personalized Probing Puzzles ($p^3$) to evaluate students'understanding of their own code, and tested $p^3$ in a graduate-level cloud computing course. Our pilot study shows that $p^3$ can help identify potential gaps in students'understanding of their own code. The puzzles are automatically generated, asynchronously administered, and finished in minutes. Future work is needed to correlate puzzle results with code understanding and to embed $p^3$ in a professional code review process.
Generative AI is increasingly permeating software engineering, enabling developers to generate functions, files, and even entire applications from natural language specifications. AI systems are also becoming more personalized, adapting outputs based on inferred user characteristics and interaction history. While perso...
A vision for inclusive, human-centered AI-assisted code review systems that adapt feedback to developers' problem-solving preferences is outlined, highlighting the need to balance personalization with clarity and trust.
Shamse Tasnim Cynthia, Ratnadira Widyasari, Banani Roy et al.· 0 citations
TARS is an LLM-powered agent integrated into Visual Studio Code that supports program comprehension through autonomous explanations anchored directly to the code under analysis, built around a lightweight Theory of Mind paradigm.
Leopoldo Todisco, Antonio Della Porta, Stefano Lambiase et al.· arXiv.org· 0 citations
The rapid advancement of LLMs has opened new opportunities in automated software engineering, driving progress in code understanding, agent-based workflows, and productivity tools. However, existing code intelligence systems have largely sidelined the end-users they aim to serve—the developers themselves. Developers ex...
Yuhong Liu, Yu Su, Zhipeng Peng et al.· SIGSOFT FSE Companion· 1 citation
This research aims to bridge the gap by providing a comprehensive evaluation methodology for LLM-powered agents that is grounded in real-world software development practice and focuses on contamination-awareness, in thewild agentic behavior assessment, and trajectory-aware benchmarks and metrics.
This paper proposes integrating video 'vivas' into programming assessments as a mechanism for validating code comprehension, and discusses implementation considerations, marking rubric development aspects, and considers the broader implications of this approach.
Filippos Pantekis, O. Petrovska· Annual Conference on Innovat...· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comOct 12, 2026
StudentSim is a research system for evaluating and improving AI tutoring tools before they are tested with learners. It models how individuals may engage with tutors, giving feedback to improve tutor performance across subjects like chess, writing, and mathematics. The post Simulating student learning for better AI tutoring appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduOct 6, 2026
Writing as a participant and researcher, PhD student JS Tan SM ’22 has co-authored a new book about the rise of tech worker protests and the employer backlash that followed.
MIT News · Artificial Intelligence· news.mit.eduOct 2, 2026
Writing as a participant and researcher, PhD student JS Tan SM ’22 has co-authored a new book about the rise of tech worker protests and the employer backlash that followed.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
