This study examined whether introductory Qiskit homework could remain autogradable while requiring students to run, review, and discuss results rather than banning AI.
Abstract
Generative AI creates an assessment challenge in quantum software education: a student can provide a homework notebook to ChatGPT and request a completed submission. This study examined whether introductory Qiskit homework could remain autogradable while requiring students to run, review, and discuss results rather than banning AI. Three packages were tested: seeded basis-state circuits with bit flips and customized measurement mappings; Quantum Fourier Transform followed by inverse-transform recovery; and seeded Deutsch-Jozsa with customized oracle masks. The designs used personalization, simulator execution, JSON submissions, hidden references, circuit metrics, reflections, and optional IBM Quantum execution. For each package, one student-visible instance was tested in 50 separate ChatGPT sessions, yielding 150 sessions overall. Every final artifact was executed and passed its grader. Nine sessions were fully archived; none required operator code changes or correction of quantum logic. Under the study's operational definition, each tested instance had zero observed ChatGPT-resiliency. Seeds changed parameters rather than task structure, expected results remained derivable from visible assignment logic, scaffolding exposed key solution steps, and hidden grading verified output consistency without establishing independent authorship or understanding. Because one instance was repeated for each package, the results do not establish solvability for every seed or possible Qiskit assessment. The tested personalized, execution-oriented take-home designs therefore did not prevent successful completion under a minimally engaged-student workflow. Correct artifacts should be complemented by direct assessment through supervised modification, oral defense, prediction, and transfer tasks.
Generative AI (GenAI) is increasingly used by students for programming explanation, debugging, and assignment support. Yet unrestricted large language model (LLM) tutors can hallucinate, contradict course policy, reveal complete solutions, and foster passive dependence. This paper presents EduGuard, a safe retrieval-augmented generation (RAG) tutoring framework for introductory programming. EduGuard integrates query understanding, instructor-approved course retrieval, pedagogical strategy selection, rubric-aware generation, claim-level verification, and overreliance control. To make evaluation provenance explicit, we construct BILearn-CS, a 600-query instructor-authored, TA-validated benchmark spanning concept questions, debugging cases, misconceptions, assignment-support requests, code-mixed Bangla-English queries, and adversarial direct-answer prompts. Moving beyond a synthetic-only benchmark, we further evaluate on a 150-query public CS50-style course-forum set and run a small controlled pilot with 10 undergraduates using a counterbalanced pre-test/post-test design. Using Meta-Llama-3.1-8B-Instruct as the primary generator, hybrid FAISS/BM25 retrieval, and DeBERTa-v3-large-MNLI as an architecturally separate verifier, EduGuard is compared against strong baselines: GPT-4o-mini Tutor, Llama Socratic Tutor, LPITutor-style RAG, RAG with rubric prompting, and RAG with same-model self-checking. On BILearn-CS, EduGuard attains the best correctness (90.1%), grounding (89.4%), and rubric alignment (90.8%), with the lowest hallucination (4.9%) and direct-answer leakage (9.8%). In the pilot, it raises immediate post-test accuracy from 68.4% to 81.2% and cuts overreliance from 38.0% to 17.0% relative to GPT-4o-mini Tutor. These results suggest safe GenAI tutoring requires not only retrieval or strong prompting, but explicit pedagogical control, evidence verification, and deployment safeguards.
S. Hossain, Ruksat Khan Shayoni, M. F. Mridha et al.· 0 citations
LLM tutoring poses a measurement problem: can a general-purpose helpfulness rubric distinguish direct answer-giving from pedagogical guidance? We audit this signal in a pre-registered study. Within each of three tutor bases, we compare conversational and pedagogical policies instantiated with the same underlying model and paired with one fixed weak simulated student. Deterministic detectors measure answer leakage and next-turn independent work. Claude Opus 4.8 is the frozen, condition-blind primary judge. After the Opus scores were fixed, GPT-5.6 Sol was prospectively specified for a post hoc robustness audit of the same 1,179 confirmatory answer-phase tutor turns under the frozen helpfulness and pedagogy rubrics. On the primary base under Opus, the policies do not differ significantly in helpfulness but are perfectly rank-separated under the pedagogy rubric (Cliff's $|\delta|{=}0.10$ vs. $1.0$). Across the two judges, pedagogy contrasts retain their direction where detected, whereas the helpfulness ordering is judge-contingent, reversing between judges on two of three bases. In an Opus-only ablation, seven primary-base policies span $2.3$ points in mean judged pedagogy within a $0.25$-point band of mean judged helpfulness. Separately, answer-revealing turns are followed by less independent student work on every base, a result that is judge-invariant by construction. In this controlled setting, general-purpose helpfulness is not a reliable pedagogy signal. Tutor evaluation should pair pedagogy-targeted rubrics with deterministic process measures.
Shuyi Fan, Boyuan Deng, Mengyu Xu et al.· 0 citations
This course teaches a repeatable, production-oriented method for debugging OpenUSD composition issues. Rather than attempting to cover every USD feature, the course focuses on how artists, technical directors, and pipeline developers can investigate real failures by tracing symptoms back to authored opinions, layer stacks, asset resolution, references, variants, edit targets, and render-facing overrides. The course is structured as a hands-on workshop that alternates between conceptual slides and guided Maya exercises. Participants first build a minimal mental model of stages, prims, layers, opinions, composition, and strongest opinion resolution. They then apply that model to a sequence of production-style case files: composed asset construction, broken-looking assets, missing scenes, references with no visible result, variant configuration failures, edit target mistakes, unexpected shot changes, and render overrides. The central pedagogy is the USD Detective Model: observe the problem, find the prim, inspect composition, identify the winning opinion, trace the source layer, and fix the correct layer. By the end of the course, attendees will be able to distinguish resolver failures from composition failures, recognize when USD is behaving correctly but the configuration is incomplete, and use edit-forwarding concepts to prevent bad opinions before they enter the pipeline.
Pallav Sharma, N. Porcino, J. Panis· Proceedings of the Special I...· 0 citations
PER has consistently demonstrated the effectiveness of small-group tutorials in helping students develop conceptual understanding and fluency, but instructor uptake is limited by resource constraints. To test the effectiveness of out-of-class tutorials using computer-generated feedback as an instructor-friendly alternative, we conducted think-aloud interviews with students in a quantum computing course who were randomly assigned to either a traditional validated small-group, pencil-and-paper tutorial on tensor products, or a solo computerized adaptation thereof. We found that while the computer-generated feedback was broadly considered useful by students, student engagement patterns were markedly different in the solo setting, with students demonstrating reluctance to use the interface's built-in help features and tending to internalize failure in unproductive ways counter to our intention of a formative learning environment. We discuss implications for curriculum design and directions for future research that may help to answer the longstanding question in PER of why tutorials work so well.
J. C. Meyer, S. Pollock, Bethany R. Wilcox et al.· 0 citations
The spread of generative AI (GenAI) in higher education has raised concerns that students offload cognitive effort to AI, earning high grades without learning. If this"GenAI substitution hypothesis"is true, grades should rise disproportionately in GenAI-susceptible courses--those relying more on assessments like take-home problem sets and essays rather than in-class exams. Substitution could also affect student satisfaction, measured here as self-reported understanding and interest in the subject, which prior research links to assessments. We test the substitution hypothesis using syllabus and administrative data from a large U.S. university (2016-2025; 138,386 students; 72,730 course offerings). We measure courses'GenAI susceptibility using a human-validated LLM pipeline to extract assessment types from syllabi, and use a differences-in-differences design comparing outcomes across courses before and after ChatGPT's release, while modeling COVID-19 pandemic effects as either persistent or transient. We find no significant differential effect of GenAI availability on grades overall or among previously lower-performing students. Effects on self-reported understanding are likewise insignificant; effects on interest are significant only assuming transient pandemic effects. Our findings temper concerns that GenAI inflates grades and reduces students'satisfaction.
J. Dumlao, Meng Wang, Zhonghan Xie et al.· 0 citations
This innovative practice full paper presents CodeStylist, a web application that provides course-standard-aware code style feedback for early undergraduate programming courses. CodeStylist addresses a common instructional gap: students are expected to follow local conventions for naming, formatting, comments, organization, and readability, but feedback on these expectations is often delayed or inconsistent. Unlike generic linters or general-purpose LLM prompts, CodeStylist supports course-specific standards, multi-file submissions, and file- and line-localized explanations intended to guide revision rather than grade correctness. We report a formative expert review with 18 instructional staff from one early undergraduate programming course. Participants explored the prototype using self-selected code artifacts and completed a survey about response quality, anticipated student use, and redesign priorities. Ratings indicated modest perceived utility but limited trust: perceived correctness averaged 60.7%, response helpfulness averaged 3.50/5, response usefulness averaged 3.33/5, and anticipated student learning averaged 2.61/5. Despite these concerns, 17/18 respondents expected students to use the tool primarily for style checking, often at least weekly. Open-ended feedback showed that respondents valued CodeStylist for making implicit course standards more visible, but were concerned about unreliable output, overreliance, and latency or cost. We interpret these findings as evidence that course-aware style feedback is promising as a pre-submission revision aid, but that future versions should combine deterministic rule checks with LLM-generated explanations, rule citations, and stronger verification support.
Ethan Dickey, L. Vento, Peter Kurto et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 17, 2026
A USAF cadet and a Lincoln Laboratory researcher found AI chatbots can help nontechnical service members produce viable software applications for their unique problems.
