An Analysis of LLM Performance on Introductory C++ Programming Assignments

Abstract

Background and Context. Large Language Models (LLMs) have become widely accessible to students in introductory programming courses [1], yet limited research evaluates their performance on authentic assignments with pedagogical constraints such as restricted language features and course-specific conventions. Existing benchmarks target capabilities that do not reflect real coursework: function synthesis [2], competition programming [4, 7], repository-level bug fixes [5], and class-level Python generation [3]. Novice programmers tend to provide minimal context and rely on single-shot interactions [1, 6], motivating our empirical experimentation with holistic prompting alongside a compositional strategy. Objectives. This study investigates (1) how current LLMs perform on introductory C++ programming assignments in terms of functional correctness and response behavior, and (2) the extent to which code generated by different models exhibits structural similarity, with implications for academic integrity detection. Method. Two multi-class C++ programming assignments were selected from a CS2 course covering dynamic memory management, class design, and complex data structures involving tree traversal and reference-counted shared ownership. We generated 62 solutions to the two assignments using models in the Claude, GPT, Gemini, and DeepSeek families. The same autograder suite used to grade student code was applied to LLM-generated solutions. For Claude and GPT, results were further compared against 1,436 student submissions from a prior course offering. We also measure the probability that its generated solution fully passes all functional tests on the first attempt using the Pass@1 metric. Finally, the winnowing algorithm is applied to evaluate structural similarity across LLM-generated solutions. Findings. On the first submission compiling with only accepted libraries, autograder scores ranged from below 60% to 94%, bimodal on the complex assignment and normal on the simpler one, while Pass@1 ranged from 0–7% and 33–100%, respectively. Enabling “thinking” mode yielded no improvement for GPT and hurt Claude Sonnet 4.5, which scored 17% lower on average. LLMs reliably handled memory management and constructors, but struggled with the test cases students found hardest and consistently failed C++20 style standards. Models differed in response behavior: Claude followed compositional instructions precisely; GPT frequently implemented more than requested; Gemini embedded reasoning and citations into code comments; DeepSeek produced verbose self-dialogue with multiple code versions per response. Within-family structural similarity substantially exceeded cross-family similarity, and all models converged on near-identical solutions upon reaching a perfect score. Implications. LLM performance on CS2 assignments depends more on task complexity and prompt strategy than on model choice, suggesting that educators should design assignments with constraints that resist one-shot automation and encourage iterative, scaffolded engagement. The high within-family structural convergence of LLM-generated code challenges the sufficiency of existing similarity-based academic integrity tools, necessitating new detection strategies that account for characteristic patterns across model families.