Large Language Models (LLMs) are being widely used for coding, with reports indicating that AI now generates an increasing share of production code. Studies show that LLMs can significantly improve developer productivity, yet they still struggle with more complex coding tasks. Just as understanding error modes in human-written code has been central to improving software quality, identifying and characterizing the errors in LLM-generated code is critical for setting realistic expectations and designing mitigation strategies. Prior research has been limited in scope, often focusing on a single language, a small number of problems, or a limited selection of models. As a result, there is still no comprehensive understanding of which errors are common and which are specific to certain models or languages. To address these gaps and develop a deeper understanding of the quality of LLM-generated code, we analyzed a corpus of 86,726 code samples that contained compilation or runtime errors. These samples were generated by seven LLMs across four compiled languages. We classified errors by their underlying causes using an LLM, manually validated these classifications, and performed a comparative analysis. This labeled data is then used to measure error prevalence by model, language, and problem difficulty, to identify common error patterns. Results show that, although error types vary strongly across languages and models, even the largest models frequently make simple mistakes. We also observe that generated code often omits basic input validation or memory-safety checks, which can lead to overflows, resource exhaustion, or other reliability/security issues.
Rodrigo Pato Nogueira, Marco Vieira, João R. Campos· 0 citations
Large Language Models (LLMs) are increasingly being used in everyday software engineering tasks, particularly in automated code generation. Despite their widespread adoption, these models remain far from perfect, making systematic and fair evaluation essential to understand their strengths and limitations. In the context of code generation, existing benchmarks are limited: they often target a single programming language and rely primarily on unit test outcomes, while overlooking other critical dimensions such as the overall quality of the generated code and its closeness to a valid solution. To address these gaps, we introduce PROBE, an extensible benchmark framework that, unlike prior work, establishes a systematic structure built on diverse and well-defined metrics, representative workloads, varied prompt templates, and a robust experimental procedure. In practice, the code generated by the LLMs is evaluated along three complementary dimensions: functional correctness, proximity to valid solutions, and code quality, enabling a comprehensive assessment of performance. We use PROBE to evaluate four open-source and two proprietary models under three prompting strategies across five programming languages. We further complement this analysis with a study of common errors in the code and provide concrete examples, offering clearer insight into where LLMs tend to struggle. Our findings show that, while LLMs achieve promising results, they struggle with harder problems and, in the case of smaller models, with programming languages that have fewer available resources for training, and they often fail due to fundamental and easily avoidable errors that underscore the unreliability of automatically generated code.
Rodrigo Pato Nogueira, Marco Vieira, João R. Campos· Empirical Software Engineeri...· 1 citation