Do Pre-Trained Code Models Add Value Beyond Software Metrics in Class-Level Defect Prediction? An Empirical Study of Input Coverage, Long-Code Aggregation, and Cross-Version Generalization
Aug 2026· Electronics· Vol 15, pp. 3544· 0 citations· 26 references
TL;DR
Findings indicate that pre-trained code models should be evaluated with explicit input-coverage reporting, chronological validation, strong metric baselines, and incremental-value testing.
Abstract
Pre-trained code models are increasingly used in software engineering, yet their incremental value beyond traditional software metrics for future-version class-level defect prediction remains unclear. This study evaluates CodeBERT, GraphCodeBERT, and CodeT5 using 16,237 class-version instances from six open-source Java systems and 11 chronological train–validation–test splits. We analyze model-specific input coverage and long-code representations and test whether learned code features add value beyond metric and size controls. Overflow affected 50.7% of instances for CodeBERT and GraphCodeBERT and 39.0% for CodeT5. Defective instances overflowed more often than clean instances, and this association remained after adjustment for size and project-version effects. Long-code strategies yielded small and inconsistent gains, none of which survived Holm correction. The best code-only model achieved a mean Matthews correlation coefficient (MCC) of 0.315, compared with 0.451 for metric-based Random Forest. Fusion produced no robust incremental gain. Multi-seed fine-tuning improved mean MCC for all encoders, but none of the paired gains remained statistically significant after Holm correction, and the best fine-tuned model remained below the metric baselines. These findings indicate that pre-trained code models should be evaluated with explicit input-coverage reporting, chronological validation, strong metric baselines, and incremental-value testing.
Investigating LLMs as metric-driven refactoring assistants rather than code generators suggests that while LLMs are valuable assistants for structural improvement, their interventions require careful monitoring to avoid unintended trade-offs.
Tindwende Thierry Sawadogo, Fadel Touré· International Conference on...· 0 citations
A transformer-based classification framework that fine-tunes CodeBERT on a balanced dataset of 25,000+ bug-report segments drawn from four Apache projects is proposed and CodeBERT is the best-performing transformer in this comparison.
Kanwal Naz, Imran Shafi, M. Z. Mehmood et al.· Computers· 0 citations
This study investigates multiple experimental configurations based on CodeBERT, GraphCodeBERT, and CodeT5, including direct transformer-based classification, sliding window processing for long code sequences, and hybrid approaches combining transformer embeddings with classical machine learning classifiers.
These findings validate cognitive theory for explainable, actionable, and interpretable safety-critical defect prediction, laying empirical groundwork to evaluate analogous issues in LLM-generated code through the behavioral study of AI.
Carlos Andrés Ramírez Cataño, Makoto Itoh· International Conference on...· 0 citations
Automated code-review systems can identify potential defects, but they often provide limited explanations for
developers. This study investigates a curriculum-based approach for fine-tuning CodeT5+ 770M to generate code-review and
vulnerability explanations for Python and JavaScript. Training was divided into three stages: code understanding, semantic
review, and vulnerability explanation. Rehearsal examples from earlier stages were included during later training to reduce
forgetting. We also compared this staged approach with an earlier mixed-phase setup that trained on all tasks together, to see
whether staging changed how the training process behaved. During Stage 3, we found that using case-specific CVE descriptions
as targets led the model to generate unsupported package names, versions, and repository references. We replaced these targets
with CWE-level descriptions and retrained the model. In a manual review of 20 outputs, the revised model did not produce the
same type of unsupported specific details. BLEU increased from 7.54 to 34.43, and ROUGE-1 increased from 0.2765 to 0.4180.
However, the revised model still confused CWE categories: exact category agreement was 27.3% after excluding samples with
incomplete ground-truth labels, and Cross-Site Scripting was predicted more often than its true frequency. We also tested
whether increasing the Stage 2 target length from 192 to 512 tokens improved later Stage 3 performance. Across six epochs, the
average validation-loss difference was +0.0008, and the final Stage 3 metrics showed no meaningful change. These results show
that target design and manual inspection are important when evaluating generated vulnerability explanations, and that negative
results, like the target-length test here, are worth reporting alongside positive ones.
Unknown authors· International Journal for Re...· 0 citations
The results indicate that traditional ML models, especially random forest and extra trees, are still very effective for metric-based defect prediction, while DL and multi-modal approaches need to be fed with richer software artifacts to reach their full potential.
Amro Mohammad Abed Alfattah Abdin, Mohanad Alayedi, Ahmad M. Jaradat· Journal of Supercomputing· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
