Curriculum Learning for Fine-Tuning Code-Review Language Models: A Case Study in Diagnosing and Correcting Fabrication in Vulnerability Explanation

Abstract

Automated code-review systems can identify potential defects, but they often provide limited explanations for developers. This study investigates a curriculum-based approach for fine-tuning CodeT5+ 770M to generate code-review and vulnerability explanations for Python and JavaScript. Training was divided into three stages: code understanding, semantic review, and vulnerability explanation. Rehearsal examples from earlier stages were included during later training to reduce forgetting. We also compared this staged approach with an earlier mixed-phase setup that trained on all tasks together, to see whether staging changed how the training process behaved. During Stage 3, we found that using case-specific CVE descriptions as targets led the model to generate unsupported package names, versions, and repository references. We replaced these targets with CWE-level descriptions and retrained the model. In a manual review of 20 outputs, the revised model did not produce the same type of unsupported specific details. BLEU increased from 7.54 to 34.43, and ROUGE-1 increased from 0.2765 to 0.4180. However, the revised model still confused CWE categories: exact category agreement was 27.3% after excluding samples with incomplete ground-truth labels, and Cross-Site Scripting was predicted more often than its true frequency. We also tested whether increasing the Stage 2 target length from 192 to 512 tokens improved later Stage 3 performance. Across six epochs, the average validation-loss difference was +0.0008, and the final Stage 3 metrics showed no meaningful change. These results show that target design and manual inspection are important when evaluating generated vulnerability explanations, and that negative results, like the target-length test here, are worth reporting alongside positive ones.