LZX-zx-em/CPP-Code-Dataset: C++-Eval Dataset v1.1.0
Abstract
C++-Eval: A C++ Evaluation Benchmark for Large Language Models C++-Eval is a comprehensive benchmark for evaluating the C++ code generation capabilities of Large Language Models (LLMs), with a focus on complex memory management and industrial-grade software engineering requirements. Dataset Statistics 200 multi-level C++ programming tasks, each with 20 human-authored, high-coverage test cases (boundary inputs, invalid inputs, exception handling) 6 engineering scenarios: basic data structures, classic algorithms, numerical computation, machine learning preprocessing, image and point cloud processing, and advanced object-oriented features 3 difficulty levels (basic / intermediate / advanced), classified by cognitive complexity and lines of code (LoC) Coverage of core C++ features and paradigms: OOP, generic programming, explicit memory management, and modern C++11 mechanisms Data Structure CPP.jsonl — one JSON object per line; each task includes the task description, complete function declarations (interface contract), and a test suite of ~20 cases Construction & Decontamination Task concepts curated from public platforms (LeetCode, Codewars, GitHub), followed by semantic-based manual verification Interface contract and signature refactoring to break verbatim prompt matching against public repositories Execution-based decontamination: all solutions compiled with g++ -std=c++11 and -fsanitize=address (AddressSanitizer) to capture low-level memory faults Evaluation Protocol Strict zero-shot prompting (see Appendix A.1 of the paper) Metrics: task-level error probability, compilation success rate, and task-level pass rate, with 95% Wilson score confidence intervals
