Replications for RealAIGC

Abstract

RealAIGC: Towards More Realistic Evaluation of AI-Generated Code Detection in Real-World Code Repository Artifact Update (Camera Ready) In response to the latest feedback, we have substantially enhanced our replication materials in terms of dataset coverage, usability, and methodological transparency. RealAIGC++ Dataset.To mitigate the issues of data skew and limited repository coverage, we construct an updated dataset, RealAIGC++.7z. Compared with the original RealAIGC dataset, RealAIGC++ incorporates substantially more repositories, approximately doubling the repository coverage, particularly for programming languages that were previously represented by only a small number of repositories. Meanwhile, we reduce the excessive number of functions for several languages to make large-scale evaluation more manageable and computationally feasible. We also reorganize and consolidate the previous dataset structure to simplify dataset loading, model evaluation, and cross-method comparison. We strongly recommend using the latest version of RealAIGC++ for evaluation. Alternatively, researchers can use our RAP toolkit to construct customized datasets according to their own repositories, programming languages, and experimental requirements. The original RealAIGC dataset is retained for reproducibility purposes but is available only upon specific request. RAP Toolkit.We provide RAP, a visual toolkit that supports the major stages of the experimental workflow, including dataset construction, model training, and model testing. The actively maintained RAP source code is available at [cyf-cyf-sec/RAP]. For reproducibility and archival purposes, we additionally provide a snapshot of the RAP materials described below. Because RAP is under active development, the archived version may differ slightly from the latest maintained version. The archived RAP package contains four components: RAP_Usage_Document.pdf: A detailed document describing how to install and use RAP. RAP_Video.mp4: A demonstration video illustrating the main workflow and functionality of RAP. RAP_Source_Code.7z: The archived RAP source code, including the Dockerfile required to reproduce the execution environment. RAP_Docker_Image.7z: A packaged Docker image of RAP for direct deployment and reproduction. Method Selection and Literature Collection.To improve the transparency and reproducibility of our method-selection process, we provide Review4AIGC_Detection.xlsx, which contains the literature collected and reviewed during our survey of AI-generated code detection methods. This file documents the candidate studies considered when selecting the detection methods evaluated in our experiments. Detailed for RealAIGC++ Dataset RealAIGC++ is a dataset collected from real-world open-source repositories. Dataset Overview - 48 open-source repositories- 8 programming languages- 90 JSONL dataset files- 50,256 pull requests- 146,118 commits- 470,149 source files- 1,408,961 function-level records- Approximately 2.70 GiB of data Pull requests, commits, and source files are globally deduplicated using `(repository, ID)` as the key. | Language | Repositories | JSONL Files | Pull Requests | Commits | Source Files | Functions || :-- | --: | --: | --: | --: | --: | --: || C | 15 | 15 | 9,384 | 20,766 | 49,645 | 133,168 || C# | 11 | 11 | 11,531 | 35,883 | 121,057 | 352,680 || C++ | 11 | 11 | 8,372 | 23,692 | 60,364 | 168,109 || Go | 8 | 8 | 5,462 | 12,922 | 40,649 | 115,937 || Java | 5 | 5 | 1,090 | 1,756 | 8,365 | 28,978 || JavaScript | 8 | 8 | 2,462 | 5,359 | 15,516 | 50,849 || Python | 14 | 14 | 8,707 | 24,503 | 61,326 | 178,705 || TypeScript | 18 | 18 | 12,956 | 35,701 | 113,227 | 380,535 | A pull request, commit, or source file may contain multiple programming languages. Therefore, summing the per-language rows may produce totals greater than the globally deduplicated counts. Directory Structure and File Naming ```textRealAIGC++/ ├── airbytehq_airbyte_Python.jsonl ├── microsoft_onnxruntime_C++.jsonl └── ...``` Dataset files follow this naming convention: ```text _ _ .jsonl``` For example, `microsoft_onnxruntime_C++.jsonl` contains C++ function records from the `microsoft/onnxruntime` repository. Data Format All data is stored as uncompressed JSON Lines. Each line is an independent JSON object containing the following fields: | Field | Description || :-- | :-- || `id` | Record identifier within the current dataset file || `pr_id` | Identifier of the pull request containing the function || `commit_id` | Identifier of the commit containing the function || `diff_id` | Identifier of the diff containing the function || `ext` | Extension of the original source file || `language` | Programming language || `file_id` | Identifier of the original source file || `code` | Function or function-level code snippet || `start_line` | Starting line of the snippet in the original file || `end_line` | Ending line of the snippet in the original file || `agent` | Code-source category; current values are `codex`, `copilot`, `cursor`, `devin`, and `human` | Example: ```json{"id": 982, "pr_id": 3565067321, "commit_id": 126, "diff_id": 518, "ext": ".js", "language": "JavaScript", "file_id": 341, "code": "(id) => !registryKeys.has(id)", "start_line": 10, "end_line": 10, "agent": "copilot"}``` Repository List | No. | Repository | No. | Repository | No. | Repository || --: | :-- | --: | :-- | --: | :-- || 1 | airbytehq/airbyte | 17 | giselles-ai/giselle | 33 | mochilang/mochi || 2 | AMICI-dev/AMICI | 18 | gofiber/fiber | 34 | near/nearcore || 3 | bruin-data/bruin | 19 | go-vikunja/vikunja | 35 | neondatabase/website || 4 | calcom/cal.com | 20 | jdereg/java-util | 36 | onlook-dev/onlook || 5 | carverauto/serviceradar | 21 | jdereg/json-io | 37 | openai/codex || 6 | celestiaorg/celestia-core | 22 | keepassxreboot/keepassxc | 38 | OpenHFT/Chronicle-Core || 7 | chrxh/alien | 23 | KomodoPlatform/komodo-wallet | 39 | OpenHFT/Chronicle-Wire || 8 | commaai/panda | 24 | liam-hq/liam | 40 | oven-sh/bun || 9 | crewAIInc/crewAI | 25 | microsoft/ebpf-for-windows | 41 | reflex-dev/reflex-web || 10 | DannyMac180/meta-agent | 26 | microsoft/msquic | 42 | rsyslog/rsyslog || 11 | dotnet/aspire | 27 | microsoft/onnxruntime | 43 | ryokun6/ryos || 12 | dotnet/aspnetcore | 28 | microsoft/react-native-windows | 44 | shader-slang/slang || 13 | dotnet/maui | 29 | microsoft/testfx | 45 | wieslawsoltes/Dock || 14 | dotnet/runtime | 30 | microsoft/typescript-go | 46 | wieslawsoltes/Svg.Skia || 15 | eigenwallet/core | 31 | microsoft/vscode | 47 | wieslawsoltes/Xaml.Behaviors || 16 | forntoh/LcdMenu | 32 | mlflow/mlflow | 48 | wolfSSL/wolfssl | Baseline Methods | Method | Type | Release | Paper || :-- | :-- | :--: | :-- || C Method | Feature engineering | 2023 | || Python Method | Feature engineering | 2024 | || Feature Method | Feature engineering | 2023 | || GPTSniffer | Deep learning | 2024 | || UniXCoder | Deep learning | 2025 | || CodeT5+ | Deep learning | 2025 | || CodeGPTSensor | Deep learning | 2025 | || ModernBERT | Deep learning | 2025 | || logp(x) | Perplexity based | 2025 | || Entropy | Perplexity based | 2025 | || Rank | Perplexity based | 2025 | || LogRank | Perplexity based | 2025 | || LRR | Perplexity based | 2025 | || NPR | Perplexity based | 2025 | || FastDetectGPT | Perplexity based | 2024 | || DetectCodeGPT | Perplexity based | 2025 | |