Separating Secrets from Placeholders: A Hybrid CNN-CodeBERT Framework for Three-Class Credential Leakage Detection

Abstract

# Separating Secrets from Placeholders: A Hybrid CNN-CodeBERT Framework for Three-Class Credential Leakage Detection ## Overview This repository contains the code and dataset for our 3-class credential leakage detection framework, which distinguishes between: - **Class 0:** No Leak- **Class 1:** Genuine Leak- **Class 2:** Placeholder/Weak Leak --- ## Recommended Experiment Order 1. **`main_method/`** — Train and evaluate the proposed hybrid model (4 seeds). This is the primary experiment.2. **`ablation/`** — Run ablation study to evaluate each architectural component (seed 42 only). Requires the same train/val/test splits as the main method.3. **`lolo/`** — Run Leave-One-Language-Out cross-validation using the full dataset. Run once per language (10 runs total).4. **`baselines/keysentinel/`** — Run KEYSENTINEL baseline. Use `--config fair` for the results reported in the paper.5. **`baselines/passfinder/`** — Run PassFinder baseline. All experiments are independent and can be run in any order. The recommended order above follows the paper's research questions (RQ1 → RQ5). --- ## Repository Structure ```├── data/ # Dataset (see data/README.md)│ ├── Sanitized_CCLD_dataset.csv│ └── splits/│ ├── train.csv│ ├── val.csv│ └── test.csv│├── main_method/ # Proposed hybrid model (RQ1, RQ2)├── ablation/ # Ablation study (RQ3)├── lolo/ # Leave-One-Language-Out evaluation (RQ4)├── baselines/│ ├── keysentinel/ # KEYSENTINEL baseline│ └── passfinder/ # PassFinder baseline``` --- ## Experiments | Folder | Description | Data Used ||--------|-------------|-----------|| `main_method/` | Proposed CharCNN + CodeBERT + Adapter model, 4 seeds | train/val/test splits || `ablation/` | 6 architectural variants, seed 42 | train/val/test splits || `lolo/` | Leave-One-Language-Out cross-validation | Full dataset || `baselines/keysentinel/` | KEYSENTINEL adapted for snippet-level 3-class task | train/val/test splits || `baselines/passfinder/` | PassFinder adapted for snippet-level 3-class task | train/val/test splits | Each folder contains its own `README.md` with setup and run instructions. --- ```bibtex@article{baby2026separating, title={Separating Secrets from Placeholders: A Hybrid CNN-CodeBERT Framework for Three-Class Credential Leakage Detection}, author={Baby, Maksuda Bilkis and Shah, Khushika and Liang, Naiyue and Zhang, Lei}, journal={arXiv preprint arXiv:2605.31520}, year={2026}}``` ## Acknowledgements Parts of the writing and implementation in this prototype were developed with assistance from Large Language Models (LLMs). These tools were used solely as helpers for tasks such as brainstorming, drafting code snippets, and refining text. All core ideas, research design, experimental decisions, and contributions in this work originate from the author(s). All code and experiments were executed, inspected, and validated by the author(s). Any errors or omissions remain the responsibility of the author(s).