An in-depth investigation into the performance of different AI models in secret detection found that the choice of feature set greatly influences the performance in secret detection, and that prompt engineering reduces false predictions significantly, yet human intervention remains necessary.
Abstract
Numerous tools have been developed to uncover secrets and credentials in source code. However, they often rely on predefined rules that yield a high rate of false positives. We conduct an in-depth investigation into the performance of different AI models in secret detection. In non-generative models, we found that the choice of feature set greatly influences the performance in secret detection. These models demonstrated the best performance using the context feature, i.e., code surrounding the secret value. Particularly, CodeBERT outperforms other models with an MCC and F1-scores of 88% and 89%, respectively. When we applied the models to an unseen dataset of secrets, the top models were Random Forest and CodeBERT, achieving an MCC score of around 65%. In generative models, we observed moderate performance (a maximum MCC of 63%) and a high number of false positives. The investigation of incorrect predictions revealed that many are within test-path files and that LLMs tend to be cautious, flagging potential secrets to promote secure coding practices. We noted that prompt engineering reduces false predictions significantly, yet human intervention remains necessary.
Hardcoded secrets in source code pose critical security vulnerabilities which can be easily exploited by malicious adversaries. Existing regex-based detection approaches suffer from fundamental limitations, as secrets often lack identifiable patterns, resulting in poor precision and recall. Recent studies have explored context-aware detection methods, as surrounding code can reveal the purpose of candidate strings. However, these methods confront three key challenges: (1) obfuscation robustness where models over-rely on easily obfuscated identifiers, (2) cross-language generalization difficulties due to uneven training data distribution, and (3) lengthy and noisy context that introduces excessive irrelevant tokens and slows inference. We observe that strings serve as a critical information source for code semantics, offering superior contextual density, obfuscation robustness, and language independence. Based on this insight, we propose StringGroup, a novel context extraction algorithm that mines strings surrounding potential secrets. By introducing a relatively simple modification to existing patterns that narrows the analysis specifically to string literals, the method achieves significant gains. With only 33.2% of the original context, it preserves over 80% of semantic information and significantly improves the signal-to-noise ratio for secret detection. We further design a context-aware secret detection tool, Secretron, based on StringGroup methods and Transformer model. Evaluation on the SecretBench dataset demonstrates high accuracy with 98.74% F1-score and strong robustness under obfuscation and cross-language scenarios, outperforming state-of-the-art LLM-based baselines. We deploy our tool in real-world environments and successfully detect 48 previously unknown secret keys from 26 applications, demonstrating the practical effectiveness of our approach.
Zhengdong Huang, Kevin Li, Jinqiu Yang et al.· 0 citations
This work presents Replicant, a deep reinforcement learning framework that learns the realistic task of evasion under a strict label-only black-box threat model and demonstrates that learning the task of evasion not only results in stronger attack performance but provides a better signal for hardening malware detectors.
Shae McFadden, Ilias Tsingenopoulos, Mario D'Onghia et al.· 0 citations
Leakage enables two practical attacks: a trained classifier that infers semantic predicates about user memories from routine natural-language outputs, and an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.
Jaiden Fairoze, Neal Mangaokar, Kamalika Chaudhuri et al.· 0 citations
The application of LLMs for detecting malicious PowerShell scripts and producing human-interpretable explanations for their classification decisions are investigated, showing that LLMs are capable of identifying and explaining malicious PowerShell scripts, although performance varies across different models.
Meng Wang, Emma Topolovec, B. Arana et al.· 0 citations
No method reliably surfaces every backdoor, but a broadly jailbreakable model is itself anomalous, a useful signal even when the exact trigger is not recovered, allowing defenders to filter jailbreaks.
A. Hughes, N. Xing, Collin Francel et al.· arXiv.org· 0 citations
CoGate is proposed, a confidence-gated co-decoding approach that controls the expert's influence on the co-decoding process based on its confidence, and outperforms existing co-decoding methods (CoSec+) across multiple benchmarks.
Minghao Hu, Lannan Luo, Allen G. Roush et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
