DAST-AI: Reducing False Positives in API Security Testing via LLM-based Semantic Analysis

Abstract

Dynamic Application Security Testing (DAST) is a critical methodology for identifying vulnerabilities in web applications. However, traditional DAST tools suffer from high false positive rates, causing alert fatigue among security teams. This issue is particularly acute in RESTful APIs regarding authorization flaws—such as Broken Object Level Authorization (BOLA) and missing authentication—where distinguishing between a legitimately public endpoint and a vulnerable one requires semantic context. This paper introduces DAST-AI, a hybrid security testing framework that integrates Large Language Models (LLMs) as a post-execution semantic oracle to analyze HTTP response payloads and determine whether sensitive data is improperly exposed. The evaluation is scoped to the zero-credential black-box scenario—testing APIs without any prior authentication—applicable to external security assessments and unauthenticated CI/CD pipelines. A controlled ablation study against two deliberately vulnerable benchmarks—OWASP Juice Shop (Node.js, 68 endpoints) and VAmPI (Python/Flask, 12 endpoints)—isolates the contribution of the LLM semantic oracle: removing it yields a syntax-based baseline (F1=0.49 on Juice Shop; F1=0.857 on VAmPI), while adding it produces DAST-AI (F1=0.80; F1=0.923), reducing false positives by 86.4% and 50.0% respectively, at a negligible cost of $0.000037 per endpoint. An authenticated reference comparison—simulating the ZAP Access Control Scanner methodology with valid credentials—scores F1=0.28 on Juice Shop and F1=0.857 on VAmPI (identical to the unauthenticated baseline), demonstrating that credentials alone do not resolve the problem: semantic reasoning about response content is the key differentiating factor.