From Trace to Line: An Empirical Study of What Drives LLM-Based OSS Vulnerability Localization

Abstract

Large language models (LLMs) show promise for vulnerability discovery, yet prevailing methods often inspect code in isolation, struggle with long project contexts, and produce coarse function- or file-level predictions that offer limited guidance for targeted patches. This paper presents a controlled empirical study of what factors help LLM agents perform project-level, line-level vulnerability localization from runtime failure evidence. We introduce T2L (Trace-to-Line), a reproducible research framework that narrows repository-scale code into candidate vulnerable lines through AST-based chunking, structured diagnostic information collection, and evidence-guided refinement. To support systematic evaluation, we build T2L-ARVO, an expert-verified 50-case benchmark spanning five crash families in real-world open-source projects, and T2L-Agent, a modular reference agent for studying localization workflows under equal per-case budgets. Across ten models and controlled ablations, we find that structured diagnostic analysis, coarse-to-fine refinement, and multi-hypothesis exploration all improve trace-to-line localization. Across these workflow-level configurations, the largest observed increase in line-level localization reaches 48.9 percentage points over the corresponding baseline, while increasing reasoning budget or adjusting sampling temperature yields limited and inconsistent benefits. Overall, our study provides an empirical reference for designing LLM-agent workflows for complex project-level localization tasks that move from runtime failure evidence to precise vulnerable-line hypotheses.