What Does a Missing Exclusion Clause Cost? Measuring the False-Positive Burden of LLM-Generated Detection Rules

Abstract

Large language models now write detection rules, and the work evaluating them asks whether the rules catch attacks. It does not ask what they cost on a quiet day. That cost sits in the exclusion clause the "alert when X, except when Y" half of a rule, which encodes local knowledge of what is benign here. Prior work reports that generated rules reach only 8.9% exclusion parity with human rules, but it compares rule text and never runs the rules, so the operational cost of that gap is unknown. We run them: 626 SigmaHQ rules, 384,191 benign Windows events from a public seven-host corpus, and an open-source engine. Deleting the exclusions multiplies alert volume by 3.6x to 29x on telemetry that contains no attack. Nine LLM configurations across five model families, writing rules for the same requirements from the same threat description, all raise more benign alerts than the human rule 1.4x to 34.6x on an identical requirement set. The quietest arms are quiet for the wrong reason: they stay silent on 40–67% of requirements, which on an attack-free corpus is indistinguishable from broken. Quietness and silence rank as near mirror images (Spearman rho = -0.852, n = 9, p < 0.01), so a false-positive rate reported without a silence rate is not interpretable and no prior work reports one. The exclusions models do write are guesses shaped like knowledge: they name what a threat report implies and miss what only operating a fleet teaches. All rules, data, and code are public, and the study re-runs end to end in under an hour on a laptop.