ConRAC-SE: Evidence-Agreement Routing for Cost-Aware Retrieval-Augmented Security Classification

Abstract

Enterprise document security-level classification must balance rare high-risk recall, precision, and inference cost. Static ML pipelines lose recall as sensitivity patterns drift, whereas retrieval-augmented classification (RAC) grounds predictions in auditable precedents but still struggles on boundary cases, and verifying every document with an LLM is expensive and can inject variance when the evidence already agrees. We introduce ConRAC-SE, a deterministic evidence-agreement routing controller for cost-aware RAC: it commits top-1/top-2 evidence-consensus cases without an LLM, verifies only non-consensus cases with a small LLM against a top-3 evidence vote, and escalates only verifier–evidence disagreements to a high-capacity LLM. On the WikiLeaks and DISC security corpora, RAG-style RAC always-LLM decoding helps when the retrieval backbone is weak but regresses when it is strong; under a limited call budget on the strong backbone, ConRAC-SE instead improves quality consistently while lowering average latency and LLM cost. LLMs are thus most useful here as residual adjudicators rather than always-LLM decoders, and the routing additionally surfaces the ambiguous tail for review.