Evaluation of a Retrieval-Augmented AI Audit Assistant for Financial Document Compliance Screening and Accounting Anomaly Detection

Abstract

Objective. Structured numeric review and unstructured textual audit evidence remain handled by largely disconnected systems, leaving a persistent gap between flagging an anomalous transaction and grounding that finding in applicable business or regulatory text. This study developed and evaluated a retrieval augmented AI audit assistant organized around a five stage Detect, Link, Retrieve, Verify, Explain pipeline, designed to integrate structured anomaly detection with citation grounded evidentiary reasoning. Methods. A three phase, sequential comparative design was adopted, escalating from entry level detection across 10,842 journal entries, to evidence level retrieval across 512 gold annotated question evidence pairs drawn from a 154 document corpus, to case level evaluation across 100 objective and 20 expert blind reviewed audit cases. Four classifiers were compared under traditional, embedding, and fused feature configurations, four retrieval strategies were compared, and four end to end system configurations were assessed against reference audit opinions and blinded expert ratings. Results. Feature fusion improved detection performance across most classifiers, hybrid retrieval with reranking achieved the strongest evidence grounding, and the fully integrated system reached 84.0 percent objective accuracy with a 9.0 percent hallucination rate, outperforming all ablated configurations while reducing review time by roughly 54 percent. Conclusion. Integrating structured risk detection with grounded retrieval and constrained generation yields audit support that is more accurate, traceable, and efficient than any single module alone, offering a reusable, accountability preserving template for embedding LLM reasoning into audit workflows.