Sensitive Data Detection in Documents with LLMs

Abstract

Detecting and extracting sensitive information from documents is essential for privacy and regulatory compliance. Existing approaches either require training on large labeled datasets or rely on brittle, costly-to-curate pattern matching, while Large Language Models (LLMs) offer a promising alternative. We present a systematic evaluation of several proprietary and open-source LLMs for sensitive entity extraction from documents. Because large datasets of completed forms containing personal information are unavailable, we also introduce a form-filling pipeline that uses vision-capable LLMs to label form fields, generate realistic synthetic personas, and fill real blank forms, enabling reproducible evaluation across diverse layouts. Evaluating on these forms and the public RVL-CDIP dataset, we find performance is uneven across entity types—and that LLMs fall short of a simple pattern-based baseline on Social Security numbers.