PurifAI: Detecting and Fixing Search-Induced Distortions in Web-Augmented LLMs

Abstract

As Large Language Models (LLMs) increasingly serve as interfaces for proprietary data (e.g., enterprise knowledge bases, legal statutes), ensuring their fidelity to trusted internal information is paramount. While integrating real-time web search can enhance model utility, it introduces a critical vulnerability: the ingestion of conflicting, misleading, or hallucinated content from the open web can override the model's adherence to its verified internal knowledge. We define this failure mode as search-induced distortion, a significant risk in high-stakes domains where the internal knowledge base serves as the absolute ground truth. To address this challenge, we present PurifAI, a proactive, model-agnostic, cache-level purification system designed for safety- and compliance-sensitive deployments. Rather than serving as a general fact-checking engine, PurifAI is explicitly designed to preserve knowledge alignment with a pre-defined trusted knowledge core. It automatically generates diagnostic probes from trusted documents to identify and neutralize searched web content that conflicts with the canonical internal source before such content distorts the LLM's responses. Extensive evaluations on news, encyclopedic, and legal domains show that PurifAI effectively improves alignment with the trusted core, identifying and blocking distortive content at the source and achieving repair success rates above 70% across mainstream LLMs. Our work offers a practical safeguard for enterprises and other high-stakes applications seeking to integrate web-augmented LLMs without compromising policy consistency and trusted knowledge integrity.