The findings indicate that off-the-shelf LLMs can effectively perform a broad range of functionalities and tasks in privacy policy and regulation analysis that previously required specialized tools.
Abstract
The advent of LLMs has significantly changed the research on privacy policy and data compliance analysis by enabling tasks that previously required specialized, domain-specific tools. However, it remains unclear to what extent LLMs can truly replicate the diverse functionalities, and the wide range of methodologies and analysis offered by prior work. In this paper, we conduct the first systematic evaluation of whether off-the-shelf LLMs can replace specialized privacy analysis tools. We study six representative tools spanning three major functionalities: contradiction detection, regulatory compliance analysis, and privacy policy summarization and aggregation, and across three intermediate tasks: structured data extraction using tuples, Semantic Role Labeling (SRL) and manual privacy policy labeling. We compare the performance of two state-of-the-art LLMs (GPT-5.2 and Gemini-2.5 in various configurations) against the tools by directly prompting the models to perform corresponding functionalities and tasks on a custom dataset of 10 privacy policies, allowing us to assess whether off-the-shelf models can produce tool-specific functionalities without further engineering or domain-specific training, major limitations in prior work. Our results show that LLMs consistently match or exceed the capabilities of existing tools across the functionalities. In manual labeling of first-party collection entities, LLMs achieved an average precision of 81.8% and recall of 70.9%, while for labeling of third-party sharing entities, they achieved an average precision of 91.4% and recall of 70.8% compared to the OPP-115 dataset. Overall, our findings indicate that LLMs can effectively perform a broad range of functionalities and tasks in privacy policy and regulation analysis that previously required specialized tools.
System{, a middleware that intercepts tool calls and rewrites their arguments to the minimum data necessary for tool functionality, combining schema-aware necessity analysis with four operations: removal, generalization, substitution, and truncation is presented.
Privacy policies may contain internal contradictions in which commitments are undermined by practices documented elsewhere in the same policy. We operationalize this phenomenon, privacy washing, through a four-stage pipeline: statement extraction, compatibility filtering and natural language inference screening, multi-model judge verification, and thematic analysis, with contradictions confirmed by majority vote of a three-model LLM panel. Applied to two corpora of website privacy policies, 123 collected in 2026 (OPPT) and 115 collected in 2015 (OPP-115), the pipeline finds the same category patterns recurring across the 11-year gap, with third-party sharing contradictions the majority of confirmed cases in each primary run, consistent with structural factors in policy composition rather than necessarily intentional deception. At least one panel-confirmed contradiction appears in 12.2% of OPPT companies (15/123; 9.8% excluding legacy pairs) and 36.5% of OPP-115 companies (42/115). A stability re-run seven months later, with a fully separated configuration (new extraction models, judges from three Chinese providers absent from both corpora, matched filters, no judge-submission similarity threshold), reproduces the OPPT prevalence under the original protocol (13.0% vs. 12.2%), finds sub-threshold pairs confirm at rates of the same order as those above (raising prevalence to 20.3% and 40.9%), and shows the third-party majority is panel-sensitive while the recurrence of the same category pairs is not. Two caveats govern all figures: panel verdicts are not validated against human expert judgment, so precision is unknown and prevalence figures are lower bounds; and the two primary runs used different filter configurations, so their prevalence difference is not interpretable as a corpus or era effect (the matched re-run reduces the gap to roughly twofold but does not eliminate it).
Large Language Models (LLMs) such as ChatGPT, Gemini, Claude, etc. have become integral tools for technical writing, software development, and communication. However, these systems, despite privacy assurances, still expose a significant privacy risk. User prompts often contain personally identifiable or proprietary information that is transmitted in plain-text to external providers, where it may be logged or accessed despite opt-out policies. This research aims to address this gap by designing a lightweight local encryption middleware that acts as a privacy firewall between users and remote AI APIs. Before a message is sent, the middleware automatically detects sensitive entities (Personally Identifiable Information or PII) and replaces them with typed placeholders such as [PERSON_1] or [ORG_1]. A local mapping (with optional authenticated encryption at rest) links these placeholders to their original values, allowing the system to reconstruct (rehydrate) the final response once the model replies. This design aims to reduce the exposure of sensitive information in human-readable form while maintaining the AI output. We evaluate the system in terms of detection accuracy, reconstruction fidelity, and utility preservation, demonstrating a practical solution for privacy-preserving LLM interaction.
Aditya Vishwakarma, Wencen Wu· International Conference on...· 0 citations
Now, AI runs on cloud platforms, edge systems with federated settings, and in large language model (LLM) pipelines or data-sharing services, creating even wider privacy leakage paths beyond classical database disclosure. This paper offers a systematic, structured review of the literature on a curated, cost-effective reference corpus for quantifying and preventing privacy leakage in AI-enabled data ecosystems. The review ties together four strands of research that are often treated separately. Firstly, the privacy risk throughout the AI life cycle. Secondly, the measurement of the quantitative leakage. Thirdly, architectures of the privacy-preserving models, and finally, operational governance for real-world deployment. Our analysis demonstrates that state-of-the-art approaches are moving from static mechanisms based on anonymization to metric-aware protections, including information-theoretic leakage scores, cumulative differential privacy accounting, personalized privacy budgets, and benchmark-driven attack evaluation. In parallel, prevention methods are evolving beyond single homomorphic noise injection and are becoming multi-layered defenses that combine differential privacy, federated learning, weight quantization, synthetic data generation, policy-driven automation, and LLM controls. The review uncovers four itchy gaps: fractured assessment metrics, shaky privacy-utility trade-offs, flimsy integration of technological controls and compliance processes, and low cross-context validation across cloud-based computing, edge computing (data processing at or near the source), federated learning (distributed machine-learning methods), and generative AI systems. The paper concludes by outlining a unified research agenda to build AI-aware, quantifiable, and usable privacy protection stacks.
Privacy documents (e.g., privacy policies) are a central mechanism through which digital services disclose data practices and seek user consent. Over the past decades, research on privacy documents has expanded significantly, encompassing not only traditional privacy policies but also short notices (e.g., privacy labels) and interface-level transparency mechanisms. As this research area continues to grow, it has become increasingly difficult to obtain a coherent view of how privacy documents are created, analyzed, evaluated, and maintained across their lifecycle. This SoK provides a unified, lifecycle-oriented view of privacy documents from a software engineering perspective. We systematically review and analyze 290 papers published between 2010 and 2025, organizing them around five research questions that examine how privacy documents are (1) defined and scoped, (2) generated, (3) analyzed and extracted, (4) checked for inconsistencies and noncompliance, and (5) evaluated and improved for usability. Building on our findings, we identify 15 key research trends and 21 open opportunities. We further chart four broader research directions that highlight (i) emerging challenges in AI-centric platforms, (ii) the need for diverse and up-to-date data foundations, (iii) LLM-based unified policy-code analysis, and (iv) dual usability for end-users and developers. We hope this SoK provides a shared foundation for future research on privacy policies and privacy documents.
Shidong Pan, Clark LaChance, Zhenyuan Tao et al.· 0 citations
: The decentralized nature of microservice architectures introduces privacy challenges that exceed the capabilities of traditional static auditing. To address this, we present a real-time privacy quantification framework based on the Privacy-sensitive Data Categorization (PsDC) model. By combining Layer 7 Deep Packet In-spection (DPI) with Natural Language Processing (NLP), our methodology continuously inspects the semantic intent of network payloads. We introduce the Privacy Exposure Index (PEI), a dynamic risk metric that links the detection of sensitive entities directly with their operational context. We validated this approach in a Ku-bernetes environment using a custom ingestion engine and analyzer. Experimental results show high fidelity in identifying complex threats, including DNS-based data exfiltration, and successfully isolated high-risk service interactions with localized PEI scores of 5.32. Ultimately, this work establishes a foundation for a proactive DevPrivOps lifecycle, demonstrating that semantic-aware observability can replace manual privacy checks and act as the core decision engine for active privacy enforcement.
Catarina Silva, Bernardo Falé, Paulo Barraca et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
