PyDataQuality: An Actionable, Lightweight Data Profiling and Distribution Drift Detection Framework for Production Machine Learning Pipelines

Abstract

In modern data analytics and machine learning operations (MLOps), the reliability of artificial intelligence systems is fundamentally constrained by the quality and distributional stability of the data they consume. The "Garbage In, Garbage Out" (GIGO) principle has taken on renewed urgency as production ML systems silently degrade when data pipelines introduce missing values, outliers, and distributional drift. This paper presents PyDataQuality, a lightweight, modular, and open-source Python library for automated data quality assessment and statistical distribution drift detection. PyDataQuality addresses the gap between overly simplistic pandas summaries and computationally bloated industrial profiling suites. It introduces three principal technical contributions: (1) a type-dispatched, column-level profiling kernel; (2) a programmatic get_problematic_rows() subset extractor for isolating anomalies; and (3) a dual-metric statistical drift engine implementing Population Stability Index (PSI) with adaptive binning and a two-sample Kolmogorov-Smirnov (KS) Test. A complementary AI Remediation Prompt generator bridges the framework with generative AI for LLM-based cleaning code generation. We present a rigorous empirical evaluation across four dataset scales (1,000 to 100,000 rows), benchmarked against pandas, YData-Profiling, and Evidently AI. A controlled MLOps case study on both synthetic and real-world datasets demonstrates the system's ability to halt inference and trigger retraining before serving degraded predictions. Crucially, PyDataQuality's sampled analysis mode demonstrates bounded post-sampling latency, ensuring predictable execution times and near-zero incremental peak memory independently of the source dataset size. The full-dataset mode delivers substantial speedups over full-featured profiling tools such as YData-Profiling, at modest overhead relative to pandas' minimal baseline. The library is released as open-source software under the MIT License (pip install pydataquality), including comprehensive documentation and a Jupyter Notebook tutorial.