Data is a critical component of modern decision-making systems; system malfunctions (e.g., performance degradation and module failure) can often be traced back to a mismatch between the properties of the data and the assumptions of the system modules that process the data. For example, with the increasing use of open-source libraries to develop data science pipelines, common causes of system malfunctions include inappropriately configured data processing libraries for data cleaning tasks such as entity resolution or missing value imputation. Our objective is to resolve malfunctioning pipelines and improve their utility; we introduce PipeLens, a framework that leverages successful and unsuccessful runs of past pipelines for fixing pipeline malfunctions. PipeLens uses an acyclic graph representation of the pipeline and performs causal reasoning through interventions: when a system malfunctions with a given dataset, PipeLens modifies the pipeline (by changing its structure or the parameters of its modules) and observes the impact of this intervention on system behavior. To focus on useful interventions, we learn a proxy function that approximates the pipeline's utility over a dataset and guides the search for the best intervention. Unlike traditional observational analysis that reports correlations between system parameters and their behavior, we provide causally verified root causes and suggest pipeline modifications that rectify malfunctions. Empirical evaluation on four data science tasks over four real-world datasets demonstrates that PipeLens consistently outperforms baselines in terms of interventions performed to repair malfunctions while maintaining practical running times.
Jahid Hasan, Stanley Jiang, T. Singh et al.· Proceedings of the VLDB Endo...· 0 citations
All fairness algorithms unavoidably rely on normative assumptions about fair treatment, yet these assumptions often remain implicit. We argue that these assumptions should be formalized as explicit, auditable policies and introduce FAIR-SHEPHERD, a structural policy-based framework for transparent fairness in real-world settings with noisy labels and shifting distributions. FAIR-SHEPHERD uses gradients as attribution signals, encoded in a Structural Fairness Specification (SFS) that defines vertical coherence and orthogonality to vulnerable proxies. We introduce SFS metrics: Vertical Coherence Score (VCS) to measure directional coherence across adjacent normative slices, and Horizontal Leakage Score (HLS), augmented by a signed directional variant, to detect gradient alignment with policy-declared vulnerable or proxy directions. To enforce these policies, we develop S-agnostic tools that optimize policy-aligned gradient objectives using lattice-defined proxies. We demonstrate that outcome-based auditing is brittle to measurement error: under 10% label noise on Adult, Worst-Group AUC for ERM drops by 0.159. In contrast, our gradient-based structural metrics provide a label-agnostic audit of the decision logic, remaining stable even when evaluation labels are corrupted. Specialized fairness baselines including ARL, JTT, and GoG retain substantial structural leakage on COMPAS, with HLS values from about 0.47 to 0.71. Some also reduce EOD relative to ERM, which shows that outcome and structural criteria can diverge. Using our S-agnostic Gradient Penalty tools, we reduce policy-specified structural leakage by roughly 85% on COMPAS (0.478 → 0.073) and over 94% on Adult (0.138 → 0.008) while retaining competitive AUC and Worst-Group AUC. We have released our source code to facilitate further research here.
Shubham Mohole, Hongjun Choi, Christian Schroeder et al.· ACM Journal on Responsible C...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
