Aug 2026· International Conference on Circuit, Power and Computing Technologies· pp. 1967-1973· 0 citations· 19 references
Abstract
In the world of modern analytics and machine learning, pipelines known as Extract-Load-Transform (ELT) pipelines are commonly used to transform raw data into representations that are suitable for models. But most of the existing ELT workflows are primarily correlation driven and don’t explicitly maintain causality during data transformation. This means that joins, filtering, aggregation and feature preparation steps can create confounding effects, selection bias, or false treatment-outcome relationship that can create unreliable AI-based decisions. This paper proposes such a causal intelligence enhanced ELT framework, where causal reasoning is integrated into the ELT process directly. Three pipeline-native components are proposed: causal graph-constrained data integration via Causal Join, covariate balancing and adjustment in-pipeline via Propensity Balance, and execution optimization via task scheduling and causal artifact reuse via Causal Scheduler. The proposed method is reduces bias earlier in the data lifecycle compared with purely post-hoc causal modelling less biased than post hoc causal modelling approaches as the transformation process itself is less biased. On experiments performed on the IHDP, Twins and Criteo uplift datasets, we have seen improved estimation of treatment effects, lower bias and higher predictive reliability. The proposed framework reduces the error in the estimation by approximately 55-65%, improves predictive accuracy from 82.0% to 91.0%, corresponding to a 9 percentage-point improvement. and gives a $7.5 \times$ speedup performance for GPU compared to CPU. These findings point towards the promise of causal intelligence in making more reliable and efficient decisions within ELT pipelines, potentially enhancing the overall reliability and trustworthiness of AI-driven decision systems in the field.
CausalShift is proposed, a modular, plugin-based framework for end-to-end dataset shift handling that reduces the in-distribution to out-of-distribution accuracy gap, while remaining competitive on real-world image shift and achieving performance parity with ERM on mild-shift tasks.
Shuang Song, Muhammad Syafiq Mohd Pozi, Nik F. Farid· Applied Sciences· 0 citations
An open-source Python workflow that records each decision, derives defaults from an extended causal-audit module, and admits domain-informed overrides, AutoCause converts inconsistent expert practice into an auditable, repeatable analysis.
M. Ruiz, Miguel Arana-Catania, D. Ardila et al.· 0 citations
Approximate Nearest Neighbor (ANN) algorithms achieve high efficiency through interdependent phases across index construction and query execution. This coupling allows upstream performance loss to propagate downstream, affecting execution behavior and measurable outputs. Existing component-level analyses mainly compare isolated design choices, while end-to-end benchmarks report aggregate metrics; neither traces loss propagation across dependent phases, hindering root-cause attribution and automated redesign. We present SmartANN, a framework based on the object causal model (OCM) for ANN bottleneck attribution and automated redesign. SmartANN represents an ANN workflow as eight ordered, replaceable objects and diagnoses them with a sequential diagnose-and-replace loop. At each iteration, it identifies the first object deviating from expected behavior or output as a bottleneck. Because an upstream bottleneck can obscure downstream ones, SmartANN replaces it with a test oracle when available, or with an implementation producing a better outcome, then continues downstream diagnosis. From the diagnosed bottlenecks and failure causes, SmartANN composes compatible actions from a pluggable action library to generate an optimized end-to-end ANN design. We instantiate SmartANN for IVF-PQ and HNSW, covering partition-and-quantization and graph-based ANN families. Experiments on eight real-world datasets show that SmartANN improves Recall by 0.24--74.20%, and increases QPS by 28.8--256.5% at comparable Recall, with low diagnosis and auto-design overhead. The code is available at https://github.com/zhouyutong20/SmartANN.
Yu-Tong Zhou, Guoxin Kang, Lei Wang et al.· 0 citations
In the highly data-centric world of today, ETL (Extract, Transform, Load) pipelines are the basic components of enterprise analytics and decision-making. As companies are employing AI more and more to automate and maximize ETL pipelines, the explainability challenge is emerging alongside. The point is that AI systems are gaining more and more autonomy and the whole process of transformation is not any longer visible to the analysts who are left with just the end results. Simply put, the rationale of those decisions seems to be vague or hard to uncover at times. This raises questions. Knowing which action the AI took is not enough for stakeholders; they also want the reasons for this action. The data scientists may be able to vouch for the outcomes, but the compliance teams, business users, and regulators all require that the results they get are clear. In the absence of explainability, trust fades and there is an increased likelihood of biased or incorrect data handling. This is exactly where Explainable AI (XAI) finds its place. .The article is suggesting a framework for integrating XAI into large data pipelines, thus presenting each AI-powered change as easily understandable. Through the use of interpretable models, embedding of audit trails, and provision of real-time justifications for AI decisions, an organization can have the best of two worlds: a smart pipeline and one that is trustworthy. Apart from meeting regulatory demands, the use of these pipelines can also help cross-functional collaboration and the maintenance of organizational governance standards. The main point is quite straightforward: transparency is not only a compliance requirement but also a benefit in terms of business. The presence of explainable pipelines enables teams to debug quicker, audit more efficiently, and gain trust to a greater extent. In a world where data is a form of currency, being aware of the handling process is of utmost importance. Explainability is the link that connects innovation and trust in the AI-augmented ETL era.
Sivadeep Katangoori· International Journal of Eme...· 0 citations
Artificial Intelligence (AI) and Machine Learning (ML) have significantly improved predictive analytics across domains such as healthcare, finance, transportation, cybersecurity, manufacturing, and smart cities. However, conventional ML models rely on statistical correlations and often fail under dynamic environments due to concept drift, distribution shifts, and changing causal relationships. Causal Machine Learning (CML) addresses these limitations by integrating causal inference techniques, including structural causal models, directed acyclic graphs (DAGs), counterfactual reasoning, intervention analysis, and invariant causal prediction, to identify true cause-and-effect relationships. This enables more interpretable, robust, and generalizable predictive models. This paper proposes a unified CML framework that combines data preprocessing, causal graph construction, structural causal modeling, causal feature optimization, predictive learning, intervention analysis, and continuous model adaptation. Mathematical formulations support causal dependency estimation, structural equation modeling, invariant risk minimization, and prediction optimization. Experimental results demonstrate improved out-of-distribution prediction, robustness, causal consistency, explainability, and computational efficiency, providing a scalable foundation for trustworthy and adaptive AI in dynamic environments.
Mahabala H. N.· International Journal of Mac...· 0 citations
Modern enterprises rely on intelligent data pipelines to collect, process, transform, and analyze data from diverse sources such as cloud platforms, IoT devices, enterprise systems, and social media. Traditional optimization techniques, including rule-based scheduling and heuristic resource allocation, improve efficiency but struggle to adapt to dynamic workloads, changing resource availability, and evolving business requirements. Artificial Intelligence (AI) addresses these limitations through predictive analytics, adaptive scheduling, anomaly detection, and autonomous resource optimization. However, the opaque nature of many AI models reduces transparency, trust, and regulatory compliance. This paper proposes an Explainable AI (XAI)-based Intelligent Data Pipeline Optimization Framework that integrates data preprocessing, predictive analytics, explainability, and adaptive optimization. The framework continuously monitors pipeline performance, generates optimization recommendations, and provides human-interpretable explanations for AI-driven decisions using feature attribution and model interpretation techniques. An automated feedback mechanism enables continuous learning and improvement. Experimental evaluation demonstrates enhanced optimization accuracy, reliability, scalability, interpretability, and administrator trust with minimal impact on performance. The proposed framework provides a transparent and trustworthy approach for next-generation intelligent data engineering systems.
Per Brinch Hansen, O. Olesen· International Journal of Dat...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
