From Reactive Pipelines to Self-Healing Data Platforms: An Agentic AI Framework for Reliable, Secure, and Cost-Efficient Azure Data Engineering
Aim: This study aimed to design, implement, and empirically evaluate an agentic AI framework that improves the reliability, governance, security, and cost efficiency of enterprise Azure data platforms. The framework was intended to move operations from reactive, manual incident handling to policy-constrained automated monitoring and remediation, while preserving auditability, least privilege, and human oversight for high-risk actions. Specifically, the study sought to determine whether bounded agentic control could reduce operational toil, improve pipeline success and recovery times, strengthen security-governance posture, and lower unit costs without violating service-level or compliance constraints. Methods: We propose an agentic AI framework that (1) continuously telemetries pipeline runs, data-quality checks, lineage, and security posture; (2) retrieval-augments reasoning on operational knowledge (tickets, runbooks, KQL logs, IaC diffs); (3) policy-constrained action execution (RBAC, approvals, change windows, least privilege) to remediate failures, enforce baselines, and optimize resources; and (4) post-action validation to confirm recovery and prevent regressions. The system was built on Azure OpenAI and Azure Databricks, Data Factory, and Microsoft Fabric and tested with an enterprise deployment, a historical incident replay, and an A/B test against standard on-call procedures. Results: Manual interventions decreased by 65% across workloads, pipeline success rate increased from 91% to >97%, and annualized savings approached USD $1M, with better security-governance scores and lower cost per successful run. Results suggest that policy-gated autonomy lowers mean time to resolve (MTTR) and incident recurrence. Conclusion: The study supports the use of bounded, auditable agentic control for recurring operational failure modes. Recommendation: Future work should strengthen robustness guarantees, standardize multi-objective evaluation, and assess portability beyond Azure.