Jul 2026· International Conference on Big Data Computing Service and Applications· pp. 125-129· 0 citations· 17 references
Abstract
Modern data platforms rely on pipeline-oriented architectures that are rigid, hard to adapt, and lack native auditability. We present Agentic Data Services, a control-planedriven architecture for Big Data as a Service (BDaaS) that models workflows as adaptive, policy-aware service entities rather than static directed acyclic graphs (DAGs). The architecture combines (i) a dual-record execution model adapted from pharmaceutical batch manufacturing-versioned Master Batch Records (MBRs) for workflow definition and immutable Electronic Batch Records (EBRs) for execution traces-and (ii) workflow-level semantic caching that reuses results across semantically similar requests. We implement the system as Agentic DataHub, a set of Rustbased microservices deployed on Kubernetes, and evaluate the semantic-caching component on a reproducible benchmark using sentence-transformer embeddings (all-MiniLM-L6-v2) and a FAISS flat inner-product index. For clustered workloads-semantically related requests grouped into 10 clusters with 70% intra-cluster similarity-the cache reduces backend requests by 81% and median latency by 93%, with 40% P95 latency reduction. We discuss generalization across domains and the architectural constraints that bound these results.
This work presents DeepEye, a workflow-centric agentic data system that turns user intents into transparent and steerable analytical workflows and develops DataMagic as the system’s Video Generator, a declarative multi-agent method that improves data-video quality.
Traditional cluster-based ETL architectures impose a structural tax on data engineering organisations: fixed compute resources provisioned for peak demand, scheduled batch cycles that introduce latency regardless of downstream urgency, and operational overhead that redirects engineering capacity from pipeline design to cluster maintenance. AWS serverless services Lambda, Glue, Step Functions, and Kinesis address this tax by decoupling compute from provisioning, enabling pipelines that execute in response to events, scale automatically with workload demand, and charge only for resources consumed. Python, with native Lambda runtime support and an ecosystem that spans Pandas, PySpark, and the AWS SDK for Pandas, has become the primary implementation language for serverless ETL logic at an enterprise scale. This article examines the engineering decisions that determine whether that architectural promise translates into operational reality. The analysis covers Lambda’s execution model and its hard constraints, Glue’s managed Spark environment and its cost engineering implications, Step Functions orchestration patterns, S3-native data lake architecture, cost optimization strategies, security and governance controls, and real-time processing patterns. The central argument is that serverless ETL does not simplify data engineering; it redistributes engineering effort from infrastructure management to execution model design. Lambda timeout decomposition, cold start management, Glue worker right-sizing, IAM role scoping, and Step Functions state machine cost design are not configuration afterthoughts; they are first-class engineering decisions with direct consequences for pipeline reliability, cost, and compliance readiness. The analysis draws on the practitioner perspective of a data engineer with extensive experience building ETL pipelines in regulated financial services and healthcare environments where infrastructure decisions carry audit consequences.
Unknown authors· East African Journal of Info...· 0 citations
DataClawEval is introduced, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios, and it comprises 100 rigorous, end-to-end tasks spanning five execution engines.
Cloud native data pipelines have become an enabling ingredient of the modern enterprise analytics to fulfill the ever-increasing demand of a scale-loving, resilient, and real-time processing of a wide range of data sources. Organizations currently produce large amounts of structured, semi-structured, and unstructured data in transactional systems, Internet of Things (IoT) platforms, digital channels, and data sources that are external (Bank of America 2017). Old monolithic data integration architectures are designed to provide batch-oriented processing and static infrastructure capabilities have challenges satisfying low latency, scale on demand, and 24/7 requirements. Reactively, cloud-native paradigms, including the foundations of microservices, container orchestration, event-driven architectures, and managed cloud services have caused a rethinking of the data pipeline design, deployment and operation. This article provides an in-depth analysis of cloud-native pipeline data to enterprise analytics along with their main architectural concepts and processing models as well as operational aspects that fall within the professional scope of IEEE publications. The research paper summarizes the literature and business methodologies to present a reference model which brings together data ingestion, stream processing, batch processing, storage, governance, and analytics consumption layers. Special concern is opened to the contributions of containerization, orchestration platforms, and serverless computing towards facilitation of elasticity and fault tolerance. The paper also examines design patterns like Lambda architecture and Kappa architecture, data mesh theory and metadata-based orchestration, with an emphasis on its application to the large enterprise environment. An organized approach to the design and deployment of cloud-native data pipelines with the inclusion of data quality management, security controls, observability, and cost optimization is suggested. Throughput, latency and scalability modeling mathematical formulations are proposed in order to facilitate capacity planning and performance measurement. Representative enterprise workloads as shown through experiment results exhibit evident increases in data processing latency, pipeline reliability and operational efficiency over traditional architectures. These findings are placed in context to the discussion of the broader transformation efforts at enterprises, whereas the conclusion provides recommendations on future research opportunities, such as autonomous pipeline optimization and AI-based orchestration.
Ethan Williams· International Journal of App...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
