Skip to content
Preprint

Live Adaptation of Dataflow Stream Processing for Hybrid Edge-Cloud Analytics

Oct 2026 · 0 citations · 62 references
Computer Science

TL;DR

Sluice is introduced, an adaptive data stream processing system for hybrid clouds, compatible with dataflow streaming engines, and shows that Sluice seamlessly shifts computations between edge and cloud, while consistently achieving higher throughput than baseline systems.

Abstract

Dataflow streaming systems, like Apache Flink and Spark Streaming, are integral to modern business analytics, providing real-time insights across industries. However, for emerging hybrid edge-cloud analytics pipelines, neither processing everything in the cloud nor static edge offloading can reliably sustain the target rate under dynamic WAN conditions and limited edge resources. In this paper, we explore how these datacenter-centric platforms can be extended to support such hybrid analytics pipelines. Our goal is to maintain full transparency for user applications, while retaining built-in fault tolerance, out-of-order processing, and existing ecosystem integrations. Achieving this is challenging because existing dataflow engines assume homogeneous resources, rely on static task placement, and do not support adaptation under dynamic network and edge conditions. We introduce Sluice, an adaptive data stream processing system for hybrid clouds, compatible with dataflow streaming engines. Sluice's key contributions are: (i) an automatic query rewriter that decomposes applications into independent, fault-tolerant segments, and (ii) a live routing mechanism that shifts computations between edge and cloud without pausing execution while preserving event-time semantics. We implement Sluice on Apache Flink and evaluate it on a physical testbed with Raspberry Pi devices, as well as on a large simulated testbed of 100 VMs. Across five real-world applications, our results show that Sluice seamlessly shifts computations between edge and cloud, while consistently achieving higher throughput than baseline systems.

View source

Similar papers

Conference Open access Sep 2026

SC-RAP: A Scalable Cloud-based Pipeline for Real-Time Big Data Analytics on IoT Sensor Streams

SC-RAP, a cloud-based analytics pipeline that pairs Kafka for high-throughput ingestion with Spark Structured Streaming for microbatch processing, is presented and compared against a Hadoop MapReduce batch baseline across cluster sizes of 2, 4, 8, 16, and 32 worker nodes.

S. Deore, Mahadev Kalyanrao Patil, S. M. Rajput et al. · 0 citations
Review Open access Sep 2026

Cloud-Native Data Architecture: Optimizing Enterprise Analytics Using Multi-Cloud Data Warehousing Platforms

The study finds that cloud-native data architecture can be an essential building block for next-generation enterprise analytics, and multi-cloud data warehousing platforms can deliver the flexibility needed for enterprise digital transformation.

Md Ali Azam, Gurpreet Kaur, Keya Karabi Roy · 0 citations
Open access Oct 2026

Benchmarking Resilience of Stream Processing Middleware Deployed on the Edge

Shifting centralized server architectures to edge devices enhances privacy and reduces network bandwidth consumption. To enable this transition, stream processing middleware on edge devices is required to pre-process and manage data effectively. In this work, the cloud-native, distributed event streaming platform Apach...

Mika Bjarne Petersen · 0 citations
Review Open access Sep 2026

Serverless Functions in Cloud–Edge Environments: A Comprehensive Critical Review and Taxonomy

An extensive critical review of serverless functions in cloud–edge environments reveals that cloud–edge serverless systems need accountable placement, state-aware workflows, reproducible benchmarking, trustworthy orchestration, and carbon-aware lifecycle control, which can be achieved only by going beyond latency and e...

Abdullah Abbasi, D. Hakro, Asad Ullah et al. · 0 citations
Book Open access 2026

Big Data Stream Processing

This chapter provides students, industry experts, and researchers a high-level and comprehensive overview of the end-to-end architectures of big data stream processing pipelines, designed to keep them informed of the latest advancements and best practices in the field. It explores crucial components such as source inge...

Ovidiu-Cristian Marcu, Pascal Bouvry · 0 citations
#machine learning Preprint Oct 2026

Accelerated Dataflow: scaling inference in streaming data analytics with efficiency

Streaming data analytics helps businesses identify opportunities and spot risks in a timely fashion. As large language model (LLM) usage grows, people are exploring incorporating agents or LLMs in streaming data pipelines to help find patterns in datasets without supervision or interpret natural languages for content m...

RuiLong Jiang, Dustin Rhodes, Danny McCormick et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.