Sep 2026· 2026 10th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC)· pp. 670-676· 0 citations· 9 references
TL;DR
SC-RAP, a cloud-based analytics pipeline that pairs Kafka for high-throughput ingestion with Spark Structured Streaming for microbatch processing, is presented and compared against a Hadoop MapReduce batch baseline across cluster sizes of 2, 4, 8, 16, and 32 worker nodes.
Abstract
Batch-oriented frameworks such as Hadoop MapReduce were built for a world where a data pipeline could afford to wait minutes for a result. Sensor-driven applications, from predictive maintenance to grid monitoring, cannot make that trade anymore: a reading that arrives 30 seconds late is often a reading that arrived too late to act on. This paper presents SC-RAP, a cloud-based analytics pipeline that pairs Kafka for high-throughput ingestion with Spark Structured Streaming for microbatch processing, and compares it against a Hadoop MapReduce batch baseline across cluster sizes of 2, 4, 8, 16, and 32 worker nodes. We model throughput scaling, end-to-end record latency, and cluster scaling efficiency using a discrete-event simulation calibrated against published per-node processing rates for both frameworks. At an 8-node cluster, SC-RAP delivers 92.2% higher throughput (8161 vs. 4246 records/sec) and a 90.5% reduction in mean end-to-end latency (1.68 s vs. 17.68 s) relative to the MapReduce baseline, while sustaining 86.5% scaling efficiency against 55.9% for the baseline at the same cluster size. The gap widens with cluster size because MapReduce's shuffle and disk-I/O overhead grows faster than Spark's in-memory micro-batch model. We discuss where this matters in practice and where a batch system remains the better, cheaper choice.
Sluice is introduced, an adaptive data stream processing system for hybrid clouds, compatible with dataflow streaming engines, and shows that Sluice seamlessly shifts computations between edge and cloud, while consistently achieving higher throughput than baseline systems.
Yuan-Li Wang, Prateek Jain, Lei Huang et al.· 0 citations
The rapid growth of Industrial IoT (IIoT) has led to large volumes of sensor data, increasing the need for scalable and data-driven maintenance solutions. This paper presents a replay-based streaming framework for predictive maintenance that integrates Apache Kafka for data ingestion, Apache Spark Streaming for distrib...
Muhammad Asad, Yaser Alhasawi, Salem Alghamdi et al.· Systems· 0 citations
This article outlines the development and testing of a real-time big-data analytics architecture that uses state-of-the-art stream-processing tools, designed for high-load and crisis-prone operational environments typical of current operations in Ukraine. The motivation for this research stems from the rapid growth of...
G. Silvanskaya, Oleksii Shapyro, K. Terletska et al.· Sustainable Engineering and...· 0 citations
Industrial Internet of Things (IIoT) systems generate sensor workloads that require scalable and workload-aware database management. This study evaluates an adaptive PostgreSQL optimisation framework combining conventional indexing, machine-learning (ML) query-latency prediction, local Generative AI (GenAI), and Hybrid...
Lakmali Karunarathne, N. Somasiri· Journal of ISMAC· 0 citations
In smart manufacturing scenarios, the processing of massive high-dimensional time-series data often faces severe I/O amplification and deserialization overhead issues. Traditional relational databases, which rely on hardware-based data management models such as network storage and backend servers, suffer from long quer...
Jiawei Hai· International Conference on...· 0 citations
While Apache Spark offers highly efficient in-memory data processing capabilities, such systems tend to exhibit poor performance and execution failures under memory-constrained infrastructure owing to high memory footprint and shuffle amplification. On the other hand, although Hadoop MapReduce is slower compared to Spa...
Aryan Bramhe, Anuja Patil, Pranjal Sharma et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.