Risk Aware Adaptive Scheduling Between In-Memory and Disk-Based Big Data Frameworks
Abstract
While Apache Spark offers highly efficient in-memory data processing capabilities, such systems tend to exhibit poor performance and execution failures under memory-constrained infrastructure owing to high memory footprint and shuffle amplification. On the other hand, although Hadoop MapReduce is slower compared to Spark in terms of job completion time, it proves to be more reliable due to its disk-based execution capabilities. The paper discusses the implementation of the Risk-Aware Dynamic Execution Brokering framework, which can dynamically choose either Hadoop or Spark as per the current workload and infrastructure conditions. The framework involves the use of a monitoring component along with a cost model and workload sampling process that allows estimation of execution risks in advance. The sampling process entails micro-execution of sampling tasks with estimation of workload complexity and memory expansion; whereas the broker utilizes a cost model to predict the feasibility of execution and computes framework suitability based on runtime telemetry data. The experiment conducted involved the design of a Hadoop-Spark pseudo distributed single node cluster and use of synthetic workloads of sizes varying from 10GB to 100GB.