Performance Prediction of Data-Rebalancing Actions in Sharded NoSQL Clusters

Abstract

Scalable storage systems typically provide mechanisms for re-distributing data over time to ensure a balanced storage allocation. Such mechanisms have traditionally been considered as background activities meant to run at low priority, to avoid penalizing applications accessing storage. In this work, we focus on an alternative design point where data re-distribution is run as a high-priority activity, meant to re-balance data at the full speed allowed by the newly added resources. Such rapid redistribution has become viable in recent years due to the rise of low-overhead technologies in the networking and disk storage space. A challenge in such a scenario is to schedule data transfer flows to newly added nodes in a way that fully and efficiently utilizes network, CPU, and disk resources in the new nodes, while avoiding overload. To understand the impact of different parameters of the rebalance process, such as the number of simultaneous senders, we develop a queueing network (QN) model of the process used in MongoDB, and describe a preliminary evaluation of performance prediction by the QN model via simulations.