2026· IEEE Transactions on Networking· Vol 34, pp. 6095-6110· 0 citations· 56 references
Computer Science
Abstract
Data processing clusters dealing with big data are facing extended completion times for jobs because the RDMA feature is not being used efficiently. Our production data from a large cluster, which has many server nodes and is responsible for processing extensive data jobs, reveals that the current use of the RDMA technology is causing some jobs to finish much later than usual, with a few taking more than double the standard time to be completed. In this paper, we introduce the design and implementation of Turbo, a high-performance, scalable communication framework tailored for large-scale data processing clusters. The essence of Turbo’s strategy lies in the use of a dynamic block-level flowlet transmission system and a non-blocking communication middleware, which are designed to boost network throughput and system scalability. Moreover, Turbo maintains high system reliability by incorporating an external shuffle service with TCP as a fail-safe option and maintaining metadata management information using the NICs. We have integrated Turbo into Apache Spark and conducted evaluations on both a small-scale test environment and a large-scale cluster with hundreds of server nodes. The findings from the small-scale testbed demonstrate that Turbo enhances network throughput by 15.1% and upholds high system reliability. Additionally, the large-scale production data indicates that Turbo is capable of decreasing job completion times by 23.9% and increasing the job completion rate by $2.03\times $ compared to the current RDMA solutions. In addition, during large-scale tests, we also found that Turbo has improved the computing efficiency of the cluster and saved approximately 24.3% of the CPU utilization.
The explosive increase in digital information newest as a result of the social media sites, Internet of Things (IoT) devices, enterprise information systems, scientific simulations, and e-business programs has radically changed the computing needs of current-day data analytics. The conventional centralized designs of data processing architecture can not handle the volume, speed, and characteristics of such data, leading to scalability bottlenecks, high latency, and lower fault tolerance. The distributed computing setups have become a key facilitator of massive data processing through the harnessing of parallelism, data locality, and elasticity of resources on groupings of commodity hardware. The current paper is the in-depth study of the large-scale data processing in the context of distributed computing structures. It looks at the architectural concepts, programming models and mechanisms of execution used to implement contemporary distributed data processing systems. This paper critically evaluates leading systems like Hadoop MapReduce, Apache Spark, and Apache Flink systems and how they evolved to be based around batch processing rather than a hybrid batch/stream processing model. An extensive literature review brings together the previous studies carried out on scalability, fault tolerance, scheduling and performance optimization in a distributed environment. The suggested methodology comes up with a distributed data processing architecture that is layered and incorporates the resource intelligent resource management, parallel execution engines, and scalable storage. There are mathematical data partitioning, execution cost, and scalability mathematical formulations that are used to formalize system behavior. Experimental measurements based on the benchmark workloads show that there is a high increase in the throughput, execution time, and fault recovery against the traditional centralized systems. The trade-offs between frameworks analyzed in the discussion are based on latency, resource efficiency, and programming complexity. The paper ends by presenting the questionable opportunities to open research, such as scheduling of resources adaptively, data processing energy-efficiently, and applying artificial intelligence to autonomous optimization. The results are very helpful to researchers and practitioners who would have to create the next-generation of large-scale data analytics platforms.
Efficient data movement is a critical bottleneck in High Performance Computing (HPC) centers, especially for large-scale migrations across parallel and distributed storage systems. Existing tools such as rsync and rclone often underutilize network and metadata bandwidth, provide limited hard link preservation, and offer no guarantees on memory usage, limiting their effectiveness for HPC environments. Motivated by a large-scale migration of data at Partnership for an Advanced Computing Environment (PACE), we present copy2, a high-performance, resource-aware data transfer tool for modern HPC storage such as Lustre and VAST Data. copy2 features a scalable metadata crawler, inode-based hard link preservation, asynchronous high-throughput transfers, and bounded memory allocation. Preliminary results show up to twice the throughput of rclone with substantially lower energy usage, and successful migration of projects with hundreds of millions of files that would be impractical with existing tools. copy2 offers HPC administrators and researchers a robust, predictable, and efficient solution for large-scale data management.
A. Lambert, Deepa Panish· Practice and Experience in A...· 0 citations
The analysis of performance has shown that the proposed solution is much better in terms of throughput, latency and fault isolation than the conventional architectures, proving that microservices architecture with proper design presents a solid base of scalable data-intensive systems.
S. Rahman· International Journal of App...· 0 citations
Financial data preprocessing is a key link in financial analysis and modeling. With the exponential growth of data scale, the single-machine architecture is facing severe bottlenecks. Distributed computing provides a feasible path to break through performa nce limitations through multi-node collaborative processing. This article systematically sorts out the research status and technical progress of distributed computing in the field of financial data preprocessing. First of all, analyze the common characteri stics and pre-processing task genealogy of financial data, and combine Tang Yao’s stock linkage effect research to show the typical process of single-machine preprocessing pipelines and its scale bottlenecks; Then review the mainstream frameworks such as Hadoop, Spark and Flink from the perspective of technological evolution, and summ arize the three core empowerment mechanisms of data parallelism, computing parallelism and stream processing; On this basis, classify and summarize the application research of distributed computing in data cleaning, feature engineering, real-time processing and other links, and take Ma Chiyu’s financial news sentiment analysis based on SparkR as a case to verify the performance advantages of distributed preprocessing in actual tasks; Finally, we will comment on the limitations of existing research and look forward to the future direction of ad aptive preprocessing, explainable attribution, privacy protection calculation, etc. This article aims to provide a systematic reference for distributed computing applications in the field of financial data preprocessing.
The proposed hybrid methodology integrates workload prediction, adaptive scheduling, and resource consolidation, demonstrating significant energy savings without compromising system performance is proposed.
Seshagiri N· International Journal of Dat...· 0 citations
Big data are enormous, intricate sets of information that are too large, too quick, and too difficult to handle using conventional methods. They consist of both structured and unstructured data. Due to the alarmingly high rates of data generation, there is pressure to adopt simple and expensive procedures for data recovery and storage. Conventional database systems cannot handle the processing power of big data. In order to evaluate and query this enormous amount of data, new technologies have recently been created. In this work, a novel hybrid approach for efficient big data query processing through a two-phase approach is introduced. In the first phase, the Hadoop Distributed File System (HDFS), Map Reduce are utilized. Second phase involves optimization using Particle Swarm Optimization (PSO). The proposed algorithm aims to enhance search speed by strategically optimizing various processes. This two-step methodology combines the parallel processing capabilities of HDFS Map-Reduce with the optimization power of PSO, contributing to an overall improvement in query processing efficiency for large-scale datasets. Experimental results demonstrate significant outperformance of the model in expediting query processing.
S. Selvan, Gowri R. Punitha, V. V. Nathan et al.· ITEGAM- Journal of Engineeri...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
