2025· International Journal of Applied Data Science & Modern Computing· 0 citations
TL;DR
A reinforcement learning (RL)-based autonomous optimization framework that integrates RL agents with data orchestration platforms to continuously monitor pipeline states and optimize operations that enables adaptive, self-managing data pipelines that improve scalability, resilience, and operational efficiency across enterprise, cloud, and edge environments.
Abstract
Modern large-scale data pipelines support analytics, AI, ML, and real-time applications but face challenges related to scalability, resource utilization, reliability, and changing workloads. This paper proposes a reinforcement learning (RL)-based autonomous optimization framework that integrates RL agents with data orchestration platforms to continuously monitor pipeline states and optimize operations. The framework uses system metrics such as workload patterns, queue lengths, execution delays, resource consumption, and failure rates to make intelligent decisions on task scheduling, resource allocation, workload balancing, and fault recovery. Three RL algorithms—Q-learning, Deep Q-Networks (DQN), and Proximal Policy Optimization (PPO)—are evaluated. Experimental results demonstrate improved throughput, reduced latency, enhanced fault tolerance, and better resource efficiency compared to traditional rule-based approaches. The proposed framework enables adaptive, self-managing data pipelines that improve scalability, resilience, and operational efficiency across enterprise, cloud, and edge environments.
Modern enterprises face increasing demands for scalable and efficient data processing due to rapid data growth. Traditional data pipeline orchestration methods, which rely on static configurations and manual intervention, often lead to inefficiencies in resource use, latency, and fault tolerance. This paper proposes an AI-assisted orchestration framework that integrates machine learning techniques to enable dynamic scheduling, workload prediction, anomaly detection, and resource optimization. By leveraging reinforcement learning, supervised learning, and heuristic methods, the system adapts pipeline configurations in real time based on changing workloads and system conditions. The proposed architecture includes data ingestion modules, AI-driven orchestration engines, adaptive schedulers, and monitoring systems. A key contribution is an intelligent scheduling mechanism that improves execution efficiency and resource utilization. Experimental results show significant improvements over traditional systems, with up to 35% increase in processing efficiency and 25% reduction in latency. The study concludes that AI-driven orchestration is a promising approach for building scalable and autonomous data processing systems, with future work focusing on deeper integration of advanced learning models and real-time adaptability.
J. Weizenbaum, S. Papert· International Journal of Dat...· 1 citation
This work proposes a scalable, intelligent, and resilient foundation for next-generation high-performance analytics and data-intensive applications that integrates adaptive resource management, intelligent workload scheduling, dynamic task migration, predictive analytics, and machine learning-based optimization to improve computational efficiency and responsiveness.
John Peterson, L. Martínez· International Journal of App...· 0 citations
Reinforcement learning-based adaptive resource management framework is proposed that enables cloud systems to autonomously learn optimal resource allocation policies through continuous interaction with the environment and significantly outperforms static and reactive baseline strategies in terms of resource utilization efficiency and response time stability.
Rajesh Sharma, Priya Natarajan· International Journal of Mac...· 0 citations
The rapid growth of enterprise applications has led to a substantial increase in the volume, velocity, and variety of data, necessitating intelligent data engineering pipelines for efficient processing and analytics. These pipelines play a critical role in enabling scalable data integration, transformation, and delivery across distributed environments. However, existing pipeline architectures face significant challenges, including limited scalability, high latency, inefficient resource utilization, and lack of adaptability to dynamic workloads. Traditional approaches rely on static scheduling and rigid execution models, which restrict their effectiveness in real-time and large-scale scenarios. To address these limitations, this paper proposes an Intelligent Data Engineering Pipeline Architecture supported by an Adaptive Optimization Strategy. The framework integrates hybrid processing, metadata-driven orchestration, and a learning-based cost optimization model to enhance system performance. A key contribution is the Intelligent Adaptive Pipeline Optimization Strategy (IAPOS), which enables dynamic scheduling, cost-aware decision-making, and feedback-driven learning for continuous performance improvement. Experimental evaluation demonstrates that the proposed approach achieves improved scalability, reduced processing latency, and more efficient resource utilization compared to conventional methods. These results highlight the effectiveness of intelligent and adaptive pipeline architectures for next-generation enterprise applications.
D. Bansal, Dinesh Kumar Garg· 2026 7th International Confe...· 0 citations
The findings demonstrate that DRL-driven adaptive orchestration can become a central mechanism for autonomous edge intelligence in next-generation AI-native communication infrastructures.
Amit K. Mogal, Rahul A. Patil, Sahebrao N. Shinde et al.· International Journal of Inn...· 0 citations
This research proposes an AI-driven resource scheduling framework that integrates workload prediction, resource classification, intelligent scheduling, and continuous feedback mechanisms that aims to optimize multiple objectives, including cost reduction, execution efficiency, energy consumption, and SLA compliance.
Michael Anderson· International Journal of App...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
