Performance Optimization of High-Speed Wireless Communication Systems Using Machine Learning

Abstract

High-speed wireless communication systems underpin the data-intensive demands of contemporary 5G deployments and the emerging 6G paradigm, yet sustaining high throughput, low latency, and dependable connectivity in channels that shift rapidly remains an open engineering challenge. This paper presents an ML-driven framework that combines a deep reinforcement learning (DRL)-based scheduler with a hybrid CNN-LSTM channel predictor to jointly optimize radio resource allocation, modulation order, transmit power, and bandwidth assignment in real-time. Evaluated across full-buffer, bursty, mixed-traffic, and high-mobility scenarios using a 3GPP TR 38.901-compliant simulator, the proposed scheduler achieved 23.0% higher throughput, 35.0% lower latency, and 18.1% higher energy efficiency than Proportional Fair scheduling, while reducing Quality of Service (QoS) violations by 75.9% and packet loss by 76.3%, with a Jain's Fairness Index of 0.887. The companion CNN-LSTM channel predictor achieved 94.7% prediction accuracy and a normalized mean squared error of −16.2 dB, reducing CSI-feedback overhead by 41.3% relative to reactive CSI feedback schemes and outperforming standalone CNN and LSTM baselines by 28% under high-mobility conditions. These results indicate that coupling predictive channel-state modeling with multi-objective reinforcement learning offers a practical and quantifiable pathway toward achieving next-generation wireless performance targets.