Hybrid Optimization-Learning for Interference Management in Coexisting NTN–TN Systems

Abstract

Spectrum sharing between non-terrestrial networks (NTNs) and terrestrial networks (TNs) represents a key enabler for enhancing spectrum utilization in next-generation wireless systems. However, such coexistence introduces severe crossnetwork interference. Power control is an effective mechanism to mitigate this interference while maintaining the spectral efficiency benefits of spectrum sharing. This paper addresses joint transmit power optimization of low Earth orbit (LEO) satellites and terrestrial base stations (BSs) operating in shared spectrum, with the objective of maximizing the network sum rate while meeting user quality-of-service (QoS) requirements. The formulated problem is complex and non-convex due to coupled interference terms, which limits the effectiveness of conventional convex-approximation techniques. To achieve a practical solution, a two-stage optimization-learning framework is developed. In the first stage, the Whale Optimization Algorithm (WOA) is applied offline to generate near-optimal power allocations. In particular, WOA yields accurate power allocations after multiple iterations; however, its iterative nature and high computational cost restrict its applicability for real-time operation in dynamic NTN-TN scenarios. To overcome this limitation, the second stage introduces a convolutional neural network (CNN)-based regressor trained on the WOA dataset to learn the mapping between instantaneous network states and optimized power values. Once trained, the CNN-based regressor predicts transmit powers directly from the current network state with negligible computational overhead, allowing fast and adaptive interference management suitable for dynamic NTN-TN systems. Simulation results show that WOA achieves performance close to the upper-bound no-interference case, while the CNN-based regressor closely matches WOA.