Digital twin enabled federated reinforcement learning for energy efficient spectrum allocation in heterogeneous vehicular networks

Abstract

With the evolution of smart networked vehicles towards smart driving, smart cabin, and ecology, the modern Internet of Vehicles (IoV) has become a typical application of new quality productivity in the automotive industry. In IoV resource allocation, intelligent decisions require Reinforcement Learning (RL). However, existing RL research methods are constrained by limited power supply capacity, and vehicles will face significant challenges in controlling energy consumption during decision training, which seriously hinders their ability to acquire sufficient data to support efficient decision optimization. This paper investigates the problems of energy consumption control and personalized spectrum resource allocation in a Federated Learning system comprising multiple IoV cells and edge clouds, with the objectives of reducing the energy consumption of end-side vehicles in the IoV system and improving the effectiveness of spectrum resource allocation decisions. Aiming at the training scale limitation and energy consumption control problems faced by end-side vehicles during spectrum resource allocation policy learning, this paper designs Digital Twin-Based Low-Energy Reinforcement Learning for Multi-Cell IoV (DT-LERL) distributed collaborative training architecture in cellular-based IoV scenarios, which allows the twin to replace the end-side vehicular entities by introducing digital twins to carry out scenario interactions and model training. It combines the decision frequency of digital twins with decision experience to dynamically balance digital twin error against model decision quality. By reducing the dimensionality of the locally trained network, the client can perform multiple local updates in each communication round, facilitating the learning of personalized local decisions. Experimental simulations show that, compared with other mainstream personalized FL algorithms, the proposed joint FL and digital twin scheme achieves better performance for the personalized spectrum resource allocation decision problem.