Digital Predistortion for Non-Differentiable Nonlinear Systems: A Reinforcement Learning Framework With Transfer Learning

Abstract

This paper proposes a reinforcement learning (RL)-based digital pre-distortion (DPD) method to mitigate pattern-dependent nonlinearities in optical transmitters. The RL agent is configured with an optimized symbol window size, allowing it to effectively model pattern-dependent nonlinearities introduced by the digital-to-analog converter (DAC) and power amplifier (PA). To assess its performance, the peak-to-peak output voltage ( $V_{pp}$ ) of the DAC is swept, and the results are compared with conventional DPD methods, including non–machine learning (ML) techniques such as linear DPD, Volterra series, and look-up tables, as well as ML-based approaches such as indirect learning architecture (ILA) and direct learning architecture (DLA). Our analysis demonstrates that the RL-based approach, enabled by an effective neural network initialization strategy, surpasses traditional methods and ML-based DPD schemes such as DLA and ILA. Furthermore, we incorporate transfer learning (TL) into the RL framework to reduce training complexity across different $V_{pp}$ levels without full retraining, achieving substantial complexity reduction. The results demonstrate that combining RL with TL provides a scalable and efficient solution for compensating pattern-dependent nonlinearities in high-speed optical communications.