A comparative study of evolutionary algorithms and activation functions in neuroevolution deep reinforcement learning

Abstract

NeuroEvolution is scalable alternative for gradient-based deep reinforcement learning, which provides parallel, gradient-free optimization that overcomes backpropagation’s limitations. However, the selection of the neural network activation function has a significant impact on the stability and rewards of evolutionary optimization. In this paper, we investigate four evolutionary optimization algorithms and six activation functions, resulting in 24 different configurations, and evaluate their performance on the Inverted Double Pendulum environment. The investigated algorithms are NoiseReuseES, PersistentES, PGPE, and ARS, while the evaluated activation functions are GELU, ReLU, SELU, CELU, tanh, and SiLU. The results demonstrate significant interaction effects where one algorithm consistently exhibits stable learning across most activation functions; while another achieves high final rewards yet suffers from mid-training collapses; the third shows high sensitivity to activation choice, ranging from smooth improvement to severe oscillations; and the fourth remains unstable across all tested settings, although certain activation functions reduce its volatility. Among the evaluated configurations, ARS–GELU achieved the highest final reward, followed closely by PersistentES–GELU and PersistentES–ReLU, while PGPE–SELU, PGPE–CELU, PGPE–tanh, ARS–ReLU, and NoiseReuseES–GELU also achieved competitive final rewards. Based on these results, the evaluated evolutionary approaches achieved high final rewards within the configurations and evaluation setting considered in this study.