Mitigating Intra-Task Policy Collapse in Proximal Policy Optimization (Ppo) Via Rollout Buffer Stabilization: a Case Study on Bipedalwalker Continuous Control

Abstract

Dynamic balance control for bipedal robots in continuous action spaces is a complex problem in deep reinforcement learning. While Proximal Policy Optimization (PPO) is a standard algorithm for such tasks, it can suffer from intra-task policy collapse in episodic environments. This study investigates the effect of rollout buffer scale and algorithmic implementation on long-term policy retention through multiseed ablation in BipedalWalker-v3. We compare dynamic perepisode updates against raw PPO with fixed-horizon buffers (1,000-8,000 timesteps) and the optimized Stable Baselines3 (SB3) default (2,048 timesteps). As a key methodological contribution, we propose the Normalized Catastrophic Forgetting Severity (nCFS) metric to quantify intra-task policy degradation. Empirical results show that raw PPO, regardless of buffer size, suffers severe collapse (below 2% post-convergence stability over 10 M timesteps), while SB3 converges $10 \times$ faster with higher retention. These results reveal a significant “implementation gap”: codelevel algorithmic optimizations appear to influence long-term stability more than rollout buffer sizing alone.