Implicit Bias of Gradient-Based Learning Under Non-Convex Constraints.

Abstract

Gradient-based learning under non-convex constraints exhibits a notable phenomenon: Despite the existence of many equivalent global minimizers, optimization algorithms consistently converge to a small subset of structured solutions. This behavior, known as implicit bias, remains insufficiently understood in constrained and non-convex settings. In this article, we investigate the mechanism of implicit bias induced by projected gradient-based optimization over general non-convex feasible sets. By modeling projected gradient descent as a continuous-time dynamical system, we derive a projected gradient flow characterized by tangent and normal cone decompositions, which capture the local geometry of the constraint set. Based on this formulation, we show that constraint geometry continuously filters gradient directions along the optimization trajectory, leading to a trajectory-dependent implicit regularization effect without modifying the objective function. We further formalize this effect through a cumulative normal projection energy functional and prove that the optimization dynamics converge to solutions minimizing both empirical risk and geometric incompatibility with the constraint set. Extensive experiments on synthetic and real-world datasets validate the theoretical predictions, demonstrating consistent alignment between solution geometry, optimization trajectories, and generalization performance. These results provide a unified geometric and dynamical explanation of implicit bias in constrained learning systems.