Traversing the solution space of neural networks with Hessian Null Space Continuation
On a single task, deep neural networks can learn a wide range of solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these solutions are surprisingly mode-connected: rather than isolated points in the weight space, they are connected by low loss regions. Despite this obser...