R. Krause

Weight initialization shapes task organization in recurrent neural networks

It is found that networks with low initial weight variance converge to solutions in which different subtasks rely on largely overlapping weight subspaces, whereas high-variance networks implement subtasks in higher-dimensional, more segregated weight subspaces.