Rethinking Neural Thickets Without Weight Perturbations
The Neural Thickets hypothesis posits that pretrained language models lie in dense neighborhoods of task-specific experts in weight space, which random Gaussian perturbations can uncover. We reproduce the reported gains, but show that they do not require this geometric explanation. Perturbing a single decoder block alr...
Nicolas Dufour, Elouan Gardes, Amil Dravid et al.
· 0 citations