Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models
Although $\mathbb{E}[\mathrm{dropout}(x)] = x$, here we show that $\mathbb{E}[\mathrm{LayerNorm}(\mathrm{dropout}(x))]$ is not equal to $\mathrm{LayerNorm}(x)$. Accordingly, the pattern of a Dropout layer followed by a LayerNorm, which is common to many AlphaFold2-based protein structure predictors, produces a systemat...
Isaac Ellmen, David Errington, Matthew I. J. Raybould et al.
· 0 citations