Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction
Gate thresholds are implemented as couplings to a constant the network builds for the distribution it is reading, with the bias parameters contributing little; how much of that constant is carried in the stream and how much in parameters depends on the architecture.