Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders
KronSAE is proposed, a design that factorizes the latent space into heads and forms post-latent features as pairwise compositions of lower-dimensional pre-latents using mAND, a differentiable AND-like interaction that imposes a compositional co-activation prior while remaining compatible with standard SAE objectives and variants.
Vadim Kurochkin, Yaroslav Aksenov, Daniil Laptev et al.
· 1 citation