Beyond Token Scale: Chunk-Level Sparse Autoencoders for Reliable Semantic Feature Discovery
A family of chunk-level SAEs that encode mean-pooled activations over chunks, each a contiguous span of tokens: Mean-Chunk reconstructs the observed chunk, Cross-Chunk predicts an independently processed neighbor, and Joint-Chunk combines both targets are introduced.
Xu Wang, Yi-Fan Yang, Ting-Ting Yu et al.
· 0 citations