Skip to content

Where Decoder Cosine Similarity Fails for SAE Feature Flow Discovery

Sep 2026 · 0 citations · 13 references
Computer Science

TL;DR

A transition atlas of triples is constructed, where a residual-state feature and an MLP-update feature jointly predict a target residual feature, and validate candidate triples by ablating the decoded update feature.

Abstract

Foundation models are increasingly adapted through fine-tuning, model editing, and alignment procedures while retaining previously acquired capabilities. Understanding the internal computations that support these adaptations is therefore becoming increasingly important for continual model evolution. Sparse autoencoders (SAEs) provide interpretable feature dictionaries for residual-stream activations and sublayer outputs, but it remains unclear how state features and update features interact to produce downstream residual features. In this work, we focus on MLP updates as a first test case. We construct a transition atlas of triples $s_k + u_j \rightarrow t_\ell$, where a residual-state feature and an MLP-update feature jointly predict a target residual feature, and validate candidate triples by ablating the decoded update feature. In a 20M-token Pythia-160M $L_7 \rightarrow L_8$ run, we find 38,125 strong ablation-effect transitions, but 88.0% have both state-target and update-target decoder cosine similarity below 0.7. As a preliminary cross-model check, a run of 20M-token Gemma-3-4B $L_{21} \rightarrow L_{22}$ causally validates only the top 30,000 ranked candidate triples by ablating the decoded update feature, and 53.6% of strong-effect triples have both state-target and update-target decoder cosine similarity below 0.7. The Gemma result is directionally consistent with Pythia, but weaker, since update-target cosine recovers many of the strongest Gemma effects and the run is not a full-atlas causal validation. Ultimately, our results suggest that feature flow atlases can serve as diagnostics of representation-update mechanisms and thereby inform tools for steering model updates. Future work will validate more complex patterns across layers, models, and SAE families.

View source

Similar papers

Architecture Alignment With Sparse Priors in Tabular Foundation Models

Together, these results support architecture-prior alignment: preserving an addressable feature axis provides an inductive bias for task-adaptive relevance inference for task-adaptive relevance inference.

Tian-Qi Zhao, Tian-Yi Zhuang, Shuo Duan et al. · 0 citations
#artificial intelligence Preprint Oct 2026

ScopeSAE: Model-Scope Feature Discovery with Interpretable Layer Selection

Sparse autoencoders (SAEs) are a central tool in mechanistic interpretability. However, existing SAEs are primarily trained per layer. The modeling subspace is therefore fixed by layer identity, independent of which token-layer states actually drive each prediction. We argue that this constraint contributes to several...

Qing-Wen Zeng, Ze-Hao Fu, Shu-Yu Meng et al. · 0 citations
Preprint Aug 2026

Fine-Tuning of Transformer models with Frames

The experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.

Harshavardhan Adepu, Li Zhang, Sanjiv Kumar et al. · 0 citations
#artificial intelligence Preprint Sep 2026

PULSE: Identifying Demonstration-Utility Features with Sparse Autoencoders

This work introduces PULSE (Paired Utility Localization over Sparse Encodings), an SAE-based framework for identifying model-internal features associated with demonstration utility and using them for demonstration selection, and PULSE-Retriever, which converts its magnitude into a feature-relevance mask for scalable po...

Chen-Duo Hao, Chuan-Bao Gao, Ping Zeng et al. · 0 citations

Output-aware Residual Stream Pruning for Large Language Models

This work characterize the effect of a residual-stream perturbation through both its activation covariance and the local sensitivity of the model output, and derives a tractable spectral upper bound that reduces subspace selection to an eigendecomposition of a sensitivity-weighted covariance matrix, retaining the effic...

Chayne Thrash, Ke Chen, Soheil Kolouri · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.