Calibrating Lightweight Sparse Autoencoder Feature Steering
TL;DR
This work introduces ContrastSteer, a complementary heuristic that measures whether post-steering activations emphasize weakly aligned features, thus indicating off-target or nonsensical outputs, and introduces contamination, a complementary heuristic that distinguishes successful steering, under-steering, and over-steering collapse.
Abstract
Sparse autoencoders (SAEs) can enable inference-time topic steering by modifying latent feature activations, but existing steering methods often fail when target-aligned features are not identified or are modified at the wrong scale. We introduce \textsc{ContrastiveSteer} to address these respective failure modes. First, features are scored by how much more strongly they activate on target-domain text than on general text. Second, steering strength is set using a model-specific calibration. We also introduce \emph{contamination}, a complementary heuristic that measures whether post-steering activations emphasize weakly aligned features, thus indicating off-target or nonsensical outputs. In our evaluations, across multiple LLM families and SAEs, \textsc{ContrastiveSteer} improves topic alignment across domains up to 3.9$\times$ over unsteered and steering baselines, and raises target-domain classification by up to 93\%. Our activation-scale normalization method also recovers at least 93\% of optimal domain-score gains across all tested model families, and contamination distinguishes successful steering, under-steering, and over-steering collapse.
