NeST: Neuron Selective Tuning for LLM Safety

Abstract

NeST: Neuron-Selective Tuning for Efficient Safety Alignment A lightweight and structure-aware safety alignment framework that selectively adapts safety-relevant neurons to strengthen refusal behavior while preserving the model's general capabilities. 🚀 Overview Safety alignment is essential for reliable deployment of Large Language Models (LLMs), but existing approaches often require costly model-wide fine-tuning or introduce additional inference-time overhead. Parameter-efficient methods such as LoRA improve efficiency but can provide inconsistent safety gains, while intervention-based methods do not directly shape the model's internal safety representations. NeST addresses these limitations by selectively tuning a small set of safety-relevant neurons while freezing the rest of the model. NeST groups functionally coherent safety neurons into clusters and learns shared updates within each cluster, enabling targeted, stable, and efficient safety adaptation. We evaluate NeST across 14 open-weight LLMs spanning multiple model families and sizes, achieving: 1.1% average ASR, down from 44.5%, corresponding to a 97.5% reduction in unsafe generations. 5,800× fewer trainable parameters than full fine-tuning and 3× fewer than the lowest-rank LoRA baseline. 1.1% average ASR on multimodal models, down from 55.3% across text, image, and reasoning-augmented settings. 0.8% average ASR on downstream-adapted models, down from 53.8%, demonstrating effective post-hoc safety restoration. 🧪 Key Contributions Neuron-Selective Tuning for lightweight, post-hoc safety alignment by modifying only safety-relevant neurons while freezing the remainder of the model. Cluster-Based Safety Adaptation that groups safety neurons by activation similarity and learns shared updates within each cluster for structured and stable refusal behavior. Extensive Evaluation across 14 open-weight LLMs, showing strong safety robustness with substantially fewer trainable parameters while preserving core reasoning and knowledge capabilities. Multimodal Safety Alignment across text-only, image-only, and reasoning-augmented generation, demonstrating robust safety improvements across diverse input modalities. Post-Hoc Downstream Hardening by transferring safety-relevant neuron clusters from base models to downstream variants, efficiently restoring safety after task-specific fine-tuning.