Skip to content

Don't Forget Your Embeddings: Robust Knowledge Erasure via Precise Editing of Embeddings

Jun 2026 · arXiv.org · Vol abs/2606.03695 · 0 citations · 66 references
Computer Science

TL;DR

This work establishes that precise embedding-level intervention is necessary for robust concept erasure, and demonstrates that existing methods can benefit from such augmentation, with minimal coherence loss.

Abstract

As language models are increasingly deployed in real-world applications, the ability to erase specific knowledge from them becomes critical for safety and compliance. Prominent methods seek persistent removal by updating the model's parameters, yet the target knowledge often can be recovered through adversarial prompting or relearning. In this work, we hypothesize this limitation stems in part from existing methods overlooking the embedding layer. To address this, we introduce EMBedding ERasure (EMBER), a plug-n-play erasure module that leverages Sparse Matrix Factorization for precise erasure of concept-related features from token embeddings. Through comprehensive evaluations across diverse concepts on Gemma-2-2B-it and Llama-3.1-8B-Instruct, we find that augmenting existing methods with EMBER consistently improves erasure efficacy and specificity across task formats, with minimal coherence loss. Moreover, it dramatically improves robustness to relearning, reducing regained accuracy by up to 50%, limiting it to 35% on Llama compared to 70%-76% for prior methods. Further analysis shows that the coherence cost is localized, affecting only a small set of concept-exclusive tokens. Our work establishes that precise embedding-level intervention is necessary for robust concept erasure, and demonstrates that existing methods can benefit from such augmentation.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup

Mixture of Memory Embeddings (MoME), a context-aware memory mechanism that replaces each token's single memory row with a mixture of M slots and uses a learned gate over the hidden state to choose which slots to read at each position, shows more promising memory-size scaling trend at sub-billion scale and remains effic...

Mu-Chen Li, Leonid Sigal, Ren-Jie Liao · 0 citations
#large language models Review Open access Sep 2026

PALRec: Large Language Model-Based Sequential Recommendation with Parameter-Preserving Augmentation

PALRec is proposed, a parameter-preserving augmentation framework that equips an LLM with recommendation capabilities while keeping its original parameters fixed and consistently outperforms fully fine-tuned counterparts in recommendation accuracy while preserving the LLM’s pre-trained knowledge.

Hyunsoo Na, Minseok Gang, Sang-goo Lee et al. · 0 citations
Preprint Sep 2026

Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility

This work introduces Multi-modal Knowledge Preserving Adapter (MKP-Adapter), the first adapter-only BCT approach for Multi-modal Large Language Models (MLLMs) that requires no backbone updates and achieves strong backward compatibility across diverse multi-modal benchmarks.

Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu et al. · 0 citations

Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions

RegEXROUTE is introduced, a pipeline that uses sparse autoencoders (SAEs) to discover interpretable regular-expression (regex) features and establishes explicit, interpretable text features as a practical basis for designing and understanding LLM routers.

Yi-Fan Lu, Qi-Yue Zhang, Haotian Shan et al. · 0 citations
Preprint Aug 2026

Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

This work introduces Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving, and trains the compact model using a dimension-agnostic objective that aligns teacher and student similarity distributions.

Egor Kolodin, Egor Krasnoperov, Evgeniy Kosarev et al. · 0 citations

ALOE: Semantically Addressed Low-Rank Operators for Knowledge Editing

Knowledge editing changes what a model knows by modifying parameters so that a requested fact updates while unrelated behavior is preserved. This is usually treated as a write problem, but editing also involves an address problem: deciding which hidden states should receive the new residual. An update that activates to...

Zeyan Li, Hu Xu, Jian-Feng Xu · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.