Morphological Hijacking in Frozen Language Models: A Contrastive, Symmetry-Regularized Projection Head for Algebraic Structure Recovery
Abstract
Preprint. Not yet peer-reviewed. Frozen autoregressive language models cluster surface-similar tokenstogether even when a stronger, task-relevant structure is available inthe input. This paper documents "morphological hijacking" — near-totalcollapse of algebraic-structure recovery under adversarial surfacecorrelation — across four frozen model families (GPT2-small, Pythia-410m,Mistral-7B-v0.3, Qwen2.5-7B), traces it to representational anisotropy,and introduces a lightweight, trainable projection head (contrastiveobjective + positional-symmetry penalty + rogue-dimension ablation +Soft-PCA initialization) that substantially recovers the structure. Theresult is validated at both the discrete-clustering level (Adjusted RandIndex) and the continuous embedding-geometry level (margin, win rate)across five architecturally diverse models, including two models trainedexplicitly for embedding/retrieval tasks, and further tested on naturalEnglish words and a freshly generated, 10x-larger constructed lexicon.Several negative results are reported alongside the positive ones,including a rejected "globally architectural anisotropy" hypothesis anda rejected layer-selection heuristic. This is a preliminary, honestly-scoped empirical report, not a claim ofgeneral applicability. Limitations, a full pre-submission checklist, andcomplete reproducibility code are included. Author: Reza NirouyarORCID: 0009-0000-4690-6842Contact: contact@varzin.orgProject website: https://varzin.org
