Preprint

Local and Global Regimes of Geometric Complexity in Language Model Representations

Arwa Osman Marco Baroni Iuri Macocco

Aug 2026 · 0 citations · 20 references

Computer Science

TL;DR

A scale-dependent transition between two ID regimes is found: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses, and conditions with more unique words produce higher ID.

Abstract

Intrinsic dimensionality (ID) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of language itself or artefacts of how the underlying dataset was constructed. In this paper, we focus specifically on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset. We find a scale-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses, and conditions with more unique words produce higher ID. We derive an exact, parameter-free formula for the point at which this reversal occurs, which matches the observed transition point at every scale tested. On the one hand, our results highlight how care must be taken when interpreting the intrinsic dimensionality of a set of representations as a straightforward cue of their complexity. On the other hand, our discovery of the two ID regimes reveals a general principle of organisation of linguistic data in LLMs that sheds new light on their inner manifold structures.

arXiv Semantic Scholar

View source

Similar papers

Jul 2026

Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures

As Large Language Models (LLMs) grow more capable across diverse tasks, their (in)ability to generalize remains difficult to quantify and poorly understood beyond limited domains. In particular, LLMs are known to struggle generalizing multilingually, to languages outside of English, and that are poorly attested in thei...

Supantho Rakshit, Adele E. Goldberg, Henry Conklin · arXiv.org · 0 citations

Preprint Jul 2026

A scaling law of contextual persistence in human language

Human language exhibits lawful structure at the level of words (frequency, vocabulary growth) and word pairs (co-occurrence across distance). Here we show that the arrangement of words in sequence -- a central determinant of meaning -- obeys a comparable law. Using large language models as probabilistic probes, we meas...

E. Barenholtz · 0 citations

Preprint Jul 2026

Geometry of Ordinal Representations in Language Models

It is found that 1D manifolds with place-cell feature tiling emerge for tasks where the ordinal variable is locally computable from token identity, while tasks requiring cross-position integration or semantic extraction produce higher-dimensional or incoherent representations.

Saksham Bassi, Sharvi Tomar · 0 citations

Preprint Aug 2026

Divergent large language model predictions from convergent representations in ambiguous word pairs

This work investigates how decoder-only transformers resolve lexical ambiguity through layer-by-layer analysis of three models spanning three parameter sizes, finding that representations become maximally distinct in middle layers, then partially reconverge in late layers, while the KL divergence between their next-tok...

K. Scott, Narun Pat, Veronica Liesaputra · 1 citation

Preprint Aug 2026

Rewriting or Reweighting? A Geometric Account in Language Models

Behavior manifold analysis is introduced, which isolates behavior-specific geometry by selecting sparse behavior-associated coordinates and lifting them into low-dimensional local charts, and provides a unified framework for understanding the mechanistic distinction between the two objectives.

Jun-Tong Wang, Shengkun Yang, Xiyuan Wang et al. · 0 citations

Preprint Aug 2026

Language models suffer from a curse of ambiguity

This work identifies a curse of ambiguity: in large language models, and more broadly in all neural networks that produce discrete probability distributions, the more ambiguous a next-token distribution is, the harder it is to learn accurately.

Nicolas Zucchet, Hyun Dong Lee, Scott W. Linderman · 2 citations