Language models suffer from a curse of ambiguity

Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures

As Large Language Models (LLMs) grow more capable across diverse tasks, their (in)ability to generalize remains difficult to quantify and poorly understood beyond limited domains. In particular, LLMs are known to struggle generalizing multilingually, to languages outside of English, and that are poorly attested in their training data. To understand why this may be, and what enables some models to perform better than others, we turn to a long history of work across the cognitive sciences, arguing that successful generalization derives from appropriate representations in similarity space. We look at how well LLMs'representations capture the hierarchical similarity structure between distinct languages. Strikingly, we show LLMs'latent representations largely recover the hierarchical structure of the Indo-European language family tree -- grouping languages that are members of the same subfamily closely together in representation space. Furthermore, we show that the degree to which models reflect the similarity structure of languages correlates with their performance on XNLI, a multilingual natural language inference benchmark. This extends classic work on similarity-driven generalization at scale, showing how models that represent similar languages similarly generalize better from one language to another.

The Pitfalls of Maximum Likelihood Training of TPMs for Controllable Language Generation

Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling

Language models encode substantial factual knowledge in their parameters, which can lead to unreliable behavior when this knowledge is outdated, incomplete, or misaligned with the provided context. In this work, we study whether modifying the pretraining signal can systematically shift models away from parametric recall and toward evidence-grounded reasoning. We introduce Knowledge--''Less''Language Models (KLLMs), a fundamentally different epistemic training paradigm for LLMs, which are pretrained on corpora in which named entities are anonymized, thereby removing a primary channel for entity-linked factual supervision. This intervention substantially reduces closed-book factual recall, while often improving performance on tasks where relevant information is provided as context. Across multiple model scales, KLLMs consistently outperform matched baselines on contextual question answering, fact verification, and hallucination detection benchmarks. Crucially, in retrieval-grounded settings with imperfect evidence, KLLMs show improved robustness and achieve up to 20--25\% relative gains over standard language models. They further exhibit better calibration, with improved ECE, Brier score, and AUROC, as well as more reliable abstention behavior. Our results demonstrate that suppressing entity-linked supervision during pretraining induces a shift in epistemic behavior: KLLMs rely less on parametric knowledge and more on external evidence, leading to improved reliability under realistic conditions. This suggests that pretraining-time control over knowledge acquisition can complement retrieval-augmented and tool-based systems by providing a more evidence-sensitive base model.

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

A large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization.

All too perfect: bias and aspiration in persona generation with LLMs

Synthetic data generated by large language models plays a central role in the training and alignment process of other AI systems. However, this process also risks inheriting the structural biases of organic corpora and embedding new biases that stem from the design choices underlying the data creation process. This paper examines the systematic biases that emerge when large language models (LLMs) are tasked with generating synthetic personas. We introduce a reproducible, minimally conditioned pipeline that produced 40,000 personas, in four different languages, using two instruction-tuned open-weight generators (Llama−3.3-70B-Instruct and Qwen2.5-72B-Instruct), followed by a battery of quantitative analyses: name match-rates, KL divergence/skew for gender, age-pyramid comparisons, profession-gender intersectionals, adjective/sentiment profiling, and Proppian role classification. Our main findings reveal that persona generations are far from neutral. Models tend to focus on middle-aged, aspirational, and overwhelmingly positive (i.e., upbeat/optimistic) characters, while non-binary identities and many real-world occupations remain underrepresented. We conclude that contemporary training and alignment regimes produce a form of narrative sanitization that both flattens representational diversity and embeds normative assumptions, and propose that persona-based evaluation can serve as a scalable diagnostic of what generative systems value and prioritize when depicting humanity.

Spectral Signatures of Large Language Models

The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation. However, task-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures. To address this challenge, we adopt spectral shape -based metrics for managing and quantifying LLMs based on Heavy-Tailed Self-Regularization theory. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model. This signature captures intrinsic properties of pretrained models and remains robust during post-training, making it suitable for model-level analysis. In addition, this metric is data-free, computationally-efficient, and scale-invariant, enabling large-scale analysis in practice. Moreover, we curate a large and diverse model corpus consisting of major open-source LLM families, and use it to systematically benchmark spectral and non-spectral metrics across models and downstream tasks. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections. The code is available at https://github.com/Ingrid-505/Spectral_Signature.