Skip to content

Structured Output Collapses Answer Diversity Across 44 Language Models

Jul 2026 · arXiv.org · Vol abs/2607.18476 · 0 citations · 9 references
Computer Science

TL;DR

When a language model must choose one answer from a large space of equally valid options, a format clause --"Reply with JSON only"-- changes which answer it chooses, and structured output is how software consumes language models.

Abstract

When a language model must choose one answer from a large space of equally valid options, a format clause --"Reply with JSON only"-- changes which answer it chooses. We re-run the One-Word Census (arXiv:2607.12796): 31 wide-answer-space category prompts asked of 44 models, now with the reply requested in JSON -- no schema enforcement, no constrained decoding, only the request. Convergence deepens sharply: on the unconstrained"Pick a word"prompt the modal answer rises from 41% to 64% of the pool and distinct answers fall from 52 to 36; mean answer-choice surprisal drops from 1.80 to 1.58 bits. The tax is progressive: six of 44 models move individually (BH-FDR q=.10), all toward the mode, led by the most distinctive models, while the conformist floor is immobile. It is a sharpener, not a re-indexer -- the plain-chat modal answer survives in 28 of 31 categories. Defaults are register-indexed: a within-run re-sample (n=20) finds JSON shifts 53% of a model's stable chat defaults, mostly back to the crowd, and installs defaults absent from chat (Claude Fable 5 answers"cerulean"for colour 0% of the time in chat, 100% in JSON). Full-battery controls reveal a register gradient: compression is significant and specific to the answer-delivery formats models are trained to speak (JSON -0.22 bits, p=.0002; XML -0.19, p=.002), absent for YAML and CSV, and reversed for an arbitrary bracket wrapper (+0.13, p=.009) -- weighing the mechanism toward tool-use post-training. Enforcing the schema at the decoder (response_format) compresses no further than the request (-0.03 bits): the collapse lives in the model's response to the register, not the decoder. Structured output is how software consumes language models, and that surface is served by a measurably more homogeneous model than the chat surface on which models are evaluated, compared, and chosen.

View source

Similar papers

#natural language process... Preprint Jul 2026

The One-Word Census: Answer-Choice Conformity Across 44 Language Models

When a language model must pick one answer from a large space of equally valid options, which does it pick -- and how often is it the same answer every other model picks? Asked to"pick a word -- any word,"44 models chose"serendipity"41% of the time. We characterize this convergence with a deliberately minimal instrumen...

Tapan Parikh · 2 citations
Review Jul 2026

Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe

For applications that require per-persona outputs, the same model that cannot sample from a distribution can describe it accurately in a single call, and is proposed Prompt-Perturbed Argyle (PPA), which reduces the same error by 21% at no added cost.

Chaemi Jang, Dong-Man Lee, Ji Hee Kim · 0 citations
Jul 2026

Where Does the Noise Come From? A Variance-Components Decomposition of Non-Determinism in LLM Brand Answers

A crossed random-effects (generalizability-theory) decomposition is specified that partitions the total variance of a response-level brand outcome into these four sources, and embeds the components in a decision-study allocation that returns how many repeats, paraphrases, models, and languages to buy for a target relia...

D. Żatuchin · 3 citations · ⚡1
Preprint Aug 2026

Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap

Multilingual evaluations report accuracy at a single output-token cap, but languages need different numbers of tokens to express the same content, so the cap is a hidden experimental variable. We test whether the native-vs-translate gap on MGSM (German, Thai, Swahili) is a token-budget artifact for Qwen3-8B and Llama-3...

Ankit Goyal, Jaideep Ray · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.