Capability, Reliability, and Generality: A Case Study in Multi-Model Elicitation and Its Confounds

Abstract

A qualitative case study in which three frontier large language models (SuperGrok 4.6 Expert, Claude Opus 5.5 High, GPT-5.6) answered the same seven questions about whether progress in frontier LLMs could become decoupled across capability, benchmark performance, reliability, real-world usefulness and properties relevant to robust general intelligence. Each model was blind to the others' responses. The purpose is hypothesis discovery, not confirmation: agreement among language models is not evidence of truth. The paper separates model-generated hypotheses, cross-model recurrence, independently verified evidence, mechanisms, causal claims and falsifiable predictions, and checks every model-supplied citation or figure against primary sources. Five decompositions recurred in all three models. Each was seeded by the questions; the models contributed operationalisations, such as capability ceiling vs. reliability floor, measuring discrimination rather than rate, and separating tools that check from tools that act. The strongest defensible proposition is that frontier capability gains do not logically entail proportional gains in robust general competence; whether they fail to do so empirically is open. This study is not evidence that decoupling is occurring. The paper states ten hypotheses with counterarguments and proposes nine falsification experiments. A second round adds a cross-review, clean-context reruns and a control question. Anonymisation did not hold: reviewers recognised their own responses (7/7, 6/6, 7/7), including GPT-5.6 with memory off; Grok with memory off scored 0/7, consistent with recall rather than style. Several model-specific themes depended on the context of the run rather than on the weights (e.g. evaluation awareness in Grok: 25 mentions in the original run, 0 in the rerun; account memory and live X search could not be separated). In a control question, all three models answered correctly in 15 of 15 fresh sessions, although two of them had implied the opposite in their essays. The paper documents the full provenance of the hypothesis, including prior joint brainstorming between the researcher and the models published on X (@YaffFesh), and argues that in-context and clean runs are complementary arms of one instrument. New in version 1.1: corrected title (the study was blind only to other models' responses); corrections to two worked examples, the motif counts and the attribution of the rerun result; a disclosed confound (possible live X search); four newly verified sources; a reading note in Supplement S1. All changes are listed in the section "Changes in version 1.1". Version 1.0 remains available at doi:10.5281/zenodo.22980932. Files: main paper (PDF); Supplement S1 (first-round transcripts); Supplement S2 (second round: cross-review, reruns, control, X-post context); LaTeX source with transcripts and analysis data. Author profiles: X https://x.com/YaffFesh · Bluesky https://bsky.app/profile/yafffesh.bsky.social