Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset

Abstract

Humanity's Last Exam (HLE) is widely used to evaluate frontier language models. HLE organizes its questions into eight subject-domain categories, whose subscores are often interpreted as evidence of distinct capabilities. However, no study has assessed whether these labels correspond to empirically separable latent constructs, nor whether the benchmark effectively differentiates between models of similar ability. We evaluate 29 LLMs on the text-only multiple-choice subset of HLE and apply psychometric methods to assess both the dimensionality of the benchmark and the distribution of its measurement precision. Fitting a two-parameter logistic IRT model, we find that HLE is dominated by a single general reasoning factor: domain labels explain only 3.5\% of variance in the leading principal components of item responses, and within- and between-domain residual correlations are nearly identical (Cohen's $d = 0.016$). A simulation study with the same number of models ($N = 29$) shows that our analyses would detect clearly distinct domain abilities, so their absence is informative; however, small domain-specific differences cannot be ruled out, and model rankings vary across domains somewhat more than a single factor predicts. A separate analysis of the test information function reveals that measurement precision concentrates at moderate ability levels and drops sharply above $\theta = 0$, where the strongest models sit. These findings suggest that the subset's domain subscores do not warrant distinct capability interpretations and that its ability to discriminate among the strongest models might be limited.