Skip to content

Likelihood Ranking doesn't Scale Like Prompting in LLMs

Sep 2026 · arXiv.org · Vol abs/2609.29390 · 0 citations · 21 references
Computer Science

TL;DR

It is suggested that likelihood preferences over controlled declarative alternatives and task-conditioned answer selection probe distinct aspects of model behavior, and should not be treated as interchangeable.

Abstract

LLM evaluation is commonly performed either by prompting models to produce answers or by scoring candidate outputs with likelihood-based metrics. In multiple-choice QA, however, standard likelihood-based scoring is still conditioned on the question and answer set, and can therefore leverage the same task-conditioned answer-selection interface used in prompting. We study a complementary protocol based on likelihood ranking of declarative statements constructed from the same question--answer pairs. Across 95 decoder-only models, ranging from 0.1B to 104B parameters, and 10 MCQA datasets, we find a systematic divergence between declarative-statement likelihood ranking and prompted answering. Statement-likelihood accuracy remains comparatively stable across scale, whereas prompted answering improves sharply with scale and instruction-tuning. These results suggest that likelihood preferences over controlled declarative alternatives and task-conditioned answer selection probe distinct aspects of model behavior, and should not be treated as interchangeable.

View source

Similar papers

Book Open access Nov 2026

Direct Answers, Hidden Criteria: Empirical Trade-offs in LLM Responses to Underdetermined Prompts

Evaluating 140 manually curated underdetermined prompts across seven categories on six LLMs suggests that asking for the missing criterion is a practical and transparent strategy for LLM systems facing under-specified user requests.

Venkata Satya Satvik Viriyala, Vinu Ellampallil Venugopal · 0 citations

Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation

Constrained-option scoring reads a model's probabilities for a fixed set of permitted answers, so it returns a score even when the model is about to write something else. We study a prompt that quotes a multiple-choice item, ending in the item's own answer instruction, and asks for a forecast of whether a reader model...

Jia-Xuan Guo, Ke-Jia Zhang, Shuo Xin et al. · 1 citation
Preprint Oct 2026

From Generation to Scoring: Scaling LLM-Based Semantic Labeling at LinkedIn

For efficiency, industrial large language model (LLM) labelers often pack many candidate labels into one prompt and generate a YES/NO answer for each, even when each decision depends only on the entity and one candidate. Generating these independent decisions as one answer list couples them, spends decoding on the list...

Jian Sheng, Sriram Vasudevan, Sundara Raman Ramachandran et al. · 0 citations

Predicting Multiple-Choice Question Difficulty and Distractor Selection with Large Language Models

Evaluating large language models’ ability to rank CS1 multiple-choice questions by difficulty and predict which distractors students select suggests that simulating student cognition on multiple-choice questions is genuinely difficult, yet the strong performance of some configurations indicates the problem is tractable...

J. Koff, Lisa Zhang, Juho Leinonen et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.