Skip to content
#small language model Book Open access

Direct Answers, Hidden Criteria: Empirical Trade-offs in LLM Responses to Underdetermined Prompts

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 9 references

TL;DR

Evaluating 140 manually curated underdetermined prompts across seven categories on six LLMs suggests that asking for the missing criterion is a practical and transparent strategy for LLM systems facing under-specified user requests.

Abstract

Large language models are increasingly used in question answering, recommendation, and conversational decision support. In many such settings, the user asks for a single answer even though the prompt does not specify the criterion needed to justify a unique choice. We study this setting as semantic underdetermination: multiple answers remain compatible with the prompt unless an additional ranking criterion is supplied. Rather than claiming a formal impossibility result, we study an empirical tension among three properties: premise-entailment correctness, criterion neutrality, and single-turn decisiveness. We evaluate 140 manually curated underdetermined prompts across seven categories on six LLMs, from 4B to frontier scale, under four response policies that handle under-specification differently. Forcing a direct answer makes models far more decisive (decisive rate 32%?93%) but lowers judged utility (4.06?3.53 out of 5, a large effect), because a confident answer must commit to a criterion the user never gave. Asking for the missing criterion instead gives the highest judged utility (4.53/5) and LLM-predicted satisfaction (4.48/5) but is rarely decisive in one turn (5.4%). A criterion-sensitivity score supports the mechanism: differences are small in size, but every one of the six models is most criterion-sensitive when forced to answer and least when it asks for the criterion. These findings suggest that asking for the missing criterion is a practical and transparent strategy for LLM systems facing under-specified user requests.

Read PDF

Similar papers

#artificial intelligence Preprint Oct 2026

Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing

LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument. Prior work has primarily explored deliberate prompt modifications such as scoring rubrics, role descriptions, or chain-of-thought reaso...

Ilija Subasic, Brett Levert, Andrew Rabinovich et al. · 0 citations

Likelihood Ranking doesn't Scale Like Prompting in LLMs

It is suggested that likelihood preferences over controlled declarative alternatives and task-conditioned answer selection probe distinct aspects of model behavior, and should not be treated as interchangeable.

Alessandro Bondielli, Lucia C. Passaro, D. Bacciu et al. · 0 citations
Conference Open access Nov 2026

Don’t Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations

When user queries are incomplete, large language models should ask targeted questions before committing to an answer, a capability we call proactive elicitation, yet evaluation on real user-assistant queries across seven LLMs (8B to frontier) shows that fewer than 11% of underspecified queries trigger a follow-up quest...

Rong-Yu Zhang, Mu-Tian Bao, Jin-Jian Zhang et al. · 0 citations

Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling

Across eight text, image, and video benchmarks and eighteen models, pooling is harder for every model, the elimination credit is largest for the weakest models, and seven of eight open-weight models answer 87 to 100% of unanswerable questions.

Mohamed Eltahir, Abobaker Ahmed, Nawaf Barebood et al. · 0 citations

Intrinsic Sequence-Likelihood Confidence in Retrieval-Dominated Extractive QA: Two Pre-Specified Negatives, and What They Do and Do Not Attribute

In extractive document question answering whose questions were generated from the passages that contain their answers -- so that retrieval recovers 92-99.8% of what any mode combination could reach, whatever its absolute accuracy -- confidence-driven mechanisms have little to gain. Fine-tuning an open language model on...

Gun-Woo Lee, Chang-Min Sung, Sang-Hwan Gwak et al. · 0 citations
Open access Oct 2026

Risk-Calibrated Prompting: Selective Answering for Robustness Under Prompt Perturbations

Risk-Calibrated Prompting is presented, an inference-time selective answering framework that combines semantic consistency across controlled prompt perturbations with model-verbalized confidence and explicitly separates uncertainty signaling from factual truth.

Neelu Verma · 0 citations

Related blog posts

Google DeepMind Blog Nov 18, 2026

Genie 4 is now available

We are rolling out a new Project Genie experience to eligible Google AI Ultra subscribers, opening access to our new Genie 4 world model.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.