Exploring the Abstention Ability of LLMs using Scientific Reasoning in Critical Fields

Abstract

Large Language Models (LLMs) can produce fluent and confident responses even when those responses lack sufficient support or contain factual errors. Reliable question answering therefore requires models not only to generate answers, but also to recognize when they cannot provide a well-supported response and abstain accordingly. This capability is particularly important in scientific domains, where inaccurate responses can cause real-world harm. This paper systematically investigates the abstention behavior of LLMs in challenging scientific multiple-choice answering tasks. We evaluate several LLMs on multiple modified versions of a scientific multiple-choice question answering dataset, varying both the availability of the correct answer and the wording of the abstention option. Our findings demonstrate that LLMs often struggle to abstain consistently when the correct answer is absent, and that their abstention behavior is sensitive to model family, abstention wording, and reasoning settings.