The results show that models often do not affirm culmination but nevertheless accept the corresponding simple-past hypothesis, a pattern the authors characterize as Sufficiency Bias, and show that prompting interventions produce a Decision Shift among labels without reliably improving the underlying semantic understanding and reasoning.
Abstract
The imperfective paradox provides a useful test of compositional semantic analysis. Recent work constructs an NLI benchmark and reports that models frequently infer completed telic events from progressive descriptions, attributing this behavior to a Teleological Bias. It further argues that prompting interventions cause a Calibration Crisis. We reexamine the benchmark and conclusions and show that it is substantially affected by conceptual and evaluation mis-specifications. We identify three conceptual mis-specifications. In particular, Aspectual Reduction affects the benchmark construction, analysis, experiments, and conclusions. Under a strict NLI standard, 76% of Group A instances do not explicitly rule out culmination. In our native-speaker annotation, 38% of Group A examples and 29% of the Group C examples were judged to permit an alternative interpretation. To control these issues and lexical variation, we construct Lexically Matched Minimal Pairs. At the evaluation level, we formulate event-semantic NLI as a Multi-step Reasoning Problem and assess both intermediate semantic decisions and final predictions. Our results show that models often do not affirm culmination but nevertheless accept the corresponding simple-past hypothesis, a pattern we characterize as Sufficiency Bias. We further show that prompting interventions produce a Decision Shift among labels without reliably improving the underlying semantic understanding and reasoning. Intermediate and oracle-guided analyses identify two additional failure modes: errors in compositional aspectual classification and Surface-form Attraction toward surface-associated answers. Our experiments on Qwen-7B with suitable prompts, GPT-5.4, and Qwen-72B provide initial evidence for the context sensitivity of aspectual classification and suggest that these models can achieve performance comparable to that of human annotators.
It is found that while all models show sensitivity to existential presupposition across syntactic embeddings, determiner types and contextual cues, their behaviour differs markedly in strength and systematicity, with NLI-fine-tuned autoregressive models exhibiting the most coherent and stable projection patterns.
Marie-Léontine Wörgötter, Shiyang Lai, Sebastian Schuster· International Conference on...· 0 citations
Whether large language models (LLMs) can perform the abductive leap from evidence to a new system of axioms, commonly referred to as a jump, has recently attracted considerable debate. A prominent position holds that LLMs are structurally incapable of such jumps, while recent studies challenge both its mechanism and empirical evidence. One of the main reasons why the debate remains open is the difficulty of defining the jump precisely enough to test it. In this paper, we attempt to develop a formal account of the jump in four steps and measure the second. These steps ask what the default completion of partial data is, when the constraints exclude it, whether the new structure agrees with later observations, and how successive jumps compound. We define a \emph{jump instance} as a finite extension problem whose constraints exclude the canonical completions given by the Kan extensions and leave one correct completion up to renaming. In this setting, a model with a canonical default performs the second step by producing the correct completion under the constraints. We evaluate fourteen models across three certified families. The canonical completion returns once in $13{,}300$ constrained answers across all runs. Several calibrated models also give the correct completion reliably, including three API models that solve $159$ of $162$ primary chain trials, suggesting that they can jump at this step. We further formalize the third and fourth steps, whose empirical evaluation remains future work. We hope our work paves the path for formalizing and measuring the full jump in the future. The code of the paper is available at https://github.com/EEthanShi/kan-jump-test.
Dai Shi, Xiao-Yu Li, José Miguel Hernández-Lobato· 0 citations
It is shown that logical incoherencies follow from an LLM’s computation of its internal representations, in particular from an LLM’s failure to take account of the different roles that different expressions may play in determining content.
Nicholas Asher, Swarnadeep Bhar· Topoi· 0 citations
CROWN-QA is introduced, comprising CROWN-Synth, a controlled paired core that fixes the question and observed facts while varying only query-relative coverage, and CROWN-Real, a real-document contrast-set evaluation with controlled coverage variants.
Byoungjae Min, Kennedy Edemacu, Sae-Hong Cho et al.· 0 citations
This work investigates how instruction-tuned Transformer models (LLaMA and Mistral) encode discourse relations in English, with a particular focus on the contrasting relations of causation and antithesis, suggesting asymmetric representation of discourse-based reasoning.
A. Bhattacharyya, Shira Wein· SIGDIAL Conferences· 0 citations
Faithfulness to supplied information is not uniform in large language models: some rules, thresholds, or features shape outputs correctly while others do not, with nothing in a model's own explanations distinguishing the two. An industrial classification task motivates this observation, where supplying a classifier's exact decision thresholds improved accuracy while degrading minority-class recall, raising the question of where such failures occur. Because that setting involved a closed model and proprietary data, the same question is examined in two domains: airline baggage fee and tax calculation. Across domains, models reliably handle directly available values but fail when a known fact must be transformed into a new value through a rule-conditioned lookup, points named here as derivation boundaries. Two error patterns recur there: misassignment, in which a valid value is assigned to the wrong side of a rule boundary, and fabrication, in which a value unsupported by the rules is introduced and justified as though retrieved correctly. Layer-wise analysis using the logit lens suggests a mechanistic explanation. Directly available values stabilize early, whereas boundary-dependent values commit later and less stably, with correct alternatives often remaining competitive until the final layers. Faithfulness failures concentrate at derivation boundaries rather than spreading uniformly, a pattern layer-wise analysis links to delayed internal resolution.
Vaishnavi Sreekumar· Journal of Computer and Fore...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
