Jun 2026· arXiv.org· Vol abs/2606.27237· 0 citations· 38 references
Computer Science
TL;DR
The findings suggest that what the model knows and how it is asked are intertwined in parameter space, undermining the "knowledge base"alogy and carrying implications for the reliability and controllability of factual knowledge in LMs.
Abstract
Language models (LMs) capture large amounts of factual knowledge applicable to a wide range of tasks, motivating the view of their parameters as a knowledge base. An important property of knowledge bases is that different queries for the same fact return consistent results, drawing on a single source of truth. We investigate whether LMs satisfy this property through behavioral and mechanistic analyses. Our results suggest that they encode knowledge in a task-specific manner. Behaviorally, facts acquired on one task frequently fail to co-emerge on others during training. Parameter localization experiments suggest a mechanistic explanation, revealing distinct parameter subsets underlying different tasks for the same fact. Finally, we show that chain-of-thought reasoning draws part of its effectiveness from engaging task-specific parameters beyond those tied to the evaluation task. Our findings suggest that what the model knows and how it is asked are intertwined in parameter space, undermining the"knowledge base"analogy and carrying implications for the reliability and controllability of factual knowledge in LMs.
Despite the success of Large Language Models (LLMs) on reasoning benchmarks, it remains unclear whether their performance stems from genuine logical deduction or the memorization of training patterns. Existing benchmarks often fail to disentangle reasoning from prior knowledge, as tasks grounded in real-world facts allow models to take ''knowledge shortcuts''. In this paper, we propose a novel diagnostic benchmark to decouple knowledge memorization from logical reasoning. Built on the DBpedia KG, our framework constructs multi-hop reasoning chains (from Q1 to Q5) across three task dimensions: Factural Questions (FQ), Counterfactual Questions (CQ) with logically consistent but counterfactual conclusions, and Questions with Similar-Entity Options (SO) to evaluate the dependence on prior knowledge. Questions without Context serve only as an intermediate form: they contain solely queries with no triples or options, so LLMs cannot answer them directly. Our core hypothesis is that genuine reasoning is demonstrated only when a model follows logical rules despite conflicting prior knowledge. Evaluating seven state-of-the-art LLMs (8B to ultra-large) reveals strong prior knowledge dependence, with performance degrading sharply on counterfactual tasks as reasoning depth grows. This work delineates LLM reasoning boundaries and presents a new paradigm for fine-grained capability assessment.
Fangfei Yan, Jianbo Yao, Michael K. Chen et al.· Proceedings of the 32nd ACM...· 1 citation
As large language models (LLMs) continue to advance, a key challenge remains their tendency to hallucinate, generating fluent yet inconsistent content that lacks factual grounding. Natural language inference (NLI)-based methods, which determine whether one statement can be logically inferred from another, are widely considered the most effective for detecting input-output inconsistencies in LLMs. However, several fundamental questions, such as whether LLMs can identify relevant information to make correct factual inconsistency detections and how different arrangements of the source document affect reasoning, are not discussed in prior studies. To bridge this research gap, we design a new benchmark, KIFI, which comprises 1032 carefully selected instances from the TRUE and ScreenEval datasets, with key information annotated. Using KIFI, we show that LLMs frequently fail to use the appropriate information to make correct decisions. In addition, we find that LLMs tend to make predictions by overemphasizing certain keywords or fragments, a new phenomenon we term "Attend to Fragments". We further introduce a novel token-based permutation method to identify untrustworthy inconsistencies. Experiments show that filtering out these instances improves the overall correlation by 1.3% on the standard TRUE benchmark. The project is available at https://github.com/VibeHPC/attend-to-fragments
Xindi Guo, Zhen Xie, Patrick H. Chen· Annual International ACM SIG...· 0 citations
Generative AI Large Language Models (LLMs) such as GPT-4, Claude, and Gemini are reshaping knowledge work across disciplines. Yet these systems exhibit a puzzling paradox: they can pass rigorous professional examinations while simultaneously failing elementary reasoning tasks. This paper presents a condensed theoretical framework explaining how knowledge is created, stored, and retrieved in LLMs through Stochastic Knowledge Aggregation (SKA) – a process fundamentally different from the Systematic Knowledge Scaffolding (SKS) that characterizes human learning. We introduce 19 foundational concepts, three formal theories, and a set of propositions collectively forming the Jagged Knowledge Frontier (JKF) framework. Empirical cases validate the framework and illuminate implications for Knowledge Management (KM). The paper argues that understanding AI knowledge creation is essential for bridging traditional human KM with the emerging discipline of AI Knowledge Management, and for designing governance structures that account for the inherent incompleteness and inconsistency of LLM knowledge.
T. Nguyen· European Conference on Knowl...· 0 citations
Recent advances in post-training techniques have endowed Large Language Models (LLMs) with enhanced capabilities for tackling complex, logic-intensive tasks through the generation of supplementary planning tokens. This development raises a fundamental question – Are these models aware of what they "learn" and "think"? To address this, we define three core competencies: (1) awareness of learned latent policies, (2) generalization of these policies across domains, and (3) alignment be-tween internal reasoning traces and final outputs. We empirically evaluate these abilities on several tasks, each designed to require learning a distinct policy. Furthermore, we contrast the profiles of models post-trained via Super-vised Fine-Tuning (SFT), Direct Policy Optimization (DPO), and Group Relative Policy Optimization (GRPO). Our findings indicate that RL-trained models not only demonstrate greater awareness of their learned behaviors and stronger generalizability to novel, structurally similar tasks than SFT models but also often exhibit weak alignment between their reasoning traces and final outputs, an effect most pronounced in GRPO-trained models.
Pratham Singla, Shivank Garg, Ayush Singh et al.· Annual Meeting of the Associ...· 0 citations
Large language models (LLMs) have achieved strong performance on text-to-knowledge graph generation and related tasks. Nevertheless, it is still unclear whether they accurately model the direction-dependent semantics of inverse relations, in which reversing the order of the arguments alters the meaning of a relation (e.g., \textit{mother} versus \textit{child}). To the best of our knowledge, this work presents the first systematic study of inverse relation directionality in LLMs, using a benchmark consisting of 5,457 instances spanning 27 distinct inverse relation labels. We evaluate five open-source LLMs under a multiple-choice prompting framework and further examine the influence of relation descriptions and entity representations by substituting the original entities with synthetic and masked entities. Our findings reveal systematic asymmetries in inverse relation classification across LLMs, indicate that relation descriptions do not consistently improve performance, and show that model performance can be sensitive to variations in entity representations.
Language models (LMs) increasingly drive real-world applications that require world knowledge. However, the internal processes through which models turn data into representations of knowledge and beliefs about the world are poorly understood. To facilitate such studies, we present LMEnt, a suite including (1) a knowledge-rich pretraining corpus, fully annotated with entity mentions based on Wikipedia, (2) an entity-based retrieval method over pretraining data that outperforms existing tools by as much as 80.4%, and (3) 12 pretrained LMs with up to 1B parameters and 4K intermediate checkpoints, with comparable performance to popular open-source models on knowledge tasks. Together, these resources provide a controlled environment for analyzing connections between entity mentions in pretraining data and downstream performance. We show the utility of LMEnt by studying knowledge acquisition over training, finding that entity co-occurrence and mention forms—which are difficult to study with existing tools—affect learning trends. Moreover, as LMs form stronger associations between entities, their facts are harder to edit in-context, whereas inconsistencies in model predictions over training are indicative of editing success. We release LMEnt to support studies of knowledge in LMs, including knowledge representations, plasticity, editing, attribution, hallucinations, and learning dynamics.
huggingface.co/LMEnt
github.com/LMEnt
Daniela Gottesman, Alon Gilaie-Dotan, Ido Cohen et al.· Transactions of the Associat...· 0 citations