Abstract Large language models (LLMs) are now embedded in scientific, educational, and governance workflows, with debates centering on their capabilities, mechanisms, and impacts. Yet these debates remain structured by persistent folk theories—intuitive, informal explanatory models that guide attitudes and actions. Deflationary slogans (“just autocomplete,” “stochastic parrots,” and “average of the internet”) and anthropomorphic framings (“emergent agents” and “proto-minds”) each capture genuine features of current systems but mistake those features for the whole. This Perspective proposes a minimal working model of LLM-based systems centered on four distinctions: between pretraining and deployed systems; between the learned distribution and particular samples; among parametric, contextual, and external memory; and between task competence and agency. The model is used to diagnose six misconceptions about LLMs: next-token prediction, regression to the mean, training-data regurgitation, model memory, alignment, and understanding. For each, the analysis identifies what the misconception gets right, which distinctions it conflates, and what follows for capability evaluation, system design, and governance. Applied to publisher AI policies as governance case studies, the framework shows both how policy language can conflate these distinctions and how such errors can be corrected. The model thereby avoids the parrot–mind binary by treating LLMs as simulators of discourse and task performance, offering a diagnostic toolkit for locating and correcting the errors these folk theories perpetuate.
The rapid evolution of Large Language Models (LLMs) has brought unprecedented capabilities across reasoning, coding, and multimodal tasks. However, as performance scales, their opaque ''black-box'' nature raises a critical challenge: How can we trace the origins of emergent intelligence, and more importantly, how can we leverage these internal mechanisms to guide model optimization? This tutorial provides a comprehensive, end-to-end view of LLM interpretability, transitioning from microscopic neural analysis to macroscopic application and deployment. It is systematically organized into five core sections: i) Unlocking the Black Box: We begin with the evolution of LLM interpretability and highlight recent breakthroughs from leading research teams. ii) Methodology: We present a rigorous overview of foundational theories (e.g., mathematical framework for transformer, biological mechanisms in LLMs) and essential methods (e.g., path patching, logit lens, and neuron description). iii) Anatomy of LLMs: Using advanced techniques to decode internal semantic features, neural circuits, and complex behaviors, we interpret how models perform reasoning, factual recall, and in-context learning. iv) Applications: We show how to transfer interpretability insights into actionable improvements across the LLM pipeline, including interpretability-guided data synthesis (data value scoring, corpus filtering, and activation-based data diagnosis). We also present Pinpoint Training and Steering for precise capability gains, and Pinpoint Quantization for extreme low-bit compression with minimal capability loss. v) Advanced Topics: We conclude by exploring how these interpretability paradigms scale and inspire the design of frontier architectures, agentic systems, and thinking models. In this tutorial, researchers and engineers will gain the theoretical frameworks and practical engineering toolkits needed to understand, steer, and efficiently deploy LLMs in real-world production environments.
Wei Zhang, Zhengfu He, Lucia Zhang et al.· Proceedings of the 32nd ACM...· 0 citations
Large language models (LLMs) are increasingly integrated into educational settings, yet their pedagogical reliability remains insufficiently understood. Beyond overt hallucinations, which informed users readily recognize, a subtler failure mode consists of explanations that are coherent, authoritative, and pedagogically plausible while harbouring hidden conceptual flaws, responses we term Socratic traps. This paper introduces SocraticTrap-CS, a publicly available benchmark that probes the capacity of open-weight LLMs to generate such strategic misconceptions on demand. A single structured prompt explicitly elicited three outputs per concept (a correct explanation, an overt hallucination, and a strategic misconception), yielding 735 expert-annotated response segments from seven open-weight models across 35 core concepts in algorithms and data structures, programming languages and paradigms, databases, computer networks, and operating systems. Three domain experts independently annotated each segment using a three-class schema, achieving near-perfect agreement (Fleiss’ κ=0.9487). Because models were explicitly instructed to produce the misconception, the central metric quantifies adversarial instruction-following capacity rather than the base rate of such errors in naturalistic use and should be read as a conservative upper bound on model capability. Under these conditions, compliance reached 91.7% overall (100% for three models; 57.1% for the smallest model, Mistral 7B, whose lower rate plausibly reflects weaker instruction-following rather than greater safety). Expert-judged persuasiveness was moderate to high, errors were predominantly conceptual rather than factual, models differed significantly, and no statistically significant domain-level differences were detected. The benchmark reframes the evaluation of educational LLMs around pedagogical trustworthiness rather than factual correctness alone.
Marijela Miličević, Mia Rovis, Ratomir Karlović et al.· Information· 0 citations
Recent advances in post-training techniques have endowed Large Language Models (LLMs) with enhanced capabilities for tackling complex, logic-intensive tasks through the generation of supplementary planning tokens. This development raises a fundamental question – Are these models aware of what they "learn" and "think"? To address this, we define three core competencies: (1) awareness of learned latent policies, (2) generalization of these policies across domains, and (3) alignment be-tween internal reasoning traces and final outputs. We empirically evaluate these abilities on several tasks, each designed to require learning a distinct policy. Furthermore, we contrast the profiles of models post-trained via Super-vised Fine-Tuning (SFT), Direct Policy Optimization (DPO), and Group Relative Policy Optimization (GRPO). Our findings indicate that RL-trained models not only demonstrate greater awareness of their learned behaviors and stronger generalizability to novel, structurally similar tasks than SFT models but also often exhibit weak alignment between their reasoning traces and final outputs, an effect most pronounced in GRPO-trained models.
Pratham Singla, Shivank Garg, Ayush Singh et al.· Annual Meeting of the Associ...· 0 citations
Large language model (LLM) evaluation spans diverse tasks and benchmarks, yet evidence remains organized around tasks rather than the capabilities they probe. This fragmentation limits cross-study comparison, obscures capabilities tasks recruit, and makes coverage gaps difficult to identify. We introduce a multi-layer taxonomy of 14 capability domains and 91 subskills across Primitive, Constructed, and Integrative layers. Human cognitive science guides capability definition and organization, not LLM architecture. Layer assignments draw on developmental precedence and hypothesized functional support, while human-origin constructs are adapted to observable model behavior. To demonstrate operational utility, we screened 31,505 papers from ACL, AAAI, ICML, and NeurIPS between 2023 and 2025 and mapped 15,934 LLM-focused papers through multi-model annotation, consensus, and arbitration. Direct research attention concentrated on Language-Semantic Competence (3,551; 22.3%), Reasoning (3,388; 21.3%), Planning and Decision-Making (2,149; 13.5%), and Perception (1,954; 12.3%), whereas six domains appeared in fewer than 2% of papers. Within domains, the most frequent subskill had a median prevalence of 97.9% and appeared in at least 90% of papers in 10 of 14 domains. Language-Semantic Competence and Reasoning formed the highest-volume pair (n = 1,864; 11.7%; lift = 2.47), whereas Theory of Mind and Social Reasoning and Interaction showed the highest lift among pairs with at least 20 co-occurrences (n = 62; lift = 30.84). By shifting the unit of analysis from isolated tasks to structured capabilities, the taxonomy supports research organization, coverage audits, evaluation interpretation, and testable hypotheses for diagnosis, training, and transfer.
Shixin Fang, Jiachen Wo, Wenjuan Qin et al.· 0 citations
The development of Large Language Models (LLMs) has expanded the function of artificial intelligence from mere automation systems toward dialogue agents used across academic, professional, administrative, and creative activities. Alignment paradigms heavily reliant on reinforcement learning from human feedback (RLHF) still face fundamental challenges including hallucination, sycophancy, overconfidence, and vulnerability to instructional manipulation. This article aims to develop a conceptual protocol framework called Sincerity Echo as a new paradigm in LLM alignment based on Cognitive Proportionality. The study employs a design science research approach with a conceptualprotocol development orientation. The model is developed through two layered validation mechanisms: the Macro Semantic Gatekeeper for semantic consistency checking and the Continuous Logic Decay Filter for propositional contradiction detection. Integration of semantic entropy and semantic uncertainty enables the system to detect potential hallucinations and adaptively manage belief calibration. Model development results show that Sincerity Echo can differentiate propositional expansions, low risk lightweight queries, and adversarial contradictions through a tiered validation mechanism. The FAST EXIT ROUTE mechanism on simple queries saves approximately 96.8% of computational resource allocation compared to deep reasoning pathways. The main contribution lies in shifting alignment from mere instructional compliance toward epistemic integrity, belief calibration, anti sycophancy, and response proportionality.
Blasius Dala Nai, Jeffrey Bram Pattipeilohy, Arief Wibowo· Greenation International Jou...· 0 citations
Large language models (LLMs) are increasingly deployed in domains where opacity, factual un-reliability, and computational cost carry real consequences, yet the research communities that address these three problems — explainability, hallucination, and inference efficiency — have largely developed in isolation. This paper argues that the three concerns are not independent: interventions designed to reduce inference cost, such as prompt compression, can silently alter both the faithfulness of post-hoc explanations and the model’s propensity to hallucinate. We present an integrated conceptual frame-work that couples attention- and perturbation-based explainability (LIME, SHAP, raw attention) with lightweight hallucination-detection signals and token-efficient inference strategies (chunking, summa-rization, prompt compression), and we instrument the framework with a set of cross-cutting consistency metrics. To ground the framework empirically, we design and execute a controlled pilot experiment that measures how rule-based prompt compression jointly affects (i) the Kullback–Leibler divergence between a model’s output distributions before and after compression, used as a proxy for hallucination risk, and (ii) the entropy and rank-correlation of last-layer attention, used as a proxy for explanation faithfulness. Because the sandboxed experimental environment used for this study has no network ac-cess to pretrained model repositories, the experiment is conducted on a compact two-layer Transformer language model trained from scratch on a controlled synthetic corpus, which allows exact, reproducible control over ground truth while preserving the qualitative mechanics of attention-based attribution and next-token prediction under compression. Across 15 held-out prompts, compression removes 52.9% of tokens on average while producing a small but non-zero mean output KL divergence of 0.0007 and re-ducing mean attention entropy from 1.062 to 0.632, with attention rank correlation across compression conditions of only 0.633. These results provide direct, quantitative evidence for the framework’s central hypothesis: token-efficiency interventions are not explanation-neutral, and systems that report compres-sion ratios without also reporting faithfulness and hallucination-risk deltas may be masking a three-way trade-off. We discuss the implications for trustworthy LLM system design and outline how the pilot findings motivate follow-up experiments on larger pretrained models.
Sakshi Parate, Shreyans Sanyal· Advanced International Journ...· 0 citations