This article offers a comprehensive overview of mechanistic interpretability, an emerging field that seeks to reverse-engineer the internal algorithms of modern neural networks, and explores methods for actively controlling and modifying model behavior through steering vectors and causal interventions.
Abstract
This article offers a comprehensive overview of mechanistic interpretability, an emerging field that seeks to reverse-engineer the internal algorithms of modern neural networks. While traditional explainable AI methods often stop at surface-level input-output correlations, this approach directly addresses the opaque"black box"nature of machine learning models, which is essential for ensuring safety and auditability in high-stakes deployments. The paper provides a detailed examination of Transformer circuit analysis, exploring how internal components like the residual stream, attention mechanisms, and induction heads drive complex tasks and in-context learning. It subsequently tackles the core challenge of superposition and polysemanticity, demonstrating how tools like Sparse Autoencoders (SAEs) and transcoders can decompose tangled network activations into distinct, human-interpretable features. Furthermore, the paper explores methods for actively controlling and modifying model behavior through steering vectors and causal interventions. Finally, it connects these mechanistic insights with neurosymbolic AI frameworks designed to translate neural representations into explicit, executable logical rules.
It is argued that mechanistic interpretability has the potential to support a more scientific understanding of machine learning systems – treating models not only as tools for solving tasks, but also as systems to be studied and understood.
Background: Deep neural networks increasingly power language, vision, and decision systems, yet many deployments require explanations that are faithful, compositional, and governance-ready. Symbolic techniques promise these properties, but the literature mixes post-hoc extraction, knowledge injection, and intrinsically hybrid designs without a unifying view.
Objectives: We provide a systematic review and synthesis of symbolic explainable AI (XAI) for deep learning (January 2017– June 2025), organize the field around a three-part taxonomy—Symbolic Knowledge Extraction (SKE), Symbolic Knowledge Injection (SKI), and Hybrid neurosymbolic architectures—and propose a conceptual framework that clarifies training–inference flows, explanation interfaces, human feedback, and governance touchpoints.
Methods: Beginning from ≈50,000 records, we deduplicated and screened full texts, analyzed 393 PDFs, and included 273 primary studies in the synthesis. We coded each paper for model domain, modality, symbolic formalism, explanation scope and stage, evaluation protocol, and governance alignment. Analyses combine descriptive statistics with stratification by domain and formalism; we qualitatively assess evidence for faithfulness, robustness, data efficiency, and constraint satisfaction.
Results: Research activity accelerates after 2020, with a marked turn toward hybrids. Across the corpus, SKE, SKI, and Hybrid account for approximately 29%, 26%, and 45% of studies, respectively. Rule sets/decision trees remain the dominant explanation artifacts, while logic- and program-based formalisms grow in NLP and planning. SKI most often targets constraint satisfaction and robustness improvements; SKE emphasizes global surrogates and faithfulness auditing; hybrids report gains in sample efficiency and traceable reasoning. However, evaluation practices are heterogeneous, human-subject studies are scarce, and explicit links to policy/risk controls appear in a minority of works.
Conclusions: Our framework unifies how data, priors, and symbolic reasoning interact with neural learners, the explanation interface, human stakeholders, and governance. We distill actionable recommendations: (1) report faithfulness and constraintsatisfaction metrics alongside accuracy; (2) specify symbolic assumptions and training-time injections precisely; (3) include user studies or auditor-centric protocols for high-stakes use; and (4) develop benchmarks that couple tasks with machinereadable knowledge bases. We highlight open problems in scalable formal reasoning with foundation models, verifying generated rationales, and measuring causal faithfulness at scale.
Eduard Ionel Stan, G. Sciavicco, Paolo Napoletano· Journal of Artificial Intell...· 0 citations
This work bridges hybrid modeling and neuro-symbolic (NeSy) AI by reconstructing these designs as instances of NeSy interface and derives metrics: structural violation rate (SVR), measuring whether the learned belief respects the mechanistic structure; and belief dispersion (BD), measuring how concentrated the learned plausibility is, serving as a hybrid model's epistemic uncertainty in its mechanistic part.
Moein E. Samadi, Andreas Schuppert· arXiv.org· 0 citations
The rapid evolution of Large Language Models (LLMs) has brought unprecedented capabilities across reasoning, coding, and multimodal tasks. However, as performance scales, their opaque ''black-box'' nature raises a critical challenge: How can we trace the origins of emergent intelligence, and more importantly, how can we leverage these internal mechanisms to guide model optimization? This tutorial provides a comprehensive, end-to-end view of LLM interpretability, transitioning from microscopic neural analysis to macroscopic application and deployment. It is systematically organized into five core sections: i) Unlocking the Black Box: We begin with the evolution of LLM interpretability and highlight recent breakthroughs from leading research teams. ii) Methodology: We present a rigorous overview of foundational theories (e.g., mathematical framework for transformer, biological mechanisms in LLMs) and essential methods (e.g., path patching, logit lens, and neuron description). iii) Anatomy of LLMs: Using advanced techniques to decode internal semantic features, neural circuits, and complex behaviors, we interpret how models perform reasoning, factual recall, and in-context learning. iv) Applications: We show how to transfer interpretability insights into actionable improvements across the LLM pipeline, including interpretability-guided data synthesis (data value scoring, corpus filtering, and activation-based data diagnosis). We also present Pinpoint Training and Steering for precise capability gains, and Pinpoint Quantization for extreme low-bit compression with minimal capability loss. v) Advanced Topics: We conclude by exploring how these interpretability paradigms scale and inspire the design of frontier architectures, agentic systems, and thinking models. In this tutorial, researchers and engineers will gain the theoretical frameworks and practical engineering toolkits needed to understand, steer, and efficiently deploy LLMs in real-world production environments.
Wei Zhang, Zhengfu He, Lucia Zhang et al.· Proceedings of the 32nd ACM...· 0 citations
Artificial Neural networks (ANNs) are often treated as black-box models, making explainability a central challenge in deep learning. Many engineering methods have been proposed to approximately explain the ANN from various perspectives, such as feature attribution and visualization. However, it remains a long-standing open question whether the complex inference logic of an ANN can be explained exhaustively and concisely as sparse symbolic patterns. This raises a deeper inquiry: does the emergence of symbolic patterns reflect a natural law rather than chance? Here, we show that across a broad class of ANNs trained on diverse tasks, their inference logic can indeed be reformulated as sparse symbolic interactions. We further prove that two common mathematical criteria, which are implicitly required across tasks, lead to the emergence of such sparse symbolic interactions. Empirical evidence confirms that the two criteria hold for the majority of input samples in diverse models. Furthermore, the faithfulness of these interactions is also demonstrated by their strong sample-to-sample and model-to-model transferability, as well as their ability to explain the overall generalization power of ANNs. Our theoretical analysis and extensive experiments provide a solid foundation for symbolic explanations of ANNs, and offer novel insights into the ANN's generalization power. Our findings also highlight the potential of communicative learning, a paradigm in which the inference logic of an ANN can be directly inspected and tuned at the level of symbolic patterns, thus complementing traditional end-to-end learning paradigm. Finally, the observed emergence of symbolic patterns in ANNs suggests that similar symbolic representations may also emerge in other types of black-box systems under certain conditions, because our proof does not depend on any specific ANN architecture.
: This paper introduces a cognitively inspired neuro-symbolic framework for extracting interpretable world models from raw video streams in dynamic 2D environments. While traditional end-to-end deep reinforcement learning systems often function as opaque “black boxes,” our approach decouples visual perception from policy learning to enhance transparency. By leveraging Core Knowledge theory (specifically object persistence, physical causality, and agent representation), the system transforms visual patches into structured symbolic entities and governing interaction rules. The core of this architecture is a symbolic module that reconstructs persistent objects, infers parametric motion laws (such as velocity inversion upon contact), and incrementally consolidates class-specific behaviors into a compact knowledge base. This world model instantiates a domain-agnostic environment wrapper, enabling a Deep Q-Network to operate on symbolic state vectors rather than pixels. We further propose a domain-agnostic agent that develops complex behaviors driven by a composite intrinsic reward based on causal impact and event-driven curiosity. Experimental evaluations on Arkanoid and Pong demonstrate that this framework achieves near-optimal performance without task-specific external rewards. On Arkanoid, the agent matches the win rate of fully supervised models while remaining robust to structural modifications, such as changes in ball size or brick configuration. In Pong, the same mechanism transfers without architectural adjustments, consistently improving survival times. These results provide a transparent, generalizable alternative to dominant AI paradigms with good performance across varied environmental conditions.
Lorenzo Cardone, Giorgia Ghisolfo, G. Mongardi et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
