This overview addresses this issue by providing a gentle introduction to RSs, discussing their causes and consequences in intuitive terms, and details methods for dealing with RSs, including mitigation and awareness strategies, and maps their benefits and limitations.
Abstract
Neuro-symbolic (NeSy) AI aims to develop deep neural networks whose predictions comply with prior knowledge encoding, e.g., safety or structural constraints. As such, it represents one of the most promising avenues for reliable and trustworthy AI. The core idea behind NeSy AI is to combine neural and symbolic steps: neural networks are typically responsible for mapping low-level inputs into high-level symbolic concepts, while symbolic reasoning infers predictions compatible with the extracted concepts and the prior knowledge. Despite their promise, it was recently shown that – whenever the concepts are not supervised directly – NeSy models can be affected by Reasoning Shortcuts (RSs). That is, they can achieve high label accuracy by grounding the concepts incorrectly. RSs can compromise the interpretability of the model’s explanations, performance in out-of-distribution scenarios, and therefore reliability. At the same time, RSs are difficult to detect and prevent unless concept supervision is available, which is typically not the case. However, the literature on RSs is scattered, making it difficult for researchers and practitioners to understand and tackle this challenging problem. This overview addresses this issue by providing a gentle introduction to RSs, discussing their causes and consequences in intuitive terms. It also reviews and elucidates existing theoretical characterizations of this phenomenon. Finally, it details methods for dealing with RSs, including mitigation and awareness strategies, and maps their benefits and limitations. By reformulating advanced material in a digestible form, this overview aims to provide a unifying perspective on RSs to lower the bar to entry for tackling them. Ultimately, we hope this overview contributes to the development of reliable NeSy and trustworthy AI models.
A novel method called DKA is presented, which introduces a limited set of concept-supervised data to enhance the knowledge base, effectively solving the reasoning shortcut problem and improving the applicability of the NeSy system.
Yu-Feng Li, Xiaowen Yang, Wenda Wei et al.· Proceedings of the 32nd ACM...· 0 citations
LLMs have demonstrated strong language-learning and human-like response-generation capabilities, and they are increasingly used to support decision-making in high-risk sectors. However, their internal decision processes remain difficult to interpret, and their responses may lack transparency. The literature has explored numerous approaches to address transparency challenges in LLMs, including Neurosymbolic AI (NeSy AI). NeSy AI approaches were primarily developed for conventional neural networks and may not transfer directly to the distinctive characteristics of LLMs. Consequently, there is a limited systematic understanding of how symbolic AI can be effectively integrated into LLMs. This paper aims to address this gap by first reviewing established NeSy AI methods and then proposing a novel taxonomy of symbolic integration in LLMs, along with a roadmap to merge symbolic techniques with LLMs. The taxonomy organises the literature across four dimensions: (1) the stage of LLM development at which symbolic information is integrated; (2) the coupling mechanism; (3) the architectural paradigm; and (4) the algorithm-level or application-level perspective. The review identifies commonly used benchmarks, recent advances and important research gaps, and uses these findings to outline directions for future research. By highlighting the latest developments and notable gaps in the literature, it offers practical insights for implementing frameworks for symbolic integration into LLMs to enhance transparency.
Maneeha Rani, B. K. Mishra, Dhavalkumar Thakker· Information Systems Frontier...· 0 citations
Background: Deep neural networks increasingly power language, vision, and decision systems, yet many deployments require explanations that are faithful, compositional, and governance-ready. Symbolic techniques promise these properties, but the literature mixes post-hoc extraction, knowledge injection, and intrinsically hybrid designs without a unifying view.
Objectives: We provide a systematic review and synthesis of symbolic explainable AI (XAI) for deep learning (January 2017– June 2025), organize the field around a three-part taxonomy—Symbolic Knowledge Extraction (SKE), Symbolic Knowledge Injection (SKI), and Hybrid neurosymbolic architectures—and propose a conceptual framework that clarifies training–inference flows, explanation interfaces, human feedback, and governance touchpoints.
Methods: Beginning from ≈50,000 records, we deduplicated and screened full texts, analyzed 393 PDFs, and included 273 primary studies in the synthesis. We coded each paper for model domain, modality, symbolic formalism, explanation scope and stage, evaluation protocol, and governance alignment. Analyses combine descriptive statistics with stratification by domain and formalism; we qualitatively assess evidence for faithfulness, robustness, data efficiency, and constraint satisfaction.
Results: Research activity accelerates after 2020, with a marked turn toward hybrids. Across the corpus, SKE, SKI, and Hybrid account for approximately 29%, 26%, and 45% of studies, respectively. Rule sets/decision trees remain the dominant explanation artifacts, while logic- and program-based formalisms grow in NLP and planning. SKI most often targets constraint satisfaction and robustness improvements; SKE emphasizes global surrogates and faithfulness auditing; hybrids report gains in sample efficiency and traceable reasoning. However, evaluation practices are heterogeneous, human-subject studies are scarce, and explicit links to policy/risk controls appear in a minority of works.
Conclusions: Our framework unifies how data, priors, and symbolic reasoning interact with neural learners, the explanation interface, human stakeholders, and governance. We distill actionable recommendations: (1) report faithfulness and constraintsatisfaction metrics alongside accuracy; (2) specify symbolic assumptions and training-time injections precisely; (3) include user studies or auditor-centric protocols for high-stakes use; and (4) develop benchmarks that couple tasks with machinereadable knowledge bases. We highlight open problems in scalable formal reasoning with foundation models, verifying generated rationales, and measuring causal faithfulness at scale.
Eduard Ionel Stan, G. Sciavicco, Paolo Napoletano· Journal of Artificial Intell...· 0 citations
In many reasoning problems, the premises are not observed as discrete symbols, but must be inferred from high-dimensional inputs. Further, the predicate vocabulary, argument structure, and trusted evidence are supplied by a Knowledge Graph (KG), or rule definitions. Classical neuro-symbolic pipelines have a discrete interface between perception and deduction. We present a neuro-soft-symbolic architecture for differentiable deductive reasoning over latent perceptual facts and knowledge-provided predicates. SoftReason removes the gradient gap by representing the deductive state as a local soft interpretation tensor over candidate constants and predicates. Perception proposes probabilistic base facts, KG triples enter as high-confidence soft evidence, and every query anchor, predicate choice, and closure update remains differentiable. Our core innovation is a learned differentiable lift of the immediate-consequence operator. It uses predicate-definition embeddings and latent composition channels to form soft body-predicate mixtures, aggregate over all possible witnesses, propose query-conditioned head facts, and update the interpretation through a monotone probabilistic OR. We instantiate the framework on Knowledge-aware Visual Question Answering (KVQA), and demonstrates how SoftReason supports end-to-end perceptual grounding, KG evidence injection, and differentiable deductive closure in one trainable architecture.
Neural models in high-stakes domains lack access to ontological domain constraints that practitioners take for granted, and retrofitting such knowledge is hard: expressive logical formalisms do not scale, while scalable ones cannot express the negation, disjunction, and quantification that real constraints require. We present a differentiable reasoning framework that resolves this tension. Operating within a decidable logic that retains full Boolean expressivity, it transforms domain rules into a training objective with guaranteed polynomial-time cost per iteration. The framework is operator-agnostic - it decouples logical structure from the choice of underlying continuous semantics, revealing, through the first controlled comparison of its kind, that this choice alone can swing performance by over 30 points on the same task. This finding motivates two adaptive mechanisms: a semantic gate that focuses gradient signal on the model's most flagrant logical violations, and a structure-aware loss that automatically reweights its objective according to the logical complexity of the input constraints. Together, they eliminate the need for per-dataset loss tuning. On eight benchmark ontologies, the framework achieves statistically significant improvements over nine baselines spanning neuro-symbolic, geometric, and probabilistic paradigms. On semantic image interpretation, it refines a frozen object detector using domain rules alone - without the need for extra labels - lifting macro-averaged F1 by up to 7.8%, showing that structured knowledge, properly injected, can turn brittle pattern-matching into logically coherent prediction.
Zi-Han Shao, Chang Lu, Renate A. Schmidt et al.· Proceedings of the 32nd ACM...· 0 citations
Artificial Neural networks (ANNs) are often treated as black-box models, making explainability a central challenge in deep learning. Many engineering methods have been proposed to approximately explain the ANN from various perspectives, such as feature attribution and visualization. However, it remains a long-standing open question whether the complex inference logic of an ANN can be explained exhaustively and concisely as sparse symbolic patterns. This raises a deeper inquiry: does the emergence of symbolic patterns reflect a natural law rather than chance? Here, we show that across a broad class of ANNs trained on diverse tasks, their inference logic can indeed be reformulated as sparse symbolic interactions. We further prove that two common mathematical criteria, which are implicitly required across tasks, lead to the emergence of such sparse symbolic interactions. Empirical evidence confirms that the two criteria hold for the majority of input samples in diverse models. Furthermore, the faithfulness of these interactions is also demonstrated by their strong sample-to-sample and model-to-model transferability, as well as their ability to explain the overall generalization power of ANNs. Our theoretical analysis and extensive experiments provide a solid foundation for symbolic explanations of ANNs, and offer novel insights into the ANN's generalization power. Our findings also highlight the potential of communicative learning, a paradigm in which the inference logic of an ANN can be directly inspected and tuned at the level of symbolic patterns, thus complementing traditional end-to-end learning paradigm. Finally, the observed emergence of symbolic patterns in ANNs suggests that similar symbolic representations may also emerge in other types of black-box systems under certain conditions, because our proof does not depend on any specific ANN architecture.
Quanshi Zhang, Qihan Ren, Siyu Lou· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
