Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities
TL;DR
The framework, Common Logic Grammar Construction (CLGC), is open-source, as the first Python library for automatically generating syllogisms in KR notations and defining their SEF categories, and proposes a syllogistic categorization method (SEF) that is used to enrich ZS prompts with logical definitions, which boost reasoning in small models.
Abstract
Language models (LMs) struggle with logical tasks like reasoning on syllogisms. It has been shown that Knowledge Representation (KR) plays a crucial role in expressing input information to help models solve tasks. This observation motivates our study of the impact of different formal KR notations on syllogistic reasoning by extending the FOLIO and P-FOLIO datasets. Our experiments on Small Language Models (SLMs) in Supervised Fine-Tuning (SFT) and Zero-Shot (ZS) settings show that the choice of input notation can yield performances competitive with natural language while enabling faster inference. We also propose a syllogistic categorization method (SEF) and use it to enrich ZS prompts with logical definitions, which boost reasoning in small models. We open-source our framework, Common Logic Grammar Construction (CLGC), as the first Python library for automatically generating syllogisms in KR notations and defining their SEF categories.