Large language models (LLMs) have achieved strong performance on text-to-knowledge graph generation and related tasks. Nevertheless, it is still unclear whether they accurately model the direction-dependent semantics of inverse relations, in which reversing the order of the arguments alters the meaning of a relation (e.g., \textit{mother} versus \textit{child}). To the best of our knowledge, this work presents the first systematic study of inverse relation directionality in LLMs, using a benchmark consisting of 5,457 instances spanning 27 distinct inverse relation labels. We evaluate five open-source LLMs under a multiple-choice prompting framework and further examine the influence of relation descriptions and entity representations by substituting the original entities with synthetic and masked entities. Our findings reveal systematic asymmetries in inverse relation classification across LLMs, indicate that relation descriptions do not consistently improve performance, and show that model performance can be sensitive to variations in entity representations.
Large language models (LLMs) have demonstrated remarkable capabilities in multistep reasoning, yet fundamental questions remain about their ability to perform reliable deductive reasoning. Two contrasting paradigms have emerged: chain-of-thought (CoT) prompting, which treats LLMs as self-contained reasoners, and solver-augmented approaches, which use LLMs as semantic parsers to translate problems into formal logic for symbolic execution. We present a systematic comparison of these approaches under strict single-pass evaluation across three established benchmarks (ProofWriter, PrOntoQA, and LogicalDeduction) and controlled synthetic datasets. Our findings reveal three key insights: (1) on standard benchmarks, solver augmentation yields minimal improvements (
<
1
%
) over zero-shot CoT for state-of-the-art models, suggesting diminishing performance margins between paradigms under current benchmark settings; (2) mid-sized open-source models (14B–32B) match proprietary model performance when solver-augmented, with the primary challenge shifting from generating valid syntax (small models) to accurate semantic parsing (larger models); and (3) on synthetic datasets with controlled difficulty variations, solver-augmented methods demonstrate superior robustness, maintaining 76.7% accuracy retention at extreme reasoning depths (14–17 steps) compared to 62.2% for CoT, and showing greater resilience to distracting facts and rules. These results indicate that while CoT-augmented LLMs achieve strong performance on deductive reasoning tasks up to five hops, solver augmentation remains valuable for deeper multihop deduction and for applications requiring robust and verifiable reasoning.
Ya Wang, Raja Havish Seggoju, A. Paschke· Neurosymbolic Artificial Int...· 0 citations
Relation extraction (RE) for low-resource languages is typically constrained by the lack of annotated corpora. We investigate the feasibility of cross-lingual RE for Romanian by combining automatic dataset translation with large language model (LLM) inference. We translate the SemEval-2010 Task 8 benchmark from English to Romanian using an LLM-based translation pipeline and evaluate Gemma 4 31B under zero-shot, few-shot, and QLoRA fine-tuned configurations, against four encoder baselines spanning 125M to 560M parameters: XLM- RoBERTa (base and large), Romanian BERT, and RoBERT- large. We assess two task formulations: relation classification with marked entities and end-to-end extraction. Our results show that Romanian incurs a 3 to 5 percentage point (pp) drop relative to English in prompt-only settings, that few-shot prompting provides marginal gains over zero-shot, and that QLoRA fine-tuning improves macro F1-Score by more than 22 percentage points in both languages while reducing the cross-lingual gap from 3.3 to 1.4pp. The encoder baselines come within 1-4pp of QLoRA Gemma on Romanian despite being 50-250 times smaller, with monolingual Romanian BERT at 125M parameters matching multilingual XLM-R at 278M. The case for using a 31B model for single-task RE on Romanian is therefore weak in deployment scenarios where compute matters. We release the translated dataset, evaluation code, and trained models.