RuleWeaver is introduced, a benchmark construction framework for evaluating rule-centered scenario reasoning that starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances.
Abstract
Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.
DynaRule is proposed, an end-to-end framework that injects the given rules into the KV cache and turns retrieval into an internal, learnable, step-wise process, and can re-attend to the most relevant rules at each step, dynamically replacing outdated ones to support more stable multi-step reasoning.
Bohan Yu, Pengfei Cao, Chen Han et al.· 1 citation
This work proposes StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility, and experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions.
Jinghan Tan, Yuanzhe Wang, Lu Chen et al.· 0 citations
Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
Despite the success of Large Language Models (LLMs) on reasoning benchmarks, it remains unclear whether their performance stems from genuine logical deduction or the memorization of training patterns. Existing benchmarks often fail to disentangle reasoning from prior knowledge, as tasks grounded in real-world facts allow models to take ''knowledge shortcuts''. In this paper, we propose a novel diagnostic benchmark to decouple knowledge memorization from logical reasoning. Built on the DBpedia KG, our framework constructs multi-hop reasoning chains (from Q1 to Q5) across three task dimensions: Factural Questions (FQ), Counterfactual Questions (CQ) with logically consistent but counterfactual conclusions, and Questions with Similar-Entity Options (SO) to evaluate the dependence on prior knowledge. Questions without Context serve only as an intermediate form: they contain solely queries with no triples or options, so LLMs cannot answer them directly. Our core hypothesis is that genuine reasoning is demonstrated only when a model follows logical rules despite conflicting prior knowledge. Evaluating seven state-of-the-art LLMs (8B to ultra-large) reveals strong prior knowledge dependence, with performance degrading sharply on counterfactual tasks as reasoning depth grows. This work delineates LLM reasoning boundaries and presents a new paradigm for fine-grained capability assessment.
Fangfei Yan, Jianbo Yao, Michael K. Chen et al.· Proceedings of the 32nd ACM...· 1 citation
Reason Popper-ly, a neurosymbolic framework that uses inductive logic programming (ILP) to learn relation composition rules from reasoning traces and deploys them as an online verifier for step-level correction, consistently improves terminal accuracy over standard CoT.
TKFQA, a factuality consistency benchmark comprising 10,130 question-answering (QA) pairs grounded in tables, texts, and knowledge graphs, is introduced and ORLF, an LLM-agnostic training framework that models cross-context topological relations through knowledge-specific latent vectors is proposed.
Shibo Chu, Yuze Liu, Tiehua Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
