Results show that verified counterfactual structure can shape post-training beyond final answers, while exact full change-record generation, invariant subtype recognition, and out-of-distribution (OOD) transfer remain open problems.
Abstract
Language models can reach the right normative verdict for the wrong reason. We introduce NormWorlds-CF, a solver-verified environment for counterfactual normative reasoning in executable rule worlds. Its deterministic solver produces final answers, proof and falsification certificates, argument statuses, support sets, and paired-world change labels, enabling supervision and evaluation without LLM judges. The benchmark contains staged SFT diagnostics and a compact paired-world task with 270 root families and 1080 canonical-to-variant pairs. The SFT diagnostics show that final-answer supervision can saturate verdict accuracy without inducing falsification competence: answer-only SFT reaches perfect answer accuracy but scores zero on joint falsification certificates, while full-mix training with targeted replay reaches strong all-task accuracy (0.99). For the structured-change task, we introduce metamorphic-relation GRPO (MR-GRPO), a class-conditioned reward for GRPO that gives partial credit for relation families and solver-visible change fields. In matched Qwen3-1.7B continuation experiments, MR-GRPO improves held-out relation accuracy and relation-family correctness, and reduces wrong-family error, compared to sparse and answer-only GRPO. In Qwen3-4B three-seed validation, sparse reward preserves coarse relation labels best, answer-only reward improves answer-change but weakens relation-family structure, and MR-GRPO leads on answer-, support-, and status-change fields as well as class-conditioned MR and change-presence. These results show that verified counterfactual structure can shape post-training beyond final answers, while exact full change-record generation, invariant subtype recognition, and out-of-distribution (OOD) transfer remain open problems.
The Reasoning Answer Faithfulness Score (RAFS) is introduced, a reference free, instance level diagnostic of whether an emitted mathematical trace is locally credible, supports its answer, and is stable under resampling and targeted counterfactual interventions.
Vivek Shukla, Varun Shukla, Atul et al.· arXiv.org· 0 citations
Large language models (LLMs) show strong reasoning ability, but their explanations can remain inconsistent, weakly grounded, or difficult to verify. We propose a verifier-guided explainable reasoning framework for transparent educational question answering that combines gold-anchored QLoRA, task-aware symbolic routing, and group-relative RLVR. Qwen2.5-3B-Instruct is first adapted with field-weighted QLoRA supervision anchored to authoritative answers. A lightweight router then assigns logic problems to a FOL/Z3 verifier and physics problems to a formula- and unit aware symbolic solver. Verifier feedback is further used to support candidate evaluation, self-revision, and reward construction during RLVR. Candidate responses are evaluated along three complementary dimensions: P1 for answer correctness, P2 for evidence or unit consistency, and P3 for reasoning depth and explainability. At inference, gold-free self-consistency aggregates multiple candidate responses before an optional question-only physics verifier performs conservative system-level correction. On 438 held-out examples, RLVR increases P3 from 50.68% to 72.20%, while hybrid P1 remains approximately stable at 55.94%. Self-consistency improves model only P1 from 48.86% to 50.23%, with symbolic verification providing the remaining hybrid gain. These results indicate that RLVR primarily strengthens explicit reasoning structure, while symbolic verification complements the neural policy by improving answer reliability at the system level.
Thi Kim Trang Vo, Nam Tien Le, Thi Kim Nguyet Vo et al.· 0 citations
TKFQA, a factuality consistency benchmark comprising 10,130 question-answering (QA) pairs grounded in tables, texts, and knowledge graphs, is introduced and ORLF, an LLM-agnostic training framework that models cross-context topological relations through knowledge-specific latent vectors is proposed.
Shibo Chu, Yuze Liu, Tiehua Zhang et al.· 0 citations
This work introduces CALVER (Causal Axiom-Level VERification), a training-free symbolic verifier that scores structured traces against Pearl's causal criteria, including -separation, backdoor adjustment, and intervention, and selects the highest-scoring candidate without consulting a reference answer.
Omatharv Bharat Vaidya, C. Jerzak, Zayne Sprague et al.· 0 citations
Neurosymbolic systems rely on mathematical solvers to guarantee reasoning correctness, yet solvers are fundamentally blind to whether a formal translation maintains strict reference-equivalence to a designated formalization. We formalize this vulnerability as Verdict-Preserving-Unfaithfulness (VPU): a failure mode where an incorrect encoding executes successfully and matches the expected verdict. We theoretically prove that structural, verdict-only verification heuristics are mathematically bounded to chance-level detection on these deceptively valid traces. To resolve this, we introduce Generative Verification (GenV), which distills an offline Z3-equivalence oracle into a reference-free, continuous reference-equivalence score by repurposing the language model's native vocabulary space. Mechanistic analysis via decision-projected logit lenses and sparse autoencoders shows this generative readout natively extracts precise spatial error coordinates without explicit localization training. Empirically, our oracle-mined verifier (GenV+HN) achieves 0.961 AUROC in reference-equivalence verification, generalizes zero-shot across unseen translators and divergent formal styles, and yields an 11.3-point downstream accuracy gain in agentic test-time compute allocation.
Vikash Singh, Debargha Ganguly, Aman Goel et al.· 0 citations
Automated fact-checking systems still fall short of producing explanations that mirror the depth and structure of expert human reasoning. In this work, we propose a multi-agent framework that integrates five specialized linguistic agents covering polarization, linguistic style, argumentation, plausibility, and contextual framing with web-based evidence retrieval, synthesized by a supervisor agent into structured reports resembling professional fact-checking outputs. We evaluate the framework on a dataset of fact-checked Brazilian news through a classification benchmark and two further quantitative studies of explanation quality, addressing: (1) Do the generated reports elicit reader confidence comparable to reports written by professional fact-checkers? and (2) Which explanatory dimensions most influence reader confidence? The classification benchmark shows the framework performs competitively with strong baselines. A blinded within-subjects study with 95 participants, analyzed via Linear Mixed Models, shows that post-verification confidence reaches levels statistically indistinguishable from expert-written reports, with plausibility and analytical depth as the strongest predictors of confidence gain and depth being especially important for implausible claims. Complementary LLM-as-a-judge experiments corroborate these findings, showing the framework’s explanations are consistently preferred for depth, persuasion, and plausibility.
Pedro Henrique de Oliveira Silva, L. Santos, L. Marinho et al.· Proceedings of the 37th ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
