GNRS-Search is introduced, a framework that utilizes Markov Chain Monte Carlo sampling to optimize a discrete, five-slot And-Or Graph (AOG) and provides a foundational paradigm for deploying verifiable and compliance-ready personal agents within regulated environments.
Abstract
Normative rules like institutional charters and workplace policies must be both human-readable and operationally verifiable against actual environment records. However, current language generation and structured-output benchmarks primarily reward surface fluency or schema compliance, leaving operational grounding weakly tested. This creates a critical vulnerability where standard language models generate plausible-sounding policies that fail during enforcement because they rely on unavailable data logs or misaligned scopes. To address this challenge, we formalize the problem as Grounded Normative Rule Synthesis (GNRS) and introduce GNRS-Search, a framework that utilizes Markov Chain Monte Carlo (MCMC) sampling to optimize a discrete, five-slot And-Or Graph (AOG). By explicitly decoupling intermediate operational structure from final prose generation, this method isolates executable feasibility from writing style and allows rule failures to be localized prior to surface realization. We evaluate our approach on GNRS-Bench, a benchmark spanning 116 controlled goals across eight scene families, and RealCharter-Bench, which evaluates transfer to 53 real-derived policy tasks with hidden source clauses. GNRS-Search raises average rubric quality from 68.8% to 81.0% and ranks first under a disclosed executable composite metric, while systematic slot interventions confirm that performance gains stem from robust operational logic rather than rhetorical tuning. Ultimately, by transforming automated rule drafting into an inspectable search problem, this work provides a foundational paradigm for deploying verifiable and compliance-ready personal agents within regulated environments.
Explainability is particularly challenging in legal prediction, where decisions are expected not only to be accurate but also to be justified under explicit legal norms and open to external scrutiny. Legal reasoning therefore provides one of the most demanding settings for explainable AI because a legal determination often depends on information distributed across the input record, external legal sources, procedural context, evidentiary assumptions, and institutional judgment. Direct outcome generation hides this decision path, while natural-language rationales and retrieved citations do not necessarily show whether a legal condition was actually applied. This article presents studies of retrieval-augmented codification as a way to produce executable explanation traces for legal large language model (LLM) predictions. The goal is not to generate human-facing rationales directly, but to produce auditable artifacts: short programs that extract signals, apply program-level conditions, follow branches, and produce final labels under a fixed output schema. The approach retrieves legal sources and fold-safe in-domain examples, compiles them into an executable intermediate representation, and refines the program using model feedback. The resulting trace records retrieved materials, generated conditions, feedback, and the final prediction. We evaluate this framework on five legal benchmarks covering Chinese criminal law, European human-rights cases, U.S. class-action complaints, Indian judgments, and Japanese tort cases. The evaluation asks not only whether the predicted label is correct, but also whether the explanation is executable, whether its conditions are supported by retrieved legal sources, whether it responds to legally material edits, and where it fails. The results show that executable explanations can be obtained without sacrificing predictive utility: the full codification setting improves mean label accuracy over direct prompting, chain-of-thought prompting, one-pass codification, and text-based retrieval baselines. More importantly, trace analysis reveals why legal explainability is difficult. Codification is most informative when decisions depend on definitions, thresholds, exceptions, and other rule-like components. It is less complete when benchmark labels depend on procedural, evidentiary, or discretionary factors outside the input record. These findings position retrieval-augmented codification as a practical explainable artificial intelligence (XAI) tool for studying when legal LLM predictions can be explained through explicit, source-supported decision paths.
Haoyang Chen, K. Tanaka-Ishii· Frontiers in Artificial Inte...· 0 citations
Jailbreak robustness has become central to large language model (LLM) safety evaluation, yet prevailing methodologies rely primarily on refusal behavior, semantic resemblance, and intent-matching heuristics that emphasize linguistic plausibility rather than correctness. We identify a key limitation in existing evaluations: many jailbreak intents depend on instructional validity rather than epistemic factuality, allowing realistic-looking responses to be labeled successful despite being factually or procedurally incorrect. To address this gap, we propose Sequential Epistemic and Action-Level Validation (SEAV), a verification-centric jailbreak evaluation framework that decomposes responses into ordered steps and evaluates both validity and correctness. SEAV combines LLM-as-a-judge mechanisms for semantic interpretation with retrieval-grounded verification using external knowledge sources, assessing whether generated content is factually correct, structurally consistent, and operationally capable of advancing harmful objectives. Empirically, SEAV cuts the false-positive rate on SD-A (a curated strategic-dishonesty diagnostic) by 14.9\,pp vs. the strongest baseline, and reclassifies 22.1\%--51.0\% of sampled prior-labeled successes as invalid across three of four public benchmarks. Together, these results show that enforcing correctness substantially reshapes measured robustness: many previously labeled jailbreak successes are reclassified as invalid, and results are stable across the tested search backends and evaluator models. Code and data are available at https://github.com/Ardor-Wu/SEAV.
Qilong Wu, Sahil Wadhwa, Pranab Mohanty et al.· 0 citations
PASK (Parser-Aware Structural KV Persistence), which turns parser-derived structure into layer-group-specific KV persistence decisions by using task-error sensitivity to set minimum protection floors and attention-output distortion to allocate residual KV capacity.
As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before taking actions. We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability across world-knowledge conflicts, input inconsistencies, and multi-source temporal conflicts. Its 238 tasks are manually screened from more than 1,000 generated candidates and combine a user simulator, stateful tools, deterministic environment assertions, an open-source natural-language evaluator, and human trajectory verification. Evaluation of nine models, including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3, shows substantial cross-domain variation: no model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings. In the simulated environments, missed conflicts can propagate to tool calls or synthetic protected-data flows. KC-Bench isolates this model-level behavior rather than ranking complete agent frameworks, and provides a reproducible diagnostic for developing conflict-aware reasoning and execution safeguards.
Yaxing Lyu, Sheng-Jie Zhou, B. Toh et al.· 0 citations
The Lit2Test benchmark centers on a six-field contract organized around a falsifying outcome, so that every proposal precommits the observation that would prove it wrong, making its quality decidable in the first place rather than merely arguable.
Ziyue Wang, Aomufei Yuan, Yi-Ran Yao et al.· 0 citations
AtmosCoder-Bench is introduced, an execution-grounded benchmark that makes the calculation process visible, and finds that multiple-choice formats inflate measured accuracy by at least 12 percentage points.
Mao-Hao Ran, Chendong Ma, Yanting Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
