This study measures that gap and releases CIV-Bench, a public benchmark of 832 clinical rule sets with safety properties across eight whole-person domains, in single-encounter and longitudinal forms, plus a computational stress tier, each with independently established ground truth.
Abstract
Clinical artificial-intelligence systems are starting to act across a course of care, not to answer one question at a time. Their safety is checked by methods that sample the input space: a test suite tries some inputs, a language-model reviewer reads some cases, a physician panel audits some cases. A sampling check can pass a safety rule and still miss the rare input that breaks it, such as a documented obligation dropped several encounters later. This study measures that gap and releases CIV-Bench, a public benchmark of 832 clinical rule sets with safety properties across eight whole-person domains, in single-encounter and longitudinal forms, plus a computational stress tier, each with independently established ground truth. We compare formal verification, which uses a satisfiability-modulo-theories (SMT) solver to check every possible input at once, against the methods used in practice: random unit testing, language-model judges, and a blinded physician panel. Formal verification detected all 612 violations, raised no false alarm, and returned no unsound verdict; for each item it returned either a proof that the rule holds over every input or one concrete input that breaks it. A frontier language-model judge matched this detection, but it returned a pass rate over sampled cases rather than a guarantee, at three orders of magnitude more compute per item. The general open-weights judge returned unsound verdicts on the computational stress tier; the medically fine-tuned judge was unsound far more widely, collapsing on the longitudinal properties despite strong single-encounter medical detection, so medical fine-tuning did not close the gap. Unit testing and the physician panel missed the deep, cross-encounter violations that hold a course of care together. Formal verification is set apart not by a higher detection rate but by the kind of evidence it returns: a proof over the whole input space, a replayable counterexample, or an explicit statement that it cannot decide. The guarantee holds for the decisions placed in this layer, and it depends on the safety rule being specified correctly.
The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model, and this work formalizes an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda.
Albus W. Ng, Yibin Han, Jusheng Zhang et al.· 0 citations
As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before taking actions. We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability across world-knowledge conflicts, input inconsistencies, and multi-source temporal conflicts. Its 238 tasks are manually screened from more than 1,000 generated candidates and combine a user simulator, stateful tools, deterministic environment assertions, an open-source natural-language evaluator, and human trajectory verification. Evaluation of nine models, including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3, shows substantial cross-domain variation: no model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings. In the simulated environments, missed conflicts can propagate to tool calls or synthetic protected-data flows. KC-Bench isolates this model-level behavior rather than ranking complete agent frameworks, and provides a reproducible diagnostic for developing conflict-aware reasoning and execution safeguards.
Yaxing Lyu, Sheng-Jie Zhou, B. Toh et al.· 0 citations
Agentic artificial intelligence, in which a model reasons, calls tools, and acts in a closed loop rather than emitting a single prediction, is arriving in medicine, where the tolerance for confident error is close to zero. This paper takes a design stance on healthcare agentic AI: rather than propose a new model, it asks how to build a clinical agentic system so that its autonomy is bounded by verification and human oversight. We distil six design requirements from the clinical AI literature, covering harm minimisation, human oversight, calibrated uncertainty, verification and provenance, equity, and interpretability. We give a reference architecture in which a reasoning agent is wrapped by a verification gate and a clinician escalation path, and we formalise that gate. We prove two guarantees: escalation is monotone in safety, so widening it never increases expected harm, and a risk-weighted gate minimises expected harm for any fixed amount of clinician effort. We validate the design with a fully reproducible simulation of an agentic clinical triage loop. The escalation gate trades autonomy for safety along a smooth curve, halving expected harm as oversight rises; layering an independent verifier and human review cuts the unsafe-action rate from 0.230 with the agent alone to 0.077; and risk-weighted escalation attains lower harm than confidence-only escalation at every clinician budget, exactly as the theory predicts. The results argue that in medicine the safety of an agentic system is engineered at the level of the loop, through verification, risk-weighted escalation, and clinician oversight, and can be designed and reasoned about rather than left to chance.
Setu S. M. Kodi, Sudheer Singamsetty· International Journal of Com...· 0 citations
Agent-safety benchmarks measure different behaviors, and their scores get quoted interchangeably as an agent's safety. We treat four of them (R-Judge, InjecAgent, AgentHarm, AgentDojo) as measurements to be validated, running each under its official implementation and author-provided scorer on up to 22 models, with MMLU and GPQA measured by us under one protocol as a capability composite. The metric is the first problem. On any binary trace-judgment benchmark scored by $F_1$, an ``always positive''policy attains $F_1 = 2\pi/(1+\pi)$; on R-Judge that is $0.690$, above five of the 21 models that actually discriminate. The three broad-coverage benchmarks then rank the same 18 models differently, and the trade-off behind that disagreement is a small-panel artifact: R-Judge specificity against AgentHarm safety correlates $-0.64$ at $n{=}7$ and $+0.02$ at $n{=}18$, and a quarter of random size-7 subsets reach $|\rho| \geq 0.5$ around that near-zero value. Held-out validity turns on which outcome you pick. Capability predicts task success ($\rho{=}{+}0.60$) but correlates negatively with misalignment safety ($\rho{=}{-}0.44$, $n{=}21$). On their paired $n{=}20$ panel, the corresponding contrast is $\Delta{=}{-}1.00$ (95% CI $[-1.48, -0.49]$, $p<0.001$), and it survives leave-one-organization-out and organization-clustered bootstrap analyses. On an expanded 41-model panel, the misalignment correlation weakens to $-0.16$ (95% CI $[-0.54, +0.22]$) and jailbreak strengthens to $+0.34$, though neither change is significant. \mbox{AgentHarm} shows the strongest held-out association, $\rho{=}{+}0.72$ with three-template jailbreak safety after controlling capability. But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety. Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.
You Wang, Xiao Han, Ding-Yan Shang et al.· arXiv.org· 0 citations
This case shows how evaluation-design validity can be checked structurally before model inference and why base correctness does not determine intervention-response fidelity.
This paper proposes SafeKeep, an inference-time safeguard that decouples safety judgment from tool execution: it assesses requests using flattened textual tool specifications while retaining the original schema-formatted specifications for execution.
