World Model Assisted Safety Planning is proposed, a model-predictive shared autonomy framework for explicit safety constraint satisfaction that substantially reduces safety violations while preserving task performance and reducing unnecessary interventions over prior shared autonomy baselines.
Requirements engineers for agentic-AI domains face challenges in evaluating, specifying, and operationalizing safe autonomy. Mainstream frameworks, such as Goal-Oriented Requirements Engineering (GORE), lack mechanisms to systematically address these challenges under epistemic uncertainty. We contribute an approach that builds on GORE to model and simulate safe autonomy in agentic-AI systems. We introduce a novel Regret-Dominance Mechanism (MS-RGR) to operationalize safe autonomy. MS-RGR uses two signals: epistemic surprise (novelty detection) and cognitive regret (evaluative risk) to address the trilemma problem: should the agent operate in routine autonomy, undergo reflective reasoning, or escalate to human? We instantiate MS-RGR in elderly care monitoring and autonomous driving. A 100-seed stochastic simulation shows MS-RGR reduces silent failures to near-zero and detects risk approximately 17.5 times faster than a sensor-only baseline, remaining formally traceable via LTL safety properties. A retrospective proxy instantiation applying the DRI gate post-hoc over execution traces from 208 AGENTHARM scenarios across seven LLMs shows the gate improves harmful-task refusal only for models with strong baseline safety (over 80% pre-gate refusal, e.g., 84.1% to 90.9%), indicating MS-RGR amplifies rather than substitutes for model-level safety training. We discuss threats to validity, positioning MS-RGR as initial feasibility evidence for design-time safety constraints in agentic-AI requirements engineering.
SafeBranch is proposed, a framework that aligns an embodied actor on safety through branch pairs constructed from the actor's own unsafe rollouts via environment rollback, achieving roughly ten times more safe successes than the untrained baseline on the unseen-object variant.
Hyunse Lee, Jiwoo Jeong, Haneul Lee et al.· 0 citations
Diffusion-based trajectory planners achieve strong nominal performance in autonomous driving, but sparse safety intervention remains difficult to evaluate and realize effectively. This study addresses this problem by proposing a safety-aware event-triggered intervention framework on top of a fixed DiffusionDriveV2 planner. The method uses candidate-level risk signals and an auxiliary semantic risk trigger to decide when intervention should be activated, and realizes the intervention through conservative re-selection and mild action-space augmentation. To evaluate sparse interventions beyond global validation metrics, we further construct normal, conservative, and actual trajectories and introduce triggered-subset counterfactual evaluation. On NAVSIM navtest, global planner metrics remain nearly unchanged across sparse trigger policies, but the semantic trigger achieves better triggered-subset final score, TTC, and progress than matched-random and TTC-based triggers. Qualitative cases show that behaviorally distinct safety responses mainly arise from action-space augmentation rather than candidate reranking alone. These results show that the proposed framework can diagnose and partially alleviate the gap between risk recognition and action realization, while revealing that stronger semantic-conditioned action generation is needed to fully overcome the trigger-to-action bottleneck.
Xue-Rui Fang, Hui Li, Ze-Hao Xue et al.· Big Data and Cognitive Compu...· 0 citations
We consider a task planning scenario in which robots sharing a persistent environment are assigned tasks one at a time from a held-out sequence. Standard task planners, lacking foresight of future tasks and inconsiderate of others'constraints, solve each task in isolation, leaving terminal states that increase future cost for all, side effects that compound over lengthy task sequences. To reduce cost over the sequence, a robot must anticipate how its actions now may impact performance on future tasks for all robots sharing the environment. Therefore, we present courteous anticipatory planning, wherein a model-based planner proposes candidate plans and selects the one that jointly minimizes immediate cost and aggregated expected future cost across all robots, estimated via independent per-robot learned estimators. This factored formulation avoids combinatorial joint rollouts and supports modular deployment: adding a robot requires only training its own estimator. We evaluate in two persistent PDDL domains, a home environment with robots that have similar capabilities but different responsibilities, and a restaurant environment where robots'distinct capabilities create states that other robots lack the capability to resolve. During lengthy task sequences, our planner reduces total cost by 10.43% versus myopic and 4.03% versus selfish anticipatory planning in a two-robot home environment and by 17.41% and 13.24%, respectively, in a three-robot restaurant.
Md Ridwan Hossain Talukder, Roshan Dhakal, Elizabeth Phillips et al.· arXiv.org· 0 citations
World models have progressed from compact latent dynamics to generative, controllable, and interactive simulators of embodied environments. However, high predictive likelihood and visual fidelity do not necessarily ensure that a model preserves the evidence required for safe decision-making. This perspective identifies three structural mismatches in current world modeling: likelihood versus risk, prediction versus intervention, and finite-horizon prediction versus accumulated consequences. We propose the Risk-Informed World Model (RIWM) as a decision-centric research direction for safety-critical embodied systems. RIWM organizes world modeling around consequences, intervention, epistemic uncertainty, and recoverability, and integrates four interdependent capabilities: decision-relevant representation, counterfactual reasoning, safety-critical episodic memory, and runtime safety assurance. It distinguishes physical, social, and operational consequences while using epistemic uncertainty to qualify the evidence supporting action. We further discuss open challenges in identifying consequential futures, validating counterfactual reasoning, maintaining revisable safety memories, translating learned consequences into executable constraints, and determining when evidence is sufficient to act. This perspective argues that future world models should move beyond predicting likely futures toward identifying which futures matter, revising judgments through experience, and recognizing when to act, revise, sense, defer, or abstain.
Kai-Lang Ma, Heyang Huang, Inhi Kim et al.· 0 citations
Real-time motion planning for autonomous racing must operate under strict computational deadlines while maintaining safe and reliable behavior. However, the quality and safety of planning algorithms depend heavily on the computational budget available on the deployment hardware. Policies tuned on powerful systems can become unsafe on resource-constrained platforms, where limited planning iterations leave aggressive actions insufficiently explored. While existing planning approaches may limit the size of their search space, they do not explicitly adapt the risk profile of sampled actions to the available relative deadline. These approaches overlook how limited computational resources leave aggressive actions insufficiently explored, thereby compromising safety. We present Deadline-Aware MCTS (DA-MCTS), which adapts action exploration in continuous-space MCTS according to the available relative deadline. Our approach partitions the action space into conservative, aggressive, and Gaussian prior sampling categories, and uses a neural network that maps runtime features to a probability distribution over these categories. This enables safe behavior under tight budgets, where sampling shifts toward conservative actions, and improved performance as additional compute becomes available. We further extend DA-MCTS with a friction-aware feature representation that allows the planner to adapt its behavior and generalize to previously unseen friction conditions. In F1TENTH racing simulations with varying maps, friction conditions, and deadlines, we demonstrate that DA-MCTS drives conservatively and collisionfree across a wide range of friction conditions under tight deadlines while achieving progressively faster lap times as deadlines increase. We further validate these results with F1TENTH hardware-in-the-loop experiments. Source code is available at https://github.com/scope-lab-vu/DA_MCTS_RTCSA_2026
Samir Gupta, Michael Yuhas, Abhishek Dubey· IEEE International Conferenc...· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
