MCSafe-GSN-HOL: A Formal Assurance Framework for Machine-Checkable Safety Cases of Deployed LLM Agents
Abstract
Deployed large language model (LLM) agents are now being used to interface with external tools, fetch information, run code, interact with user data and help with decision making at the workflow level. Therefore, their safety issues are not only related to the underlying model, but also to tool permissions, prompt design, retrieval sources, policy constraints, evidence validity, and varying operational contexts. While Goal Structuring Notation (GSN) has become a popular means of structuring safety arguments, many GSN-based safety cases are still reliant on manual inspection and can suffer from unsupported goals, incomplete evidence paths, undischarged assumptions, circular arguments, unaddressed defeaters, or insufficient coverage of hazards. This paper introduces MCSafe-GSN-HOL, a formal assurance framework for creating machine-checkable safety cases for deployed LLM agents that overcomes these disadvantages. The proposed framework translates the structured GSN arguments into Isabelle/HOL proof obligations, and formalizes the typed GSN elements, the hazard contracts of LLM-agents, the evidence-completeness conditions, the assumption-discharge rules, the defeater-resolution rules, and the incremental re-verification rules after deployment changes. The prototype was tested with 25 safety-case fragments from five representative scenarios of LLM-agents. Results demonstrate that MCSafe-GSN-HOL has 97.3% defect detection, 97.2% evidence coverage, 96.4% hazard-control coverage, and 62.7% reduction in review time compared to manual GSN review. The mean Isabelle/HOL checking time was 2.53 s/safety case, showing that theorem-prover supported assurance can still be practical for medium scale safety-case checking. The results overall support the conclusion that formally checked GSN safety cases can help with the traceability, auditability, consistency and maintainability of assurance processes for deployed LLM-agent systems.