A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools

Abstract

: AI adoption has accelerated with the rise of LLMs, and people within the field of security are increasingly exploring their practical value. Threat modelling is central to secure system development, yet it remains largely manual and the value of LLM-driven tools is unclear. Even when LLMs prove useful, selecting the right one can be more challenging than using it. This paper introduces a systematic evaluation framework for LLM-driven threat modelling tools to support tool selection, observing the general LLM-integration, governance risks, and allowing for comparison of tool output. Using Goal-Question-Metric, we derive evaluation metrics and show the value of the framework on a set of state-of-the-art LLM-driven threat modelling tools. The results show our metrics distinguish both performance and governance risks, providing a basis for organisations to ensure automation strengthens rather than burdens their threat modelling process.