This work proposes MITRE-SAGE, a multi-agent retrieval-augmented generation framework that integrates semantic and structural cybersecurity knowledge to improve the reliability and interpretability of LLM-based QA systems and proposes MITRE-QA, a comprehensive benchmark for evaluating LLMs across diverse cybersecurity knowledge tasks.
Abstract
Effective cybersecurity operations require timely and accurate analysis of large-scale heterogeneous security information; however, analysts increasingly struggle with information overload, alert fatigue, and time-constrained decision-making. Although large language models (LLMs) have demonstrated promising capabilities for question answering (QA), their effectiveness in cybersecurity remains limited by insufficient domain knowledge, a tendency to hallucinate, and difficulties in capturing both semantic and structural relationships. This work proposes MITRE-SAGE, a multi-agent retrieval-augmented generation framework that integrates semantic and structural cybersecurity knowledge to improve the reliability and interpretability of LLM-based QA systems. By decomposing complex tasks into query interpretation, evidence retrieval, and answer synthesis, MITRE-SAGE effectively supports cybersecurity tasks such as vulnerability assessment, threat profiling, and relationship extraction. Furthermore, we propose MITRE-QA, a comprehensive benchmark comprising 3,000 question-answer pairs for evaluating LLMs across diverse cybersecurity knowledge tasks, and use it to systematically evaluate MITRE-SAGE against representative baseline methods. Extensive experiments demonstrate that MITRE-SAGE consistently outperforms standalone LLMs and conventional RAG approaches. Notably, a lightweight configuration comprising Qwen2.5-7B sub-agents and a Qwen2.5-14B orchestrator achieves superior performance on five of the eight benchmark tasks, indicating the effectiveness of the proposed multi-agent framework. The results highlight the potential of MITRE-SAGE as a scalable and interpretable approach for reliable cybersecurity QA, while MITRE-QA provides a standardized benchmark for future research.
To improve cybersecurity across industries, Cyber Threat Intelligence (CTI) is becoming increasingly crucial. This systematic review explores how CTI practices are evolving in response to advancements in Artificial Intelligence (AI), particularly in the context of Large Language Models (LLMs). We examined 61 peer-reviewed studies using the PRISMA methodology, which demonstrates a strict selection procedure founded on specified inclusion, exclusion, and quality standards. This approach aligns with the scope of similar systematic reviews in the field of cyber threat intelligence. The review provides a comparative synthesis of CTI research capabilities across threat detection and prediction, attribution, forecasting, and automated reporting. We classify these approaches into three categories: conventional methods, those enhanced by AI and Machine Learning, and those based on LLMs. Our findings indicate that LLMs offer significant advantages in contextual reasoning, processing unstructured threat intelligence, and generating actionable mitigation plans. However, challenges such as model explainability, data privacy, system interoperability, and standardization impede their integration into operational environments. In addition to highlighting the potential and practical limitations of LLMs in CTI, this study identifies research gaps and proposes methods to create scalable, secure, and flexible CTI systems that support real-time cyber defense.
Hilalah Alturkistani, Abdul Ghafar Jaafar, S. Chuprat et al.· International journal of res...· 0 citations
Artificial intelligence improves many cybersecurity areas, from proactive network monitoring to reactive incident handling. Since cybersecurity copes with critical decisions and data, approaches often aim to augment human intelligence instead of replacing it. Intelligence augmentation in mind allows designing tools that empower humans to process larger volumes of data, solve issues faster, and make cybersecurity analysis easier. In this paper, we propose how to apply large language models in the Resilmesh platform for cyber situational awareness. We implemented a visualization panel in the platform's dashboard and a service answering a user's questions concerning the current network state based on the content of a graph database. They were used for orientation in graph data and prioritization of vulnerabilities. Our results reveal that language models, retrieval augmented generation, and meta prompting usually allow answering clear and straightforward questions correctly, while sometimes not strictly following formatting instructions present in prompts. It implies that analysts could create their own analysis easily, quickly, and on demand. Despite several advantages, models must be instructed concerning several details in the future, such as outputting only distinct values from the database and not making up their own results when a question has no answer.
Lukás Sadlek, Milan Boháček, Jan Rolínek et al.· IEEE Conference on Network S...· 0 citations
This paper provides a tutorial and survey of LLM-based agents for automated vulnerability discovery and penetration testing. The rapid advancements in Large Language Models (LLMs) have opened new possibilities for their application in cybersecurity. Unlike traditional tools, LLMs can process natural language, retain context, and perform structured reasoning, enabling automation of complex security workflows. By equipping these models with memory mechanisms and tool-use interfaces, researchers have developed LLM-driven autonomous agents capable of performing specialized tasks such as web application testing, source code auditing, fuzzing, and network security assessment. Multi-agent architectures have emerged in which agents assume specialized roles (e.g., distinct penetration-test phases or exploit types). In addition to surveying existing applications, this work provides a step-by-step walkthrough of integrating agentic AI into penetration testing workflows. The walkthrough explores four phases: (I) mission scoping and prompt engineering for test definition and constraint enforcement, (II) autonomous exploration and tool selection for target interaction, (III) vulnerability hypothesis formation and verification through experiment design and feedback, and (IV) payload generation and refinement to transform validated findings into concrete exploits. The survey synthesizes 155 cited sources (2022–early 2026) using explicit inclusion/exclusion criteria, with coverage extending to reasoning-class models, the Model Context Protocol ecosystem, autonomous bug-bounty agents, and standardized agentic security benchmarks. Despite progress, challenges persist, such as model hallucination, context limitations, operating costs, and ethical concerns. Future directions include advancements in multi-agent collaboration, mechanisms for agent self-improvement, and improved transparency into agent behavior.
Robel Addis, Kiran Porter, J. Ryan et al.· IEEE Access· 0 citations
The Multi-Level Protection Scheme (MLPS) is a foundational system in China's cybersecurity governance framework. Therefore, accurate analysis and understanding of MLPS requirements are essential. At present, MLPS analysis still relies mainly on manual interpretation of standards and rule-based tools. This makes it hard to provide stable and consistent compliance analysis in complex application scenarios. The rise of large language models has created new opportunities for making MLPS work more intelligent. However, in standards-intensive and security-sensitive scenarios, general-purpose large language models often cannot ensure controllable reasoning or complete understanding of rules. This paper proposes a large language model framework for MLPS that integrates multiple retrieval strategies. It combines hierarchical retrieval, tree-based retrieval, and tokenization-based matching retrieval. This design helps maintain retrieval coverage while reducing the interference of irrelevant context in the reasoning process. To address the requirements of MLPS question answering for clause accuracy, conclusion traceability, and practical deployability, this paper adopts a evaluation method based on multi-dimensional weighted scoring to quantitatively assess model responses. In comparative experiments on ten typical questions, the proposed domain-specific large language model for MLPS achieved higher overall scores.
Qian Li, Zhenyan Qi, Liang Shen et al.· 0 citations
Cyber-capable AI agents combine language models with tools, memory, and execution environments to perform multi-step offensive-security tasks. Existing work separately measures cyber capability and catalogs attacks against agent components, but provides less guidance on containing a capable agent within the environments used to evaluate it. This review synthesizes five vulnerability classes at that boundary: multi-step offensive chains, objectives that conflict with sandbox boundaries, supply-chain and credential exposure, persistent command-and-control, and the speed of automated action. We use two separate preliminary incident records: the reported July 2026 Hugging Face/OpenAI evaluation breach and Anthropic's subsequent three-incident evaluation review. A comparative evidence protocol distinguishes record-specific factual claims from the shared systems lesson: the evaluation environment is itself part of the security boundary. Across the taxonomy and records, we examine controls for containment, privilege separation, provenance, and responder access, including the dual-use problem that defensive artifacts may also enable misuse. The review identifies practical priorities for evaluating cyber capability together with the security of the environment in which that capability is exercised.
Cyber Threat Intelligence (CTI) enables Security Operations Centers (SOCs) to understand adversary behavior,
prioritize risks, and respond to cyber threats. However, cur-rent CTI workflows still depend heavily on manual analysis
of unstructured threat reports, vulnerability advisories, open-source intelligence, social media posts, and structured feeds. This
creates operational latency, inconsistent extraction quality, weak provenance, and limited scalability. This paper presents Auto
Threat AI, an agentic and explainable framework for automated CTI extraction, correlation, scoring, and analyst-governed SOC
operationalization. The proposed framework integrates determin-istic indicator extraction, Natural Language Processing (NLP),
schema-guided Large Language Model (LLM) agents, graph-aware threat correlation, bounded risk scoring, evidence-first
explainability, and Human-in-the-Loop (HITL) governance. The system ingests heterogeneous CTI sources, extracts entities and
relations such as IOCs, CVEs, malware, campaigns, threat actors, tools, and techniques, constructs a threat knowledge graph,
gen-erates campaign candidates, and presents risk-ranked intelligence through a SOC dashboard. Experimental evaluation on
safe demonstration CTI data shows that the implemented prototype ingested 6 sources, extracted 36 entities, generated 33
relations, detected 11 threat events, identified 5 campaign candidates, and routed 8 items for HITL review. The results
demonstrate that Auto Threat AI can reduce manual CTI processing effort while improving traceability, explainability, and
analyst trust.
Methari Keeravani· International Journal for Re...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 18, 2026
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
