A large language model framework for MLPS that integrates multiple retrieval strategies that combines hierarchical retrieval, tree-based retrieval, and tokenization-based matching retrieval is proposed that helps maintain retrieval coverage while reducing the interference of irrelevant context in the reasoning process.
Abstract
The Multi-Level Protection Scheme (MLPS) is a foundational system in China's cybersecurity governance framework. Therefore, accurate analysis and understanding of MLPS requirements are essential. At present, MLPS analysis still relies mainly on manual interpretation of standards and rule-based tools. This makes it hard to provide stable and consistent compliance analysis in complex application scenarios. The rise of large language models has created new opportunities for making MLPS work more intelligent. However, in standards-intensive and security-sensitive scenarios, general-purpose large language models often cannot ensure controllable reasoning or complete understanding of rules. This paper proposes a large language model framework for MLPS that integrates multiple retrieval strategies. It combines hierarchical retrieval, tree-based retrieval, and tokenization-based matching retrieval. This design helps maintain retrieval coverage while reducing the interference of irrelevant context in the reasoning process. To address the requirements of MLPS question answering for clause accuracy, conclusion traceability, and practical deployability, this paper adopts a evaluation method based on multi-dimensional weighted scoring to quantitatively assess model responses. In comparative experiments on ten typical questions, the proposed domain-specific large language model for MLPS achieved higher overall scores.
With the development of smart civil aviation, the cybersecurity situation for air traffic management (ATM) continues to be critical. The increasingly exposed attack surface demands more advanced technologies and methods for protection. Large language models (LLMs) have been widely applied in cybersecurity, which has provided a novel paradigm for reconstructing ATM cybersecurity defense systems. However, existing vertical domain LLMs in ATM focus on providing passenger services and supporting daily operations, which do not concern cybersecurity. This neglect is a potential risk in the transition from security to safety. In this paper, a prompt-engineering-based method for generating question–answer pairs for ATM cybersecurity is proposed. Knowledge is extracted from multi-source heterogeneous primary sources and converted into uniformly formatted question–answer pairs, and an ATM cybersecurity fine-tuning dataset and benchmark are constructed. By fine-tuning the Deepseek-llm-7B-base model and DeepSeek-R1-Distill-Qwen-14B model using both instruction fine-tuning and reasoning fine-tuning methods, based on Low-Rank Adaptation (LoRA) and full-parameter fine-tuning technologies, a vertical domain LLM named “AeroSec” for ATM cybersecurity was constructed. By comparing the fine-tuned model with the base model, DeepSeek-V3 model, and DeepSeek-R1 model through the third-party model Qwen-Plus, the useful value of AeroSec in answering questions in specialized domains such as ATM network threat analysis and offensive-defensive techniques has been validated.
Ruochen Dong, Chengkai Piao, Buhong Wang et al.· Scientific Reports· 0 citations
This work proposes MITRE-SAGE, a multi-agent retrieval-augmented generation framework that integrates semantic and structural cybersecurity knowledge to improve the reliability and interpretability of LLM-based QA systems and proposes MITRE-QA, a comprehensive benchmark for evaluating LLMs across diverse cybersecurity knowledge tasks.
Ali Habibzadeh, Farid Feyzi, Reza Ebrahimi Atani· 0 citations
The growing complexity and frequency of cyberattacks make cybersecurity risk assessment an increasingly demanding task for organisations, requiring substantial expertise, resources, and adherence to established standards. This work explores the applicability of Large Language Model (LLM) to cybersecurity risk assessment, with a focus on threat identification and risk scoring. The paper presents a standalone consistency analysis across five models, measuring accuracy and stability under lexical, structural, and noisy prompt perturbations using an OWASP-oriented rubric. Building on the analysis results, we present a modular LLM-based system that combines Retrieval-Augmented Generation, MITRE ATT&CK-Aligned threat evaluation, rubric-constrained risk scoring, and a Judge Reviewer, orchestrated through a Beliefs–Desires–Intentions control loop. The validation against incidents from the VERIS and EuRepoC datasets highlights limitations and weaknesses, and allows identifying the architectural and structural mitigations that can reduce prompt sensitivity in LLM-based risk assessment.
This paper provides a tutorial and survey of LLM-based agents for automated vulnerability discovery and penetration testing. The rapid advancements in Large Language Models (LLMs) have opened new possibilities for their application in cybersecurity. Unlike traditional tools, LLMs can process natural language, retain context, and perform structured reasoning, enabling automation of complex security workflows. By equipping these models with memory mechanisms and tool-use interfaces, researchers have developed LLM-driven autonomous agents capable of performing specialized tasks such as web application testing, source code auditing, fuzzing, and network security assessment. Multi-agent architectures have emerged in which agents assume specialized roles (e.g., distinct penetration-test phases or exploit types). In addition to surveying existing applications, this work provides a step-by-step walkthrough of integrating agentic AI into penetration testing workflows. The walkthrough explores four phases: (I) mission scoping and prompt engineering for test definition and constraint enforcement, (II) autonomous exploration and tool selection for target interaction, (III) vulnerability hypothesis formation and verification through experiment design and feedback, and (IV) payload generation and refinement to transform validated findings into concrete exploits. The survey synthesizes 155 cited sources (2022–early 2026) using explicit inclusion/exclusion criteria, with coverage extending to reasoning-class models, the Model Context Protocol ecosystem, autonomous bug-bounty agents, and standardized agentic security benchmarks. Despite progress, challenges persist, such as model hallucination, context limitations, operating costs, and ethical concerns. Future directions include advancements in multi-agent collaboration, mechanisms for agent self-improvement, and improved transparency into agent behavior.
Robel Addis, Kiran Porter, J. Ryan et al.· IEEE Access· 0 citations
Compliance with regulatory rules becomes an increasingly complex task because of the constantly changing legal regulations and policy regulations. Currently, the existing approach to compliance verification is mostly based on the manual analysis of documents and is characterized by high time consumption, inconsistency, and high levels of operational risks. To solve the problem, this paper introduces CompVault, an Enhanced Retrieval-Augmented Generation (ERAG)-based Artificial Intelligence Compliance Monitoring and Report Generation System for intelligent regulatory compliance assessment. The introduced approach combines the semantic document retrieval, vector-based knowledge representation with ChromaDB, and contextual reasoning by means of the large language model to check compliance of organizational policies with regulatory requirements. It also determines the compliance gaps and compliance risk level and generates structured reports with recommendations to decision-makers. Web-based implementation was implemented using FastAPI, all-MiniLM-L6-v2 embeddings, and LexGLUE legal benchmark as a knowledge source about regulations. As a result, the following metrics were achieved: 97.42% accuracy, 96.88% precision, 97.15% recall, 97.01% F1-score, and 98.80% AUC-ROC. Therefore, the achieved results indicate that the ERAG-based framework can be used as an efficient, scalable, and explainable solution for regulatory compliance monitoring and automated report generation.
S. N., Sathyapriya P., Vishnu Priya R M. et al.· Journal of Information Techn...· 0 citations
With more than 20,000 Common Vulnerabilities and Exposures (CVEs) reported annually, software vulnerabilities represent a critical cybersecurity challenge. This volume has intensified the demand for automated detection and analysis, motivating the integration of large language models (LLMs) for such tasks. However, existing vulnerability benchmarks are not suitable for evaluating LLMs' capabilities in vulnerability assessment, as most of them 1) rely on narrow data sources, 2) lack deep context, and 3) focus on single-turn Q&A rather than realistic, multi-stage analyst workflows. To address this gap, we introduce OmniVul, a comprehensive multi-turn benchmark for LLM-based vulnerability assessment. OmniVul comprises 2,000 CVEs with question–answer pairs spanning 23 attributes, including detection, code localization, root cause analysis, and patch suggestion. We employ an automated workflow to aggregate multi-source data via Retrieval-Augmented Generation (RAG), ensuring quality through LLM-as-a-Judge filtering and conformal prediction calibrated by human expert annotations. An evaluation of five state-of-the-art LLMs on OmniVul reveals distinct performance gaps, with top-1 accuracy remaining below 50% on average for vulnerable code detection and CVE identification. Our evaluation also demonstrates that current models lack critical reasoning capabilities for reliable vulnerability assessment. These results highlight the importance of OmniVul for advancing research in evaluating and fine-tuning LLMs for vulnerability assessment.
Vishnu Teja Kandalam, Viet Duong, Xiaochang Li et al.· Proceedings of the 32nd ACM...· 0 citations