Jul 2026· Annual International Computer Software and Applications Conference· pp. 2883-2890· 0 citations· 28 references
Abstract
Penetration testing reports are a critical artifact in the cybersecurity workflow, yet their technical complexity frequently limits their utility for non-specialist stakeholders involved in risk remediation decisions. This paper investigates the feasibility of using four open-weight large language models, DeepSeek-r1:32b, Qwen3.5:35b, Gemma4:31b, and GLM-4.7-flash:32b, to generate plain-language summaries of penetration testing reports. A corpus of 65 publicly available pentest reports was used for evaluation. Model outputs were assessed across four dimensions: readability, technical term density, semantic similarity to the source document, and factual correctness using LLM-as-a-judge evaluation. Two classical extractive methods, LSA and TextRank, were included as baselines. Readability analysis using seven established metrics showed that Qwen3.5 and Gemma4 produced the most accessible summaries, reducing mean Flesch Reading Ease scores from 25.2 in the originals to 49.2 and 52.1 respectively, and lowering grade-level scores from post-graduate to high-school equivalents. Results across the remaining evaluation dimensions further indicate that appropriately selected open-weight LLMs can produce accessible and factually grounded summaries of technical security documents, offering a practical alternative to proprietary solutions in privacy-sensitive deployment contexts.
An evaluation of five state-of-the-art LLMs on OmniVul reveals distinct performance gaps, with top-1 accuracy remaining below 50% on average for vulnerable code detection and CVE identification, demonstrating that current models lack critical reasoning capabilities for reliable vulnerability assessment.
Vishnu Teja Kandalam, Viet Duong, Xiaochang Li et al.· Proceedings of the 32nd ACM...· 0 citations
This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting and proposes a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification.
Background: LLM-as-a-judge workflows are increasingly used to evaluate open-ended model outputs, but the judge model can itself become a source of error in safety assessment. SafetyJudge-LLM audits local open-weight LLMs as semantic safety judges. Methods: This study reused a fixed set of previously reviewed safety-boundary responses and their hidden reference labels. Two independent human evaluations (R1 and R2) quantified reference-layer ambiguity. Seven local open-weight judge models were evaluated under a common Ollama inference protocol. A paired C6 sensitivity analysis reran llama3.2:3b and qwen3:8b through Hugging Face Transformers. Results: The final judge-output matrix contained 10,612 retained outputs. R1–R2 agreement was 95.45% (Cohen’s κ = 0.612) overall but 47.80% (κ = 0.341) in secondary cases. Several judge models detected more than 90% of confirmed safety-boundary failures, but high detection was not always accompanied by low false-unsafe behavior on control cases. Output-format reliability also varied across models: overall label parseability was 98.11%, while strict JSON schema compliance was 92.55%. The llama3.2:3b schema-failure rate persisted across engines (52.06% under Ollama; 59.60% under Transformers), whereas qwen3:8b maintained complete compliance. Conclusions: SafetyJudge-LLM shows that local open-weight LLMs can support semantic safety judging, but their reliability must be evaluated across multiple dimensions.
Cătălin Anghel, A. Anghel, M. Craciun et al.· Applied Informatics· 0 citations
A large empirical study on using LLMs to generate CI configurations from natural language across services and model families suggests that similarity and validity are distinct objectives for CI generation and motivate schema-aware evaluation and tooling for LLM-based configuration generation.
Findings show that evidence-grounded orchestration can substantially improve the performance of collaborative SLMs for supporting interpretation of malware detonation reports.
A reproducible, human-validated evaluation framework applied to 13 strategies—four architectural families crossed with four reasoning variants crossed with four reasoning variants—across three SLMs spanning 3B–14B parameters, plus targeted ablations.
Balaji Venktesh, Amsaprabhaa M, G. Sundaram· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
