Results indicate good agreement between the deployed model and independent LLM judges, demonstrating scalability and reduced analyst workload in safety‑critical monitoring contexts.
Abstract
This paper introduces a Customer Threat Detection Model leveraging a pre-trained large language model (LLM) on a major cloud platform to analyze customer service call transcripts and social media posts for potential security threats. The solution was developed in response to a critical need by the corporate security team to proactively identify threats during high-risk periods—such as the Southern California wildfires in January—when call volumes to the Customer Contact Center surged and employees and property faced elevated safety risks. Historically, manual identification of threats was slow and inconsistent, creating potential exposure for the organization. Operating in batch mode, the system processes daily calls and assigns each interaction a threat score (0–100), mapped to five ordinal bins from Low to High. The model combines expert-defined keywords with semantic embedding techniques to expand its threat lexicon, enabling detection of evolving language and context. Each transcript is transformed into a structured prompt and evaluated by the LLM to produce a threat score and category.Manual review sampled calls showed ~93% accuracy but proved resource-intensive and impractical for ongoing monitoring. To address scalability, we applied an “LLM-as-a-Judge” framework, where LLMs act as surrogate evaluators of model outputs. For 10K sampled calls, two summaries per call, overall and threat-focused, were generated and independently assessed by a second LLM to assign ordinal threat categories. Agreement metrics (accuracy, Cohen’s kappa, mean absolute difference), triadic consistency, and keyword sensitivity were computed. A small Keyword Influence Delta indicated strong contextual detection and guided keyword refinement.Results indicate good agreement between the deployed model and independent LLM judges, demonstrating scalability and reduced analyst workload in safety‑critical monitoring contexts.
It is shown that the quality of the generated risk indicators is the main driver of zero-shot cyber threat detection performance, and that retrieval mainly benefits weaker LLMs by generating more discriminative risk indicators, whereas stronger models achieve comparable performance without retrieved context.
A. Al-Ghamdi, S. Layeghy, Marius Portmann· 0 citations
STINER, a taxonomy and expert-annotated corpus for extracting strategic intelligence from social media streams is introduced, and how social-media-driven extraction can surface early signals of the SafePay ransomware campaign prior to its retrospective characterization in vendor threat landscape reports is illustrated.
Yasir Ech-Chammakhy, Oussama Azrara, J. Chbili et al.· 0 citations
: Insider threats represent a big challenge for organizations because insiders have legitimate access and knowledge of organizational processes. Effective defense requires both accurate detection of anomalous behavior and timely response actions. This paper proposes a unified approach that classifies log attributes by their importance for insider threat detection using process mining and an anomaly score to identify process deviations. To address the lack of automated responses, we propose to generate Incident Response (IR) playbooks for business processes using Named Entity Recognition (NER) on insider threat scenarios, combined with Part-of-Speech (POS) tagging and the RE&CT framework, a knowledge base of attack techniques and tactics inspired by MITRE ATT&CK that allows for categorizing IR techniques and actions, to map entities to response actions. The proposed approach is evaluated using coverage, redundancy, usefulness, and efficiency metrics, demonstrating its ability to produce comprehensive and compact response playbooks.
Kéren Saint-Hilaire, Nora Boulahia-Cuppens, Frédéric Cuppens· Proceedings of the 23rd Inte...· 0 citations
A role-layer framework is introduced to unify LLM risks and defenses, and identifies three key open challenges: moving from static detection accuracy to budgeted ecosystem-level risk evaluation, hardening LLM-centered verification pipelines against adversarial manipulation, and deploying auditable human-in-the-loop verification systems for trustworthy real-world misinformation defense.
Traditional signature-based Web Application Firewalls (WAFs) have difficulty detecting increasingly complex assaults that target web applications, such as SQL injections, Cross-Site Scripting (XSS), and API misuse. In this study, we introduce LLM-WAF, a new intelligent firewall architecture that uses Large Language Models (LLMs) to analyze HTTP traffic contextually and semantically. Our framework integrates pre-trained language models with realtime traffic monitoring pipelines to identify malicious payloads through natural language processing capabilities rather than static rule matching. The system incorporates a continuous learning mechanism using reinforcement signals from detected attacks to adapt to emerging threat vectors automatically. In comparison to conventional WAF systems, experimental evaluation on benchmark datasets such as the CSIC 2010 HTTP Dataset and real-world traffic scenarios shows that LLM-WAF achieves 96.8% detection accuracy with an F1=0.95cand dramatically lowers false positives.
Y. Khalaf· International Journal of Wir...· 0 citations
The increasing deployment of Large Language Models (LLMs) in critical infrastructure has introduced a class of security risks that remain insufficiently characterised and poorly tooled in practice. Attack vectors including prompt injection, jailbreaking, code execution facilitation, covert data exfiltration, and training data poisoning present quantifiable threats to systems that rely on LLM outputs, yet no lightweight, provider-agnostic tool exists to measure these risks systematically. This paper presents VectorSec, an open-source, web-based LLM security scanner built with Python and Dash. The tool executes a structured test suite of 320 adversarial prompts spanning 16 vulnerability categories aligned with the OWASP Top 10 for LLMs. Each response is scored through a four-stage pipeline combining pattern matching, sentiment analysis, semantic similarity, and secondary LLM verification. Evaluation across three open-source models demonstrates that pattern-matching-only filtering misses approximately 15% of High and Critical findings that semantic verification correctly identifies, motivating the multi-layer design. VectorSec provides interactive dashboard reporting, real-time progress tracking, and audit-ready PDF and CSV exports, lowering the barrier to structured LLM security assessment in operational settings.
M. Yamin· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
