Jul 2026· ACM Transactions on Software Engineering and Methodology· 0 citations· 53 references
TL;DR
An autonomous defense agent is designed: NymGuard, which provides a proactive defense by automating the real-time detection of hallucinated packages, their preemptive registration to block malicious use and the maintenance of a public warning registry.
Abstract
In code generation tasks, large language models (LLMs) frequently recommend nonexistent software packages, referred to as package hallucinations. This may pose a security risk, as attackers could register these fake packages and exploit them in supply chain attacks. To investigate the real-world feasibility, this paper conducts a systematic empirical study of this potential threat. We extracted 107 hallucinated packages from eight prominent LLMs and uploaded them to the npm registry over six months. The results demonstrate a tangible risk: 35.2% of package installations are attributed to developers, with most initial interactions occurring within a two-week window post-registration. A name similarity analysis further reveals that 25.2% of hallucinated names fall within an absolute edit distance of \(\leq\) 1 from existing npm packages, posing a direct typosquatting risk. In our evaluated setting, tool-augmented generation, specifically using web search, fails to mitigate this threat. Instead, it significantly amplifies hallucination persistence from 11.2% to 17.3% through context contamination. In the npm registry, the entire attack campaign costs only an estimated $496, making it far more accessible than traditional supply chain attacks yet capable of enabling poisoning effects. To mitigate this emergent threat, we design an autonomous defense agent: NymGuard. It provides a proactive defense by automating the real-time detection of hallucinated packages, their preemptive registration to block malicious use and the maintenance of a public warning registry. The full implementation has been open-sourced and is available on GitHub (https://github.com/xiaoye798/Package-Hallucination-Research).
The results recast package hallucination as both a measurement problem and a decoding-time control problem, and they demonstrate that the choice of defense must be matched to the threat model and recommendation utility.
Albérick Euraste Djiré, Iyiola E. Olatunji, Melissa Tessa et al.· 1 citation
A two layer detector to counter'slopsquatting of Python package names, with findings that hallucination rate scales almost linearly with prompt adversariality, and when primary and fallback share a model family, approximately 84% of primary failures recur on the fallback, motivating cross family pairing.
LLM agents acquire new capabilities by downloading skills from open registries. Instead of browsing these catalogs manually, developers typically ask the agent to recommend and install a skill. This convenience hides a risk: agents frequently invent names for skills that exist in no registry. We term this flaw skill name hallucination. A fake name may seem harmless, but it opens the door to supply-chain attacks. Because registries rarely verify publishers, an adversary can prompt the agent, collect the fake names it returns, pre-register malicious skills under them, and wait for a victim to install the payload. We conducted the first large-scale measurement of skill name hallucination, evaluating 15,000 prompts across 12 configurations (4 standalone LLMs and 8 agents). We conservatively counted a name as hallucinated only if it was missing from all live registries and GitHub. The results reveal a systemic vulnerability: every configuration hallucinates. Rates average 36.0% for standalone LLMs and 36.9% for agents, rising to 43.1% on real-world developer questions. In total, the systems generated 5,669 distinct hallucinated names. Crucially, these names are not random noise. Agents repeat the same fake names across prompts and models, giving attackers highly reliable targets to hijack. Finally, we tested four model-level defenses and found a severe conflict between security and usability. The strongest, retrieval grounding, cut the hallucination rate from 40.8% to 3.2% but crippled usefulness: even the best-defended system recommended the correct skill only about one in six times. Skill name hallucination is thus a highly exploitable vulnerability requiring minimal attacker effort. Fixing it cannot rely on prompt engineering or model tuning alone. It demands ecosystem-wide structural changes: registry-level name reservations and verified recommendation pipelines.
In a closed-inventory large language model (LLM) system such as Online-CADCOM, which recommends engineering tools from a verified inventory, we measure inventory non-compliance, that is, a mention-level event in which the model recommends a tool not present in the verified inventory. We use this inventory-relative sense of hallucination throughout: an out-of-inventory mention may be a fabricated tool or a real commercial tool absent from the curated inventory, so the metric reports inventory non-compliance rather than factual fabrication. We evaluate a three-mechanism mitigation stack consisting of database-grounded context injection, fixed vocabulary constraints, and enforced JavaScript Object Notation (JSON) output across three commercial LLM providers (OpenAI, Anthropic, Google), two model generations, and two output modes (standard and reasoning), totaling 6912 Application Programming Interface (API) calls over 12 configurations. Under a recall-equalized detector adopted as the primary metric, the inventory non-compliance rate, which we denote the hallucination rate (HR) following common usage, decreases from roughly 69–80% to 4–13% under the full architecture. The cross-provider average is similar across the two generations tested (8.5% Generation 1 (Gen1), 6.9% Generation 2 (Gen2)), although per-provider directions diverge. We also examine the C3 configuration, in which only JSON output enforcement is active without grounding. A naive detector reports a large hallucination increase over the unconstrained baseline (+10.1 percentage points (pp) Gen1, +15.1 pp Gen2), but we show this gap is largely a detection-format artifact: structured JSON fields make out-of-inventory tools easy to extract, whereas the same real tools are frequently missed in free text. Under a recall-equalized detector the gap narrows to +2.6 pp (Gen1) and +4.8 pp (Gen2) and remains statistically significant only for two current-generation models, indicating a small, current-generation effect rather than a universal one. Reasoning-mode models provide no statistically significant improvement under architectural constraints. A frequency-weighted audit shows that the majority of remaining out-of-inventory mentions correspond to real engineering tools absent from the platform’s inventory. Under the full architecture, roughly half of responses (pooled Pany≈49.5%) still contain at least one such mention, indicating that handling unseen tools remains an open challenge for closed-inventory recommendation systems. Our evidence comes from a single engineering platform with four related electronic-design and power-electronics domains, so the findings characterize this setting rather than recommendation domains in general.
Lavdim Menxhiqi, Galia Marinova· Applied Informatics· 0 citations
Large Language Models (LLMs) introduce serious risks of content misuse, spanning copyright infringement in the legal domain and plagiarism in the ethical and academic domain. Although prior work has studied these risks, researchers and practitioners still need practical ways to audit, interpret, and evidence them. This tutorial presents a unified forensic perspective on LLM content misuse. First, we introduce Copyright Detective, an interactive forensic system for detecting, analyzing, and visualizing potential copyright leakage in LLM outputs. Participants will learn how inference-time scaling reveals sporadic memorization under output uncertainty, and how persuasive jailbreak probing can serve as defensive red teaming for examining alignment-suppressed leakage. Second, we introduce LLM Plagiarism Detection, covering verbatim copying, paraphrased reuse, and idea-level appropriation. We will discuss, and where appropriate demonstrate, how candidate source retrieval and text alignment support plagiarism analysis, while highlighting factors such as model size, decoding strategies, and fine-tuning corpus similarity. By combining hands-on copyright-risk auditing with a flexible plagiarism module, this tutorial equips attendees with practical and conceptual tools for auditing black-box models, interpreting similarity evidence, and understanding content misuse beyond simple text matching.
Denghui Zhang, Guangwei Zhang, Dongwon Lee· Proceedings of the 32nd ACM...· 0 citations
With the increasing usage of AI-generated code in software development workflows, new security challenges and concerns arise. This paper analyzes five LLMs: ChatGPT, Claude, Gemini, DeepSeek, and Grok in three phases of security assessments against web vulnerabilities listed by the OWASP Top 10. Phase 1 (December 2025) evaluated 13 prompts in XSS, Authentication, API Security, Hardcoded Secrets, and React Server Component CVE. Phase 2 (March 2026) evaluated 11 prompts using newly disclosed CVEs, complex authentication, and API security, with updated models. Phase 3 (May 2026) revisited phase one prompts against current models. Across all 185 samples, 49 out of the 120 Phase 1 and 2 samples (40.8%) were identified as vulnerable, 65 (54.2%) as secure and 6 (5.0%) partially vulnerable. The CVE-based prompts recorded the highest percentage of vulnerability (76%). No failures were observed in the authentication prompts with simple patterns, while complex access control resulted in 45% failures. Phase 3 (65 samples) re-ran Phase 1 prompts on current models; no progress was observed; the partial result rate went up from 7.7% to 27.7%. The results revealed hedged responses from the updated models. All results were cross-validated using Semgrep static analysis.
Malak Mansour, Anas AlMajali· IEEE Jordan Conference on Ap...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
