A Virtual Lab for Learning AI Security and Adversarial Prompt Engineering
TL;DR
This paper presents a virtual learning environment that integrates a secure sandbox with access to multiple commercial and open-source LLMs to support experiential learning in AI security.
Abstract
Computing education increasingly focuses on teaching secure coding and developing secure web applications. Now, with the rise of generative AI, we face new challenges, particularly the potential misuse of this technology for identifying and exploiting software vulnerabilities. This paper presents a virtual learning environment that integrates a secure sandbox with access to multiple commercial and open-source LLMs to support experiential learning in AI security. The lab offers scenario-based exercises that cover web, code, and system-level vulnerabilities. Students craft adversarial prompts, test LLM-generated exploits, and analyse models behaviour using established metrics such as attack success rate (ASR), exploit generation accuracy, and refusal rate. Our expert validation revealed distinct model behaviours: GPT-4o achieved the highest ASR (77.5% in web testing), demonstrating consistent exploit generation, while Claude exhibited the highest safety refusal rate (32.56%). By engaging with these observable outcomes, students develop a critical understanding of LLM capabilities, limitations, and ethical risks. We argue that AI-integrated virtual labs are essential for preparing students to work responsibly with emerging AI-assisted security tools.