Layered security and trust mechanisms for GenAI applications in critical domains

Abstract

Large Language Models (LLMs) are increasingly being deployed in critical domains such as healthcare, finance, and public infrastructure to support intelligent decision-making and conversational interactions. However, these systems introduce significant challenges related to security, reliability, and trustworthiness. Vulnerabilities such as adversarial prompt injections, behavioral manipulation, and multi-stage attacks can lead to unsafe outputs, privacy risks, and loss of user trust. There is a need for robust approaches that ensure both safe application-level interactions and adaptive system-level defenses against evolving LLM threats. In this thesis, we propose a unified two-layer approach to enhancing the trustworthiness and security of LLM-enabled systems. At the application layer, we develop EmpathAI, a RAG-based mental healthcare chatbot that incorporates source tagging, sentiment-aware context retrieval, and a two-layer defense mechanism using regex filtering and prompt engineering to mitigate prompt injection attacks. Building on this, at the system layer, we introduce the Adaptive LLM Threat Response (ALTR) framework, which integrates behavioral anomaly detection, context-aware prompt classification, and temporal threat memory to identify and mitigate adversarial interactions in real time. We evaluate both layers using conversational datasets and adversarial interaction traces. At the application layer, EmpathAI achieves high semantic alignment (similarity scores >0.80–0.85), with all prompt injection classes successfully mitigated. At the system layer, ALTR attains strong detection performance (accuracy and AUC of 0.961, false-negative rate of 0.9 percent) under low-latency constraints (<20 ms). Together, these results demonstrate that securing LLMs in critical domains requires both application-layer trust and systemlayer defense, and that combining domain-aware RAG systems with adaptive multi-layer security frameworks enables their trustworthy and reliable deployment in high-risk environments.