Defensive Reverse Engineering of LLM Applications: A Black-Box Framework for Security Risk Scoring and Mitigation

Abstract

Large language model (LLM) applications now combine hidden prompts, retrieval pipelines, memory stores, content filters, tool calls, delegated identities, and downstream automation. Security reviewers are increasingly asked to assess such systems without access to source code, model weights, prompt templates, vector-store configuration, or internal logs. This paper presents D-RELLM, a defensive reverse-engineering framework for black-box security assessment of deployed LLM applications. The framework maps observable application behavior to an attack-surface graph, executes bounded and non-destructive probes, converts evidence into reproducible risk scores, and links each finding to mitigation and re-test criteria. Unlike model-only jailbreak evaluation, D-RELLM treats the deployed application as a socio-technical system whose risk depends on instruction hierarchy, retrieval trust, authorization, tool agency, output handling, monitoring, and operational controls. The paper defines a threat model, a probe taxonomy, a weighted scoring equation, a confidence equation, an application-level aggregation method, an evidence schema, and a mitigation playbook. A synthetic pilot across three representative archetypes—chatbot, retrieval-augmented generation (RAG) assistant, and tool-using agent—illustrates how direct prompt injection, indirect prompt injection, retrieval poisoning exposure, excessive agency, output-handling weaknesses, and sensitive-information disclosure can be prioritized before and after remediation. The result is a practical, auditable, and publication-ready method for defenders who need to quantify residual LLM application risk from black-box evidence while avoiding unsafe exploitation.