A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots
Article Information
Abstract
Prompt injection is ranked as the most critical vulnerability in large language model (LLM) deployments by the OWASP Top 10 for LLM Applications, yet existing defenses operate at isolated pipeline stages and remain incomplete. Input filters cannot inspect retrieved documents, while output monitors cannot prevent malicious payloads from reaching the model. Consequently, retrieval-augmented generation (RAG) chatbots remain vulnerable to indirect injection, where a poisoned knowledge-base document compromises every user whose query retrieves it. We present a three-layer framework that intercepts both direct and indirect prompt injection throughout the inference pipeline. Layer 1 screens user input using a rule-based pattern library and a fine-tuned semantic anomaly classifier. Layer 2 enforces a provenance-based instruction hierarchy during context assembly, preventing retrieved content from overriding operator policy. Layer 3 audits model output using a policy rule engine and semantic drift detector before delivery. A continuous audit loop aggregates structured logs and supports retraining to adapt the classifier to emerging attack patterns. The framework is target-model agnostic: it treats the protected LLM as a black box and deploys as middleware without modifying it, while relying on lightweight auxiliary models (a MiniLM sentence encoder and Llama Guard~2) for detection. Evaluation on 5,080 samples across GPT-4o, Llama 3, and Mistral 7B - restricted to single-turn and non-adaptive attacks - shows that the framework reduces Attack Success Rate (ASR) from 71.4% to 11.3%, outperforming the best single-layer baseline by 27.3 percentage points and the evaluated NeMo Guardrails configuration by 23.8 percentage points, while maintaining a 4.8% false positive rate and a median latency overhead of 61.2 ms. Ablation studies show that all three layers contribute and that the full framework achieves a lower attack success rate than would be expected if the layers acted independently.
Graphical Abstract
Keywords
Data Availability Statement
Funding
Conflicts of Interest
AI Use Statement
Ethical Approval and Consent to Participate
References
- OpenAI. (2024). GPT-4o System Card. Retrieved from https://openai.com/index/gpt-4o-system-card
[Google Scholar] - Meta AI. (2024). Llama 3 Model Card. Retrieved from https://ai.meta.com/blog/meta-llama-3/
[Google Scholar] - Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Le Scao, T., Lavril, T., Wang, T., Lacroix, T., & El Sayed, W. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825.
[CrossRef] [Google Scholar] - Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33, 9459-9474.
[Google Scholar] - OWASP Foundation. (2023). OWASP Top 10 for Large Language Model Applications. Retrieved from https://owasp.org/www-project-top-10-for-large-language-model-applications/
[Google Scholar] - Perez, F., & Ribeiro, I. (2022). Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
[CrossRef] [Google Scholar] - Willison, S. (2022). Prompt Injection Attacks Against GPT-3. Retrieved from https://simonwillison.net/2022/Sep/12/prompt-injection/
[Google Scholar] - Halfond, W. G., Viegas, J., & Orso, A. (2006, March). A Classification of SQL Injection Attacks and Countermeasures. In Proceedings of the IEEE International Symposium on Secure Software Engineering (pp. 13–15). IEEE. https://cgi.cse.unsw.edu.au/~meyden/3441/halfond.pdf
[Google Scholar] - Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023, November). Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM workshop on artificial intelligence and security (pp. 79-90).
[CrossRef] [Google Scholar] - Branch, H. J., Cefalu, J. R., McHugh, J., Hujer, L., Bahl, A., Iglesias, D. D. C., ... & Darwishi, R. (2022). Evaluating the susceptibility of pre-trained language models via handcrafted adversarial examples. arXiv preprint arXiv:2209.02128.
[CrossRef] [Google Scholar] - Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., ... & Liu, Y. (2023). Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.
[CrossRef] [Google Scholar] - Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024). The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208.
[CrossRef] [Google Scholar] - Rebedea, T., Dinu, R., Sreedhar, M. N., Parisien, C., & Cohen, J. (2023, December). Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. In Proceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations (pp. 431-445).
[CrossRef] [Google Scholar] - Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., ... & Khabsa, M. (2023). Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.
[CrossRef] [Google Scholar] - Phute, M., Helbling, A., Hull, M., Peng, S., Szyller, S., Cornelius, C., & Chau, D. H. (2023). Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308.
[CrossRef] [Google Scholar] - Kumar, A., Agarwal, C., Srinivas, S., Li, A. J., Feizi, S., & Lakkaraju, H. (2023). Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705.
[CrossRef] [Google Scholar] - Yi, J., Xie, Y., Zhu, B., Kiciman, E., Sun, G., Xie, X., & Wu, F. (2025, July). Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1 (pp. 1809-1820).
[CrossRef] [Google Scholar] - Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., ... & Liu, Y. (2023). Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860.
[CrossRef] [Google Scholar] - Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
[CrossRef] [Google Scholar] - Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P. Y., ... & Goldstein, T. (2023). Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614.
[CrossRef] [Google Scholar] - Robey, A., Wong, E., Hassani, H., & Pappas, G. J. (2023). Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684.
[CrossRef] [Google Scholar] - Xie, Y., Yi, J., Shao, J., Curl, J., Lyu, L., Chen, Q., Xie, X., & Wu, F. (2023). Defending ChatGPT Against Jailbreak Attack via Self-Reminder. Nature Machine Intelligence, 5(12), 1486--1496.
[CrossRef] [Google Scholar] - Chen, S., Piet, J., Sitawarin, C., & Wagner, D. (2025). \{StruQ\: Defending against prompt injection with structured queries. In 34th USENIX Security Symposium (USENIX Security 25) (pp. 2383-2400). https://www.usenix.org/conference/usenixsecurity25/presentation/chen-sizhe
[Google Scholar] - Hines, K., Lopez, G., Hall, M., Zarfati, F., Zunger, Y., & Kiciman, E. (2024). Defending Against Indirect Prompt Injection Attacks With Spotlighting. arXiv preprint arXiv:2403.14720.
[CrossRef] [Google Scholar] - Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., ... & Xie, X. (2023, November). Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts. In Proceedings of the 1st ACM workshop on large AI systems and models with privacy and safety analysis (pp. 57-68).
[CrossRef] [Google Scholar] - Schulhoff, S., Pinto, J., Khan, A., Bouchard, L. F., Si, C., Anati, S., ... & Boyd-Graber, J. L. (2023, December). Ignore this title and HackAPrompt: Exposing systemic vulnerabilities of LLMs through a global prompt hacking competition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 4945-4977).
[CrossRef] [Google Scholar] - Shostack, A. (2014). Threat Modeling: Designing for Security. Wiley. https://dl.acm.org/doi/book/10.5555/2829295
[Google Scholar] - Zhang, Y., & Ippolito, D. Prompts Should not be Seen as Secrets: Systematically Measuring Prompt Extraction Attack Success (2023). arXiv preprint arXiv:2307.06865.
[Google Scholar] - Zou, W., Geng, R., Wang, B., & Jia, J. (2025). PoisonedRAG: Knowledge corruption attacks to {Retrieval-Augmented generation of large language models. In 34th USENIX Security Symposium (USENIX Security 25) (pp. 3827-3844). https://www.usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag
[Google Scholar] - Bagdasaryan, E., Hsieh, T. Y., Nassi, B., & Shmatikov, V. (2023). Abusing images and sounds for indirect instruction injection in multi-modal LLMs. arXiv preprint arXiv:2307.10490.
[CrossRef] [Google Scholar] - Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., ... & Kaplan, J. (2022). Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
[CrossRef] [Google Scholar] - Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., ... & Yih, W. T. (2020, November). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) (pp. 6769-6781).
[CrossRef] [Google Scholar] - Johnson, J., Douze, M., & Jégou, H. (2019). Billion-scale similarity search with GPUs. IEEE transactions on big data, 7(3), 535-547.
[CrossRef] [Google Scholar] - Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., ... & Rush, A. M. (2019). Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
[CrossRef] [Google Scholar] - Reimers, N., & Gurevych, I. (2019, November). Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) (pp. 3982-3992).
[CrossRef] [Google Scholar] - Loshchilov, I., & Hutter, F. (2017). Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
[CrossRef] [Google Scholar] - Meta AI. (2024). Llama Guard 2: Safeguarding Human-AI Conversations. Retrieved from https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md
[Google Scholar] - Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016). MS MARCO: A human generated machine reading comprehension dataset. OpenReview. https://openreview.net/forum?id=Hk1iOLcle
[Google Scholar] - Zheng, L., Chiang, W. L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., ... & Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36, 46595-46623.
[CrossRef] [Google Scholar] - Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
[CrossRef] [Google Scholar] - Carlini, N., & Wagner, D. (2017, May). Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) (pp. 39-57). IEEE.
[CrossRef] [Google Scholar]
Cite This Article
TY - JOUR AU - Saleem, Gulshan AU - Ahmed, Nisar AU - Zaman, Muhammad Imran AU - Hassan, Ali AU - Mujahid, Umar PY - 2026 DA - 2026/09/25 TI - A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots JO - ICCK Transactions on Information Security and Cryptography T2 - ICCK Transactions on Information Security and Cryptography JF - ICCK Transactions on Information Security and Cryptography VL - 2 IS - 3 SP - 192 EP - 215 DO - 10.62762/TISC.2026.712476 UR - https://www.icck.org/article/abs/TISC.2026.712476 KW - prompt injection detection KW - retrieval-augmented KW - generation KW - large language models KW - chatbot security KW - large language model guardrails AB - Prompt injection is ranked as the most critical vulnerability in large language model (LLM) deployments by the OWASP Top 10 for LLM Applications, yet existing defenses operate at isolated pipeline stages and remain incomplete. Input filters cannot inspect retrieved documents, while output monitors cannot prevent malicious payloads from reaching the model. Consequently, retrieval-augmented generation (RAG) chatbots remain vulnerable to indirect injection, where a poisoned knowledge-base document compromises every user whose query retrieves it. We present a three-layer framework that intercepts both direct and indirect prompt injection throughout the inference pipeline. Layer 1 screens user input using a rule-based pattern library and a fine-tuned semantic anomaly classifier. Layer 2 enforces a provenance-based instruction hierarchy during context assembly, preventing retrieved content from overriding operator policy. Layer 3 audits model output using a policy rule engine and semantic drift detector before delivery. A continuous audit loop aggregates structured logs and supports retraining to adapt the classifier to emerging attack patterns. The framework is target-model agnostic: it treats the protected LLM as a black box and deploys as middleware without modifying it, while relying on lightweight auxiliary models (a MiniLM sentence encoder and Llama Guard~2) for detection. Evaluation on 5,080 samples across GPT-4o, Llama 3, and Mistral 7B - restricted to single-turn and non-adaptive attacks - shows that the framework reduces Attack Success Rate (ASR) from 71.4% to 11.3%, outperforming the best single-layer baseline by 27.3 percentage points and the evaluated NeMo Guardrails configuration by 23.8 percentage points, while maintaining a 4.8% false positive rate and a median latency overhead of 61.2 ms. Ablation studies show that all three layers contribute and that the full framework achieves a lower attack success rate than would be expected if the layers acted independently. SN - 3070-2429 PB - Institute of Central Computation and Knowledge LA - English ER -
@article{Saleem2026A,
author = {Gulshan Saleem and Nisar Ahmed and Muhammad Imran Zaman and Ali Hassan and Umar Mujahid},
title = {A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots},
journal = {ICCK Transactions on Information Security and Cryptography},
year = {2026},
volume = {2},
number = {3},
pages = {192-215},
doi = {10.62762/TISC.2026.712476},
url = {https://www.icck.org/article/abs/TISC.2026.712476},
abstract = {Prompt injection is ranked as the most critical vulnerability in large language model (LLM) deployments by the OWASP Top 10 for LLM Applications, yet existing defenses operate at isolated pipeline stages and remain incomplete. Input filters cannot inspect retrieved documents, while output monitors cannot prevent malicious payloads from reaching the model. Consequently, retrieval-augmented generation (RAG) chatbots remain vulnerable to indirect injection, where a poisoned knowledge-base document compromises every user whose query retrieves it. We present a three-layer framework that intercepts both direct and indirect prompt injection throughout the inference pipeline. Layer 1 screens user input using a rule-based pattern library and a fine-tuned semantic anomaly classifier. Layer 2 enforces a provenance-based instruction hierarchy during context assembly, preventing retrieved content from overriding operator policy. Layer 3 audits model output using a policy rule engine and semantic drift detector before delivery. A continuous audit loop aggregates structured logs and supports retraining to adapt the classifier to emerging attack patterns. The framework is target-model agnostic: it treats the protected LLM as a black box and deploys as middleware without modifying it, while relying on lightweight auxiliary models (a MiniLM sentence encoder and Llama Guard~2) for detection. Evaluation on 5,080 samples across GPT-4o, Llama 3, and Mistral 7B - restricted to single-turn and non-adaptive attacks - shows that the framework reduces Attack Success Rate (ASR) from 71.4\% to 11.3\%, outperforming the best single-layer baseline by 27.3 percentage points and the evaluated NeMo Guardrails configuration by 23.8 percentage points, while maintaining a 4.8\% false positive rate and a median latency overhead of 61.2 ms. Ablation studies show that all three layers contribute and that the full framework achieves a lower attack success rate than would be expected if the layers acted independently.},
keywords = {prompt injection detection, retrieval-augmented, generation, large language models, chatbot security, large language model guardrails},
issn = {3070-2429},
publisher = {Institute of Central Computation and Knowledge}
}
Article Metrics
Publisher's Note
ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and Permissions
Portico