Abstract
<title>Abstract</title> <p>The use of Large Language Models (LLMs) and Generative Artificial Intelligence (GenAI) is widespread across various applications like healthcare, educational technologies, finance, cybersecurity, and intelligent virtual assistants. Unfortunately, these models are susceptible to prompt-injection and jailbreak attacks that exploit their existing protection systems to produce unsafe, unintended, or misleading outputs. Currently, the methods for detecting prompt attacks are limited to particular benchmark datasets, which makes them unable to adapt to new attack situations in practice. In this context, this paper presents an Adaptive Multi-Dataset Framework (AMDF) for the detection of prompt-based and jailbreak attacks in Large Language Models. The proposed framework unites three benchmark datasets, namely, JailbreakBench, PKU-SafeRLHF, and Anthropic HH-RLHF for the broader coverage of both harmful and non-dangerous prompts. After preprocessing the datasets, Sentence-BERT is used for semantic embedding generation, and the resulting semantic embeddings are classified using XGBoost for the identification of malicious prompts before they reach the target LLMThe suggested framework is assessed by using general performance measures like accuracy, precision, recall, and F1-score. The results indicate that AMDF has an overall accuracy of 84%, including high precision, recall, and F1-score, thus proving its capability to differentiate between malicious and benign prompts. By taking advantage of multiple heterogeneous datasets, the proposed framework helps significantly improve the generalization in the context of the evolving prompt injection and jailbreak attacks while ensuring that a practical, scalable, and adaptable defense mechanism is provided.</p>