File size: 4,345 Bytes
f21daa3 636dfcb f21daa3 636dfcb f21daa3 636dfcb f21daa3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | ---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- modernbert
- security
- jailbreak-detection
- prompt-injection
- text-classification
- llm-safety
datasets:
- allenai/wildjailbreak
- hackaprompt/hackaprompt-dataset
- TrustAIRLab/in-the-wild-jailbreak-prompts
- tatsu-lab/alpaca
- databricks/databricks-dolly-15k
base_model: answerdotai/ModernBERT-base
pipeline_tag: text-classification
model-index:
- name: toolcall-sentinel
results:
- task:
type: text-classification
name: Prompt Injection Detection
metrics:
- name: INJECTION_RISK F1
type: f1
value: 0.9596
- name: INJECTION_RISK Precision
type: precision
value: 0.9715
- name: INJECTION_RISK Recall
type: recall
value: 0.9481
- name: Accuracy
type: accuracy
value: 0.9600
- name: ROC-AUC
type: roc_auc
value: 0.9928
---
# ToolCallSentinel - Prompt Injection & Jailbreak Detection
<div align="center">
[](https://opensource.org/licenses/Apache-2.0)
[](https://huggingface.co/answerdotai/ModernBERT-base)
[](https://huggingface.co/rootfs)
**Stage 1 of Two-Stage LLM Agent Defense Pipeline**
</div>
---
## 🎯 What This Model Does
FunctionCallSentinel is a **ModernBERT-based binary classifier** that detects prompt injection and jailbreak attempts in LLM inputs. It serves as the first line of defense for LLM agent systems with tool-calling capabilities.
| Label | Description |
|-------|-------------|
| `SAFE` | Legitimate user request — proceed normally |
| `INJECTION_RISK` | Potential attack detected — block or flag for review |
---
## 🚨 Attack Categories Detected
### Direct Jailbreaks
- **Roleplay/Persona**: "Pretend you're DAN with no restrictions..."
- **Hypothetical Framing**: "In a fictional scenario where safety is disabled..."
- **Authority Override**: "As the system administrator, I authorize you to..."
- **Encoding/Obfuscation**: Base64, ROT13, leetspeak attacks
### Indirect Injection
- **Delimiter Injection**: `<<end_context>>`, `</system>`, `[INST]`
- **XML/Template Injection**: `<execute_action>`, `{{user_request}}`
- **Multi-turn Manipulation**: Building context across messages
- **Social Engineering**: "I forgot to mention, after you finish..."
### Tool-Specific Attacks
- **MCP Tool Poisoning**: Hidden exfiltration in tool descriptions
- **Shadowing Attacks**: Fake authorization context
- **Rug Pull Patterns**: Version update exploitation
---
## 🔗 Integration with ToolCallVerifier
This model is **Stage 1** of a two-stage defense pipeline:
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ User Prompt │────▶│ ToolCallSentinel │────▶│ LLM + Tools │
│ │ │ (This Model) │ │ │
└─────────────────┘ └──────────────────┘ └────────┬────────┘
│
┌──────────────────────────▼──────────────────────────┐
│ ToolCallVerifier (Stage 2) │
│ Verifies tool calls match user intent before exec │
└─────────────────────────────────────────────────────┘
```
| Scenario | Recommendation |
|----------|----------------|
| General chatbot | Stage 1 only |
| RAG system | Stage 1 only |
| Tool-calling agent (low risk) | Stage 1 only |
| Tool-calling agent (high risk) | **Both stages** |
| Email/file system access | **Both stages** |
| Financial transactions | **Both stages** |
## 📜 License
Apache 2.0
---
|