- what it reads
userandtool_resultsegments, by default- the detectors
- a
classifycheckpoint, or anllmjudge on a prose policy - the verdict
- the whole segment, at threshold 0.5
- where it starts
mode: shadow, enforcing nothing
An assistant follows instructions, and the instructions it follows are assembled at runtime out of a system prompt, a conversation and whatever its tools returned. An attacker who can write into any of those has written into the prompt. A guard that reads only the user turn misses the one an outsider can reach.
Three classes, and who writes each
The attacker is whoever can put text in front of the model. For an assistant with tools that includes people the user has never met: the author of a web page, a document in a shared drive, the sender of an email the assistant was asked to summarise. MITRE ATLAS gives the classes their names.
| class | ATLAS | who writes it | where it arrives | what it asks |
|---|---|---|---|---|
| Direct injection | AML.T0051.000 | the user | the user turn | the assistant to drop its instructions and follow new ones |
| Indirect injection | AML.T0051.001 | a third party | a page, a file or an email a tool fetched | the same, from inside content the user never wrote |
| Jailbreak | AML.T0054 | the user | the user turn | the model to answer what its training refuses |
An instruction written into a document your agent fetched is inspected on the same pass as the words the user typed.
Three classes, kept apart
A jailbreak and an injection are different classes with different victims. An injection changes what the assistant does for somebody else. A jailbreak changes what the model will say to the person asking. A harmful request in plain words is a third thing, and a classifier trained on one of the three fires on the others.
The evaluation set Gate is building records a class on every row (injection, harmful or benign), so a classifier's fire on a harmful request counts against its precision.
What it reads
Left unset, a classify or llm detector reads apply_to: [user, tool_result]: the two kinds of segment an outsider writes. Tool results are where indirect injection arrives, so they are in the default. Name the kinds to change it, and add output to judge what the model says back.
A segment longer than the model's window is scored window by window, overlapping by an eighth, and the best score any window gave a label is kept. That is the published guidance for these checkpoints, and under it cutting text into windows cannot hide anything. The threshold is 0.5 unless a rule says otherwise.
A finding is a verdict over the whole segment, recorded with kirto.gate.span: segment. It says the segment carries an injection, and it does not say where.
A rule acts on the verdict with deny or annotate. mode: shadow runs the detector, writes the verdict to the audit trail and enforces nothing. A new detector starts there: the false-positive rate is read off the trail before a deny rule is written.
Classifier or judge
Two detectors cover this class, and they answer different questions at different prices.
classify | llm judge | |
|---|---|---|
| the question | the one the checkpoint was trained on | anything you can write down |
| can be argued with | no, nothing in it follows instructions | yes, it reads what the attacker wrote |
| cost | a forward pass on the CPU, no bill | a model call and its tokens |
| scores | calibrated probabilities | the model's opinion |
| languages | the checkpoint's | the judge model's |
The classifier is a pinned ONNX checkpoint run through the gateway's own tokenizer binding. It catches the phrasings it was trained on and has no instructions to exploit. The judge takes a policy in prose, sends the segment to a model and reads a verdict back. A model that follows instructions can be told what to answer, so a judge's verdict on text an attacker wrote is a verdict the attacker had a hand in.
Use the classifier where a trained checkpoint exists, and the judge for policy in prose. Run both when the cost allows, because they fail differently, and read them with the trail open. The how-tos are Stop prompt injection with a classifier and Judge with a model.
Nine ways to phrase an override
Injections fall into nine pattern families, sorted by phrasing and not by what they ask for. A guard measured on one family says nothing about the others, so the set Gate is building carries all nine. The field guide has a fresh example of each.
| family | the shape |
|---|---|
override | an instruction to ignore, forget or replace what came before |
extract | a request to reveal the system prompt, the tools or the policy |
persona | a new identity with different rules, adopted by decree |
hypothetical | a frame in which the rules are said not to apply |
format | an output shape whose production requires breaking the rules |
encoding | the instruction carried in base64, a cipher or a spelling |
authority | a claim to be the developer, the operator or the system |
embedded | the instruction inside a document the assistant is asked to process |
split | the attack spread over turns, or over two languages in one turn |
Precision and recall on three public sets
These are precision and recall at the default threshold on three public English sets, taken through the gateway's own inspection path, for two checkpoints Gate pins.
| deepset | jailbreak-classification | safe-guard | |
|---|---|---|---|
| ProtectAI DeBERTa v2 (fp32) | 1.00 / 0.37 | 0.98 / 0.86 | 1.00 / 0.85 |
| Prompt Guard 2 86M (int8) | 1.00 / 0.05 | 1.00 / 0.84 | 1.00 / 0.37 |
Neither produced a false positive on those sets, and a deny rule needs that. The recall says what each was trained to call an injection. Prompt Guard 2 is built for explicit jailbreak techniques and finds those. ProtectAI's also catches the instruction-style attacks the other sets are full of.
Those are English numbers. On ordinary Arabic customer turns the ProtectAI checkpoint fires on 65 of 81, which is an outage on a deny rule. The language page says why, and which checkpoint to pin instead. Both rows were taken through the gateway, off no model card, and a third checkpoint is a commit and a rebuild away: choosing a detector.
Threat model
The asset is the set of instructions the assistant follows. Anyone who can put text where the model will read it can try to replace them. Every control on this page lowers the chance of that, and none removes it. The entry this section expands sits with the others in threats and limits.
- Tool results are inspected by default. A
classifyorllmdetector readsapply_to: [user, tool_result]when it is unset, because indirect injection arrives in what a tool returned. - Windowing cannot hide anything. A segment longer than the model's window is scored window by window, overlapping by an eighth, and the best score any window gave a label is the one kept.
- An uninspected body is counted, never assumed clean. A body over
max_body_bytes, one that does not parse, and a content type that is neither JSON nor SSE each incrementkirto_gate_guardrail_uninspected_total. Alert on it. - A new detector starts in shadow.
mode: shadowruns it, writes the verdict to the trail and enforces nothing, so the false-positive rate comes from your own traffic before a deny rule is written. - Two detectors that fail differently. The classifier has no instructions to exploit and catches the phrasings it was trained on. The judge takes a policy in prose and answers anything you can write down.
- The numbers are published with their sets. Precision and recall at the default threshold, on three public English sets, through the gateway's own inspection path, for both pinned checkpoints.
Where a verdict stops
- It reduces the risk and does not prevent it. The classifier catches the phrasings it was trained on and false-positives on prose about security. A judge reads exactly the text the attacker wrote, and a model that follows instructions can be told what to answer.
- It does not locate. The verdict covers the segment, so a rule can deny or annotate that segment and cannot redact the sentence that caused it.
- Arabic recall is unmeasured beyond eight turns. No public Arabic prompt-injection set exists, so the false-positive half is measured and the recall half is not.
The posts behind this
- The tool result is the channel. Prompt injection reaches an agent through anything it reads, from an attacker who is not the user. Direct injection comes from the user turn, indirect injection from a tool result: mail, a document, a web page. Gate reads both.
- Nine ways to phrase an override. An injection is written in one of nine pattern families, and a classifier trained on some of them misses the others. A field guide, with one fresh example of each.
- Injection, jailbreak, harmful. Injection, jailbreak and harmful requests are three classes of text, and a control built for one does little against the others. A classifier trained on one class fires on another, and one MultiJail prompt shows how.
- Classifier or judge. A classifier and a judge fail at different things, so the choice depends on what the policy can be written as. A classifier learns one question from a training set. A judge reads the policy in prose and decides on exactly what the attacker sent.
Start with Stop prompt injection with a classifier, then run the detector in shadow mode before it enforces.