- what is pinned
- two injection checkpoints that fail in opposite directions
- what picks one
profiles[].detectors, and the rule's ownrequires- the trade
- 80.2% to 1.2% of Arabic customers flagged, at 0.914 to 0.815 English F1
- what is not claimed
- any language outside the ten, and Arabic recall past eight turns
A guard's published precision is an English number, and outside English it fires on ordinary customers. The checkpoint that fixes that gives up English quality to do it, so a deployment serving both languages has no single right answer. Gate pins both checkpoints in one ensemble and lets a profile choose, and it publishes the measurement behind the choice along with the languages it has not run.
Ten languages have been measured through the gateway's own inspection path. All ten carry the same 315 red-team prompts, so the verdicts compare row for row: English, Chinese, Italian, Vietnamese, Korean, Arabic, Thai, Javanese, Swahili and Bengali. English and Arabic also carry a set of ordinary customer turns, because Arabic is the language the team can author and review gold in.
Ten is not your deployment. The command that produced every number below takes a file of your own labelled turns, so the answer for a language nobody here speaks is a measurement you run. How a detector is measured.
Two checkpoints, failing in opposite directions
Gate pins both. protectai/deberta-v3-base-prompt-injection-v2 is the better English guard and one of the most widely deployed open ones. proventra/mdeberta-v3-base-prompt-injection is the same architecture family trained on the same English sources, with a multilingual vocabulary in place of an English one. The four rows below decide between them.
| ProtectAI v2 | mDeBERTa | |
|---|---|---|
| what it costs to serve Arabic | 65 of 81 ordinary turns flagged | 1 of 81 |
| what it costs to serve English | 1 of 81 ordinary turns flagged | 1 of 81 |
| Arabic jailbreak turns caught | 8 of 8 | 6 of 8 |
| English quality, safe-guard F1 | 0.914 | 0.815 |
Picked correctly, your Arabic traffic is flagged at the rate your English traffic is: 1 turn in 81, against 65 in 81 on the English-vocabulary guard.
Neither column is the one to ship everywhere. Serving Arabic on the left column refuses four in five ordinary customers, which on a deny rule is an outage. Serving English on the right gives up a tenth of the F1 on the one English set neither model trained on, and two of the eight Arabic jailbreak turns with it. The choice is a property of the traffic a team carries.
Which one inspects which traffic
Both are declared as detectors in one ensemble, and profiles[].detectors narrows which of them inspect a profile's traffic. A profile selects and does not define. There is one set of detector instances behind the deployment, so two profiles that both select the multilingual guard are inspected by the same one and share its delta-scan cache. The Arabic-serving team gets one checkpoint and the English-only team keeps the other, on one deployment, and neither pays for the other's trade.
A rule then says which findings it acts on. Every annotation carries the name of the detector that produced it, so a CEL condition can deny on one checkpoint's verdict and annotate on the other's while both run. A rule also names the detectors its decision depends on, in requires. A profile that drops a detector a rule requires is refused at load, so a selection cannot leave a deny rule that never matches. Give a team its own policy walks it, and the policy reference lists the annotation fields a condition reads.
Safety is measured in English
A guard's blind spot is a property of its vocabulary, and the vocabulary is a property of the checkpoint you pin. So Gate measures it per checkpoint, and the two tables below are why a deployment holds both.
Without that measurement the choice is a preference between two model cards, both of them reporting English.
Why no single checkpoint wins
MultiJail carries the same 315 red-team prompts in ten languages. The paper sorts them into three tiers by each language's share of the CommonCrawl corpus: above one percent is high-resource, below a tenth of a percent is low. None of the 315 is a prompt injection, so every fire below is a false positive. The fires are protectai/deberta-v3-base-prompt-injection-v2 through the gateway's own inspection path, measured on 31 August 2026 on build 82ccc3176ba3.
| language | MultiJail tier | fires of 315 | rate |
|---|---|---|---|
| English en | high | 0 | 0.0% |
| Chinese zh | high | 19 | 6.0% |
| Italian it | high | 38 | 12.1% |
| Vietnamese vi | high | 138 | 43.8% |
| Korean ko | medium | 63 | 20.0% |
| Arabic ar | medium | 239 | 75.9% |
| Thai th | medium | 294 | 93.3% |
| Javanese jv | low | 129 | 41.0% |
| Swahili sw | low | 141 | 44.8% |
| Bengali bn | low | 310 | 98.4% |
The fires do not follow the tiers. Vietnamese is high-resource and fires on 138. Arabic is medium-resource and fires on 239. Swahili is low-resource and fires on 141. Italian at 38 says the effect reaches Latin scripts. A language's share of a training corpus does not predict what this checkpoint does to it.
So the answer for a language you serve is a measurement. Point kirto-gate guardrails eval at your own labelled turns and read the rate.
The vocabulary predicts it. Arabic costs ProtectAI's tokenizer 3.18 pieces a word against English's 1.52, because 347 of its 128,001 entries contain an Arabic character.
| base | vocabulary | Arabic entries | pieces per Arabic word | per English word |
|---|---|---|---|---|
xlm-roberta-base | 250,002 | 14,639 | 1.76 | 1.41 |
mdeberta-v3-base | 250,102 | 7,408 | 2.03 | 1.60 |
deberta-v3-base-prompt-injection-v2 | 128,001 | 347 | 3.18 | 1.52 |
A sequence of unfamiliar fragments is what the training data called an attack, which is how ordinary Arabic becomes one. Swapping to the multilingual vocabulary takes the Arabic false-positive rate on ordinary customer turns from 80.2% to 1.2%, English's rate to the case. That is the left column of the first table turning into the right one. The report carries the full tables, the pins and the build.
Read it with recall in the other hand
A parallel set exposes this with no gold in any of the ten languages. The same meaning in every column of a row has the same right verdict in each, so a verdict that changes between them is a defect on its own. The rows a language lost from the reference are the bypass an operator would see. The harness reads that off with -invariance, and choosing a detector covers what else it answers.
It cannot tell a quiet detector from a consistent one. Meta's Prompt Guard 2 86M scores perfectly on the table, zero fires across all 3,150 cases, because it is quiet on nearly everything: it caught one of eight Arabic jailbreak turns and five of eight English ones. Invariance says whether a verdict depends on the script. Recall says whether the verdict is ever INJECTION, and that half needs gold in the language.
Threat model
The asset is the deny rule's false-positive rate, and behind it the deployment's ability to serve customers who do not write English. Nobody has to attack it. An ordinary customer asking about an invoice in Arabic is enough to trip it, and on a deny rule that is an outage for four in five of them. The attacker's version of the same defect is the language a guard stays quiet in, which is a bypass, and that half needs gold in the language to measure.
Two mechanisms produce the gap. The MultiJail paper's thesis is about the model: alignment data is English, so a harmful request translated into a language with less of it is more likely to be answered. That is a resource-level effect. The guard's defect is about vocabulary: a tokenizer with 347 Arabic entries turns ordinary Arabic into unfamiliar fragments, and a sequence of unfamiliar fragments is what its training data called an attack. The entry this section expands sits with the others in threats and limits.
- Both checkpoints are pinned, and a profile chooses. The English guard and the multilingual one fail in opposite directions, so the team carrying the traffic picks per profile instead of inheriting one vendor's default.
- Picked correctly, Arabic is flagged at the English rate. 1 turn in 81, against 65 in 81 on the English-vocabulary guard.
- Ten languages measured through the inspection path. All ten carry the same 315 red-team prompts, so the verdicts compare row for row inside one harness.
- The measurement runs on your own turns. The command that produced every number here takes a file of your labelled traffic, so a language nobody here speaks gets a number you took yourself.
- The numbers that argue against us are published too. The multilingual checkpoint gives up a tenth of the F1 on the one English set neither model trained on, and that is on the page beside the rest.
What has not been measured
- No claim about a language it has not run. Ten languages are measured for false positives and most languages are unrun. An unrun language has no number here and should not be given one.
- Arabic recall is unmeasured beyond eight turns. Everything above is about false positives, the half that can be measured without an attack set. There is no public Arabic prompt-injection set.
- The model we want does not exist. Nobody has published a checkpoint with ProtectAI's English quality and a multilingual vocabulary's indifference to script, so the choice stays a trade.
The posts behind this
- Safety is measured in English. Models give unsafe answers more often in low-resource languages, because alignment data is English. The guards built to stop them share the same defect from the same root.
- Read invariance with recall in the other hand. Some metrics reach their best value as a detector fires less, so a detector that answers nothing scores perfectly on them. Invariance is one. So is a false-positive rate, an agreement score, and a benchmark holding no positive cases. Each is read beside the number that breaks the silence.
- An evaluation set in a language with none. No public Arabic prompt-injection set exists, so recall on Arabic attacks is unmeasured. Gate is building one with parallel pairs so the verdict can be compared across languages, hard negatives that outnumber attacks, and an origin tag on every row.