Research
How Kirto measures the models it ships and the ones it rejects, on the languages a deployment actually carries.
Reports
How a report is made
Every number comes from kirto-gate guardrails eval, the same binary and the same code path that enforces. The command takes a gateway config, a file of labelled cases and the types to score. It presents each case as one chat turn through Gate's own inspection and scores what came out against what a person marked, at byte offsets into the text as sent.
The result measures the pipeline, and that is where these numbers part company with the ones beside a model on the Hub. A model ships with a tokenizer binding, a window scheme, a script fold, a decoder and a threshold. Each of those can lose a span the model found or land it on the wrong bytes, and a score computed in a notebook describes a model nobody is running.
Each report names its evaluation sets and their sha256, and the build every model ran on. Where the same sentences exist in several languages, the run is an invariance sweep read against a reference language, English by default. The table says which rows a model keeps, loses and gains against that reference, so a guard that holds in English and fires on everything in Arabic shows the drop instead of an average.
What a report here is held to
- Negative results are published. A model we rejected, a technique that cost more than it returned, a clean-looking score that turned out to mean silence. Leaving those out is how a measurement becomes marketing.
- Reports are re-issued. A blog post describes the day it was written. A report is a measurement, so when a model, a set or the harness moves, it is re-run and carries both dates.
- The run is yours to repeat. The sets are named, the digests are printed and the command is documented, so a report can be disagreed with on its own numbers.
What we do not publish
No corpus text. A report names every evaluation set and its digest, and the mismatched cases stay local, because they are the corpora's text and not ours to redistribute. No ranking of vendors we have not measured on the same build. No claim about a language we have not run, which is why several reports end by naming the languages they could not characterise.
The models we publish are on Hugging Face, and the harness is documented in Measure a detector.