AI for operators3 min read
Benchmarks you can actually run
Most AI security numbers are unfalsifiable. Here's the standard I hold my own project to, and why it made the numbers worse.
I maintain an open-source prompt-injection detector. When I started writing its README, I had to decide what kind of numbers to put in it — and that turned out to be the most consequential design decision in the project.
The genre problem
Read enough AI security marketing and you notice the numbers all look the same. Ninety-something percent detection. Sub-one-percent false positives. No test set you can download, no baseline to compare against, no description of what counts as an attack.
These aren't lies, exactly. They're unfalsifiable. If I don't know what you tested on, "97% detection" carries no information — I can hit 100% detection on a set I chose myself, by choosing a set my detector happens to catch.
The tell is always the missing denominator.
What I made myself publish
The rule I settled on: every number in the README has to be reproducible by a stranger with one command. That meant committing to a few things that were uncomfortable.
The test set ships with the code. Including the attacks that get through. It's tempting to quietly drop the cases you fail, and the only defence against that temptation is making the set public and versioned, so removals show up in the diff.
Every number has a baseline. Detection rate alone is meaningless without the false-positive rate on benign input, and both are meaningless without a naive baseline to compare against. If a regex on the word "ignore" gets you 60%, then your 85% is worth a lot less than it sounds.
Multilingual results are reported separately. Aggregating them hides the failure. A detector that scores 95% in English and 40% in Japanese reports as "87% overall" if your test set is mostly English, and that average describes nobody's actual situation.
It made the numbers worse
Predictably. The honest headline number was lower than the one I could have published, because the honest test set included the hard cases and the honest false-positive rate counted the benign prompts I was flagging.
What I got in exchange was that the numbers became usable. Someone evaluating the library can run the suite against their own traffic and get a number that means the same thing as mine. Two people can now disagree productively, which they cannot do about a marketing claim.
It also made the project easier to improve. When your benchmark is honest, a regression is visible. When it's curated, every change looks like progress.
The kitchen version
Food cost percentage is the number every restaurant reports and almost every restaurant fudges — usually not deliberately, just by leaving things out. Waste. Staff meal. The comps. The walk-ins. Each omission is defensible on its own and the aggregate is fiction.
The chefs who actually ran profitable kitchens were the ones who counted the embarrassing numbers. Not because they enjoyed it. Because a food cost you've flattered tells you nothing about what to fix on Monday.
If you're publishing numbers
Three questions, and if you can't answer all three, the number isn't ready:
- Can someone else run this and get the same result?
- What's the denominator, and who chose it?
- What does a dumb baseline score on the same test?
That's the whole standard. It's not sophisticated. It's just not very common.