The surrogate gap: why a tiny model spots ChatGPT but misses everything else

Some AI-text detectors need no training data at all. You take a small language model, show it a passage, and ask how surprised it is by each word. Human writing tends to run a little rougher and less predictable; machine writing runs a little smoother. Score the text on that smoothness and you have a detector, no labels, no training, just a surrogate model you already had lying around. These "zero-shot" methods, log-rank, DetectGPT, and their relatives, are cheap, transparent, and genuinely clever. On the right test, they look almost perfect.

On clean ChatGPT text, a standard zero-shot method (log-rank) scores about 0.99 AUROC. AUROC is a ranking score, not a grade: it measures how often the detector ranks a random AI document as more suspicious than a random human one, where 1.0 is flawless and 0.5 is a coin flip. So 0.99 says the method almost never gets the ordering wrong. It is exactly the kind of number that ends up on a landing page.

Then you change the generator, and the floor gives way.

One detector, two verdicts

The same method, unchanged, scores about 0.59 AUROC on MAGE, a benchmark that pools text from many different models, and about 0.55 across our own 30-generator benchmark. That is barely better than flipping a coin. We did not touch the detector. We changed only which machines wrote the "AI" half of the test set, and a 0.99 became a 0.55.

This is the surprising part, and it is worth sitting with: the demo number and the collapse are the same detector. Nothing separates a 0.99 result from a 0.55 result except the company it was tested against. Which means a zero-shot detection score, reported on its own, tells you almost nothing about how the detector behaves in the wild. It tells you which generator someone chose to test.

Why it collapses

A zero-shot detector never actually learns what "machine-written" looks like. It learns what its surrogate model finds predictable. When the text you are testing came from a model that writes the way the surrogate expects, as early ChatGPT did, smoothly and cautiously, the two line up and the score soars. When the text came from a different family of models, with different quirks and rhythms, the surrogate has no strong opinion, and the signal washes out.

So the method isn't really measuring "human versus machine." It is measuring "close to my surrogate versus far from it." On a single, familiar generator that distinction happens to coincide with the truth. Across thirty generators it does not, and the gap between those two situations is what we mean by the surrogate gap.

The false-positive side makes the cost concrete. Because AUROC sweeps across every possible threshold, a score near 0.55 is not merely a mediocre grade, it means there is no cutoff at which you catch AI text without flagging human text at nearly the same rate. There is no safe operating point to choose. A detector at 0.55 cannot be tuned into a trustworthy one; the separation simply isn't there.

What a trained detector does instead

Our own detector is a mixture of experts trained across that same diverse corpus, and on the held-out portion, generators and documents it never saw in training, it scores about 0.97 AUROC. The difference is not a cleverer idea than log-rank. It is coverage. Because the model has seen many generators, it learns the features that AI text tends to share across families rather than the fingerprint of one surrogate, and those features mostly survive the jump to unseen generators.

That 0.97 also buys something the 0.55 could not: a usable threshold exists. When a ranking score is that high, you can pick an operating point that catches most AI text while keeping false accusations rare, which, for a tool that people might point at real writing, is the only version of "accurate" that counts. A headline detection number is only as good as the false-positive rate hiding behind it.

Zero-shot still earns its keep

None of this makes zero-shot detection a bad idea, and the point isn't to dunk on it. These methods have real advantages we don't: they need no labeled data, they adapt to a brand-new generator the day it ships without anyone collecting examples, and they are far easier to inspect than a trained model. As a research baseline, or against a known generator that resembles the surrogate, they are a reasonable tool.

The failure here isn't the method. It's the evaluation. Reporting the 0.99 without the 0.55 turns an honest, narrow result into a misleading broad one. A zero-shot detector that scores 0.99 on ChatGPT has told you it can spot ChatGPT, nothing more. The only way to know whether a detector works on the text you actually care about is to test it on that text, across the generators you will actually meet, and to publish the number where it breaks.

That is the whole reason we train, and the whole reason we test the way we do. A single impressive number is not a detector working. It is a detector being asked an easy question.

See how every method scores across all thirty generators, and try the detector yourself, at Verdict. Next up: we tried nine different ways to disguise AI text and ran them against our own model.