Introducing Verdict: an AI detector that publishes its own error rate

On a frozen, leak-free test split, Verdict scores 0.91 accuracy (0.97 AUROC) at separating AI-generated text from human writing. That's measured on documents the model never saw while training, and it's the first thing we want you to know , because most detectors lead with a demo and quietly bury the error rate. We think the error rate is the product.

So here is our whole position in one sentence: a detector produces an AI-likelihood score, not a verdict on a person. Used that way, as a reason to look closer at a document, not as a confession, it's genuinely useful to editors, reviewers, educators, publishers, and anyone screening a large pile of writing. Used as proof, it does real harm. This post is mostly numbers, and specifically the numbers most tools won't print.

Accuracy is the easy half

Accuracy on a balanced set tells you how often the model is right when AI and human text show up in equal measure. That's a fair sanity check, but it hides the question that actually matters in the real world, where the overwhelming majority of text you screen is human: at a threshold where we almost never flag a person by mistake, how much AI writing do we still catch?

That framing, recall at a fixed, low false-positive rate, is the one we hold ourselves to, and it's why every catch rate below arrives with its false-positive number attached.

On modern GPT-4o, Claude, and GPT-5-class text, Verdict catches 86% of AI writing at a 1% false-positive rate: one wrong flag for every hundred human documents. We report the pair deliberately. An 86% catch rate means nothing without the 1% next to it, and any detector that quotes you a recall without its false-positive rate is showing you half a number.

Academic writing is one of the use cases people ask us about most, though it is far from the only one. There, the model now flags about 84% of AI-generated text at a threshold that wrongly flags only 0.4% of genuine papers , roughly 1 in 250, after a recent fine-tune of its neural expert. We lead with that 0.4% on purpose: in a setting where a false flag lands on a real author's reputation, the false-positive rate isn't a footnote, it's the whole ethical budget.

The part that surprised us

We assumed, like most people, that the easy way to beat a detector is to rephrase the text, run it through a paraphraser, lower the register, restructure the sentences. So we built that attack and pointed it at our own model.

It mostly didn't work. Against single, meaning-preserving rewrites, Verdict held about 94% of its detections. The intuition we walked away with: to make AI text read as genuinely human, you have to change what it says, not just how it says it, and by the time you've done that, you've written it yourself. Rephrasing shuffles words around; it doesn't remove the statistical fingerprint underneath.

That's the one finding we'll wave a flag about. Everything else here is us trying hard not to oversell.

Where we're weak

A detector that only publishes its strengths is marketing. Here is the honest frontier.

• Non-native English is a known confound. Text written by fluent non-native

speakers can carry surface patterns, simpler constructions, repeated phrasings , that overlap with what models produce. We treat an elevated score on such writing as a reason for more care, not less, and we're actively measuring this bias rather than waving it away.

• Base rates dominate. If almost none of the text you screen is actually

AI-generated, even a 1% false-positive rate can mean most of your flags are false alarms. A score is only interpretable alongside how much AI text you'd plausibly expect in the pile to begin with.

• Short snippets are unreliable. The fingerprint we detect is statistical, and

statistics need length. A tweet, a lone paragraph, a single email, these carry too little signal to score with confidence, and we'd rather tell you that than return a confident-looking number built on noise.

• A score is a signal, never a person's verdict. Nothing Verdict outputs should be

read as evidence that a specific human did or didn't use AI. It flags text for review. What happens next is a human conversation, not an automated accusation.

Why it's free

We're building a public track record before anything else, so the core detector is free to use. The model is tuned on output from the newest systems, GPT-5 Sol and Claude Fable, and retrained continuously as new models ship, so it keeps pace with how current AI actually writes. Every number here is measured on data the model never saw in training, every catch-rate comes with its false-positive rate, and a Verdict score is a signal to look closer, never proof. If we ever get one wrong, we'll say so here.

It's also why we publish the benchmark behind every number above. You shouldn't take our accuracy claim on faith any more than you should take a detector's flag on faith. Read the method, check the splits, go find the places we're weak. If we ever get a number wrong, we'll say so.

Try the tool, or read the full benchmark, at Verdict.