--- title: "The gate is your hallucination detector" description: "GPT-4 scored zero hallucinations on a major benchmark. Re-annotation found 83. Here's how operators catch confident errors without an ML team: the review gate." image: "https://storage.googleapis.com/promptmetrics-uploads/website/posts/1786011926549-563389471.png" author: "Yash Raval" category: "Agent Trust & Guardrails" publishedAt: "2026-01-29T15:26:47.389Z" updatedAt: "2026-08-06T10:25:26.988Z" canonical: "https://www.promptmetrics.dev/blog/llm-hallucination-detection-benchmarks" --- # The gate is your hallucination detector A coding agent I was testing drafted a field update into a CRM record for Acme Corp. Renewal date, contract value, a note about a call that never happened. It was confident. It was formatted perfectly. It was wrong. Nobody would have caught it for weeks. Not because the team was careless, but because the workflow wrote straight to the system of record. There was no moment where a person saw the change before it landed. That moment is the whole subject of this post. > **The short version** > > * You can't stop a model from being confidently wrong. Re-annotation of the RAGTruth benchmark found 10x more hallucinations than the original labels caught, and GPT-4 went from zero to 83 flagged responses ([Blue Guardrails, RAGTruth++](https://blueguardrails.com/en/blog/ragtruth-plus-plus-enhanced-hallucination-detection-benchmark), 2025). > > * Detection models exist. They're built for teams running 100,000 requests a day, with ML engineers and GPU budgets. > > * At your scale, the cheapest and most defensible detector is a review gate: the agent drafts, a person approves, the approval gets logged. This post shows you how the big-budget options work, why they don't fit, and what to build instead. > ## How often is your AI actually wrong? More often than the benchmark said. In November 2025, the team at Blue Guardrails re-annotated a 408-response subset of RAGTruth, the standard benchmark for hallucinations in retrieval-augmented answers. Annotated hallucinations went from 86 to 865. Tenfold. The overall hallucination rate in that subset jumped from 15.9% to 74.75% ([Blue Guardrails, RAGTruth++](https://blueguardrails.com/en/blog/ragtruth-plus-plus-enhanced-hallucination-detection-benchmark), published 20 Nov 2025, fetched 2026-08-06). The detail that should bother you: GPT-4 scored zero annotated hallucinations on the original benchmark. After re-annotation, 83. The model didn't get worse. The benchmark had been missing the errors the whole time. One caveat, because I'd want it if I were you: RAGTruth++ is a single vendor's re-annotation, not a peer-reviewed study. The original [RAGTruth corpus](https://arxiv.org/abs/2401.00396) (Niu et al., 2024) is the academic baseline it extends. Treat the exact figures as one team's count and the direction as the finding. > If you've been burned by a vendor whose demo was flawless and whose production build wasn't, this is the mechanism. The demo wasn't lying. It was measured the way the original benchmark was measured: by people who didn't look hard enough at the confident answers. ![](https://storage.googleapis.com/promptmetrics-uploads/website/content-images/1786011784184-481729089.png) ## What do the big-budget teams do about it? They buy or build detection layers. Four strategies dominate the field right now. The clearest published comparison comes from the December 2025 [HaluGate release from the vLLM project](https://blog.vllm.ai/2025/12/14/halugate.html) (published 14 Dec 2025, fetched 2026-08-06), which benchmarked its own latency against the judge approaches. | Strategy | What it is | Added latency (vLLM's comparison) | What it needs | | --- | --- | --- | --- | | LLM-as-Judge (frontier model) | A second big model audits every response | 2–5 seconds | Double the inference: every audited response is a second API call to a model of the same class | | Local LLM judge | Same idea, on your own hardware | 500ms–2s | A GPU and someone who runs it | | HaluGate | Two-stage: a fast prompt classifier (Sentinel), then token-level detection with an NLI explainer that labels each claim CONTRADICTION, NEUTRAL, or ENTAILMENT | 76–162ms | A vLLM serving stack you operate yourself | | Token probability | Read the model's own confidence scores | Near zero | White-box access to model internals, which API-only tools don't give you | Two honest details from vLLM's own write-up. The speed is real: 76 to 162 milliseconds against seconds for a judge model. And detection is hard even for the specialists: their token-level detector alone reaches only 59% F1 on hallucinated content before the explainer stage filters the results. Impressive engineering. Real trade-offs. And one assumption underneath all of it matters more than any benchmark: production request volume, and a team to run the stack. You run 200 requests a day. Maybe 500 on a heavy day, if your agent is drafting CRM updates, renewal emails, and meeting summaries across a 12-person team. Audit every response with a second frontier model and you've simply bought the same inference twice, plus 2 to 5 seconds of waiting. The self-hosted options are cheap to run and expensive to own: the cost isn't the GPU, it's the person who patches it. ## Why doesn't any of this work at your scale? Because every row in that table assumes someone on staff who can run it. Token probability needs white-box model access. Local judges need a GPU and a person who owns it. Even HaluGate, the fast one, assumes you operate a vLLM serving stack. You run HubSpot and Salesforce. Different world. The trap I want to name, because it's the most common misconception I hear on calls: treating hallucination detection as a technical product you buy. It isn't. At operator volume, it's a workflow design question. Gartner [predicted in July 2024](https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025) that 30% of generative AI projects would be abandoned after proof of concept by end of 2025; Gartner's follow-up analysis in January 2026 found [at least 50% were actually abandoned](https://www.gartner.com/en/articles/genai-project-failure), and the causes it names are unclear business value, unready data, escalating cost, and poor change management. Not model quality. Not once. I wrote about that shape problem in [it's not the tools, it's the shape](/blog/it-s-not-the-tools-it-s-the-shape). At 200 requests a day, a person can read the outputs that matter. The question isn't how to detect errors at scale. It's whether your workflow puts the draft in front of a person before it acts. Most don't. That's the actual gap. ## What is the gate, mechanically? The gate is draft-then-review, enforced by the workflow instead of promised in a policy. The agent proposes. A person approves. Nothing writes to your CRM, your inbox, or your invoicing tool until the approval happens. And the approval is logged: what was proposed, what the reviewer saw, who said yes, when. Not an approval button bolted onto a black box. That version is theater, and the market has a name for what it produces. IBM calls it "liability laundering": accountability redirected from system design to whoever clicked approve ([IBM Think, "Why 'human in the loop' alone is not a governance strategy"](https://www.ibm.com/think/insights/liability-laundering-problem-human-in-the-loop-not-governance-strategy), Boinodiris & Mackenzie, 17 Jun 2026). The difference is what the reviewer sees. A real gate shows the concrete thing: * **The proposed action.** Not a summary of it. The actual field change, the actual email, the actual payload. * **The evidence.** What the agent looked at to produce the draft. * **The diff.** What the record says now versus what it will say after. * **The rollback path.** How to undo it if the approval was wrong. > Why so strict? Because gates fail when reviewers approve a performance instead of a decision. In February 2024, a finance employee at Arup approved US$25.6 million across 15 transfers after a video call where every colleague on screen, including the CFO, was a deepfake ([CNN](https://edition.cnn.com/2024/05/16/tech/arup-deepfake-scam-loss-hong-kong-intl-hnk), company-confirmed May 2024). The gate existed. It failed because the reviewer couldn't inspect what they were approving. They saw a video call, not a payload. The skill of judging what comes back has a name: discernment, one of the 4 Ds from the AI Fluency framework by Dakan, Feller and Anthropic ([AI Fluency: Framework & Foundations](https://anthropic.skilljar.com/ai-fluency-framework-foundations), CC BY-NC-SA). The gate is the infrastructure that gives discernment somewhere to happen. Skip the infrastructure and the skill has no room to work. I wrote up the implementation side, including where the gate has to live so the agent can't route around it, in [the gate belongs inside the tool](/blog/the-gate-belongs-inside-the-tool-giving-an-ai-agent-write-access-to-a-production-crm). ## What does the audit trail give you that a detector can't? Proof. A detection model gives you a score. An audit trail gives you a replayable record: proposed action, evidence, approval, rollback path, signed decision. When something goes wrong three weeks later, the score tells you nothing. The record tells you exactly what happened and who saw what. In the EU, this stopped being optional in 2026. Article 50 of the AI Act ([Regulation (EU) 2024/1689](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)) sets the transparency duties, and under the Act's own timeline they went live on 2 August 2026. Workflows deployed after that date get no grace period. Penalties for transparency breaches run to €15 million or 3% of global annual turnover, whichever is higher. Your DPO's question was never "what's your F1 score." It's "show me the record." One honest flag, because we hold ourselves to this: the full replayable approval object I'm describing is a designed specification that gets built per engagement. It's not an off-the-shelf product you can install today, from us or from anyone. Anybody who tells you otherwise is selling the approval button. ## Which red flags show your workflow trusts the model too much? Run this against your own setup. Each one is checkable in an afternoon. * **Nobody has measured how often outputs are wrong.** Not a survey. An actual count, on your data, in the last 90 days. * **Outputs write directly to the system of record.** No draft state, no approval step. The agent acts and you find out later. * **The same tool generates and checks the work.** Asking a model to grade itself finds the errors it already didn't notice. * **Flagged outputs go nowhere.** There's a review step, but nobody owns it, and reviewers who raise problems never see them fixed. * **The reviewer sees a summary, not the payload.** If your approval screen says "update 3 fields" without showing the fields, you have the Arup problem. Two or more of these and you don't have a hallucination problem. You have a workflow problem that hallucinations will eventually expose. The short form, if you want to forward it to someone: a workflow trusts its model too much when any of five conditions hold. No measured error rate in 90 days. Direct writes to the system of record. The same tool generating and checking work. Flagged outputs with no owner. Reviewers shown summaries instead of payloads. ## Where it still leaks The gate catches what a reviewer inspects. It doesn't catch what they rubber-stamp. Review fatigue is real: put forty approvals a day in front of one person and by Thursday they're approving from muscle memory. The IBM piece names this exactly, and I've watched it happen. What we do about it: scope the gate to action classes that matter (writes to the system of record, external sends, money), keep read-only and draft work ungated, and show the diff instead of a summary so approving takes judgment instead of faith. What I can't promise you: a distracted reviewer. No architecture fixes attention. Anyone who claims their governance layer removes the need for judgment is selling you the thing this post is warning you about. And the honest limit on the economics: if you genuinely run tens of thousands of requests a day, the table above stops being trivia and becomes your shopping list. The gate still belongs in front of the actions that matter. But at that volume you need detection too, and you should hire for it. ## FAQ ### Can you stop AI hallucinations completely? No. Grounding the model in your own data reduces them, and re-annotation studies show even top models produce confident errors the benchmarks missed. You manage hallucinations the way you manage any error rate: reduce what you can, and gate the actions that matter so one wrong draft can't become a wrong record. ### Do I need a hallucination detection model for my business? At operator volume, almost never. Detection stacks earn their keep at production request volumes, and each option carries a real bill: an LLM-as-judge layer buys the same inference twice, and the fast self-hosted options assume you operate your own model serving. At 200 requests a day, a review gate plus an audit log covers the risk for the cost of designing the workflow properly. ### What's the difference between "human in the loop" and a review gate? The phrase is a policy; the gate is a mechanism. If the workflow can write to your CRM without a person approving the specific payload, you don't have a gate, whatever the vendor slide says. A gate means the approval is enforced by the workflow and each decision is logged with what the reviewer saw. ### Does the EU AI Act require me to catch hallucinations? It doesn't name hallucinations. It requires transparency, and Article 50 has applied since 2 August 2026 with no grace period for newly deployed workflows. What survives an audit is your record of review: what the system proposed, who approved it, and on what evidence. An accuracy claim without a record satisfies nobody. ## The detector you can run today Govern the work, not the model. The model will be confidently wrong sometimes, whatever the leaderboard says, and the benchmark re-annotations prove the leaderboard misses it anyway. What you control is whether a wrong draft can act. The gate plus the audit trail is the detection stack you can run today, with the team you have, at the volume you actually run. If you've got one workflow where a confident error would hurt, that's a conversation we have. You'll talk to Ranya first, and she'll tell you honestly whether we can help. If we can, the [First Skill Sprint](/services#sprint) is the smallest real thing that proves the point: one workflow, one governed skill, the gate designed before anything runs, fixed scope, two weeks. If a plain automation script is all you need, she'll say that too. Or just come compare notes in [Operator Stack](/community). No pitch. Operators in the same moment you're in. Tell me where I'm wrong. Or tell me what your reviewer rubber-stamped last week.