10 August 2026, 06:01 PM
Not All Hallucinations Are the Same Bug, So Stop Testing Them Like They Are
Ask five engineers to define "hallucination" and you'll usually get five overlapping but different answers. Made-up facts. Confident wrong answers. Fabricated citations. Answers that contradict the source material. All of these get lumped under one word, and that's part of why so many hallucination testing efforts underperform: teams build one detection method for a problem that actually has at least four distinct shapes, each with its own root cause and its own test design.
I learned this the slow way, reviewing a client's chatbot logs after a compliance team flagged a handful of bad answers. The team's hallucination testing had been passing for months. When we broke the flagged cases down by type, none of them were the kind of hallucination their tests were built to catch. Their test suite was solid at detecting fabricated facts. It had nothing in place for a model contradicting its own source document, which is exactly what had gone wrong.
The Four Shapes Hallucinations Actually Take
Before you can test for hallucinations, it helps to be precise about what you're testing for. I break them into four categories, and in enterprise LLM applications, I see all four show up, usually in different proportions depending on the use case.
Fabrication. The model states something as fact that doesn't exist anywhere, no source document, no training data basis anyone can point to, nothing. This is the hallucination type most people picture: invented statistics, made-up case law, citations to papers that were never written.
Contradiction. The model has correct source material available and says something that directly conflicts with it. This one is sneakier, because the model isn't inventing information from nothing, it's misreading or misrepresenting information it actually had access to. It often reads as more confident and more polished than a fabrication, which makes it harder to catch by eye.
Extrapolation. The model takes something true and stretches it past what the source material actually supports. A document that says a feature is "available in the enterprise tier" becomes an answer claiming it's "included in all plans." Not invented from nothing, not contradicting the source outright, just pushed further than the evidence goes.
Context blending. In multi-document RAG setups specifically, the model combines accurate information from two separate sources into a single claim that's wrong in the combination, even though each individual fact was correct in isolation. This is a failure mode almost unique to retrieval-augmented systems, and general-purpose hallucination tests built for single-document LLM apps tend to miss it entirely.
Why Lumping These Together Breaks Your Testing Strategy
If you build one hallucination detection method, say, a general fact-checking pass comparing the output against a reference answer, you'll catch fabrications reasonably well. You'll catch some contradictions. You'll likely miss most extrapolations, because the claim isn't false exactly, it's overreaching, and a similarity-based check often scores an overreaching answer as "close enough." You'll almost certainly miss context blending unless your test design specifically accounts for multi-source scenarios.
This matters because the business risk profile is different for each type. A fabrication in a customer support bot is an obvious, embarrassing failure that gets caught fast. An extrapolation in a compliance-adjacent answer, something technically defensible-sounding but overstated, can sit in production for months before anyone notices, precisely because it doesn't read as obviously wrong.
Mistakes I See Enterprise Teams Make Repeatedly
Treating hallucination testing as a launch-week activity. It's tempting to run a hallucination check once before go-live and move on. Model updates, prompt changes, and knowledge base updates can all shift hallucination rates without anyone touching the "hallucination testing" step of the process again. This needs to be a recurring pass, tied to any change upstream, not a one-time gate.
Testing only with questions the system is designed to answer well. The test cases that catch hallucinations are usually the ones sitting outside the system's comfortable zone, ambiguous questions, questions with no good answer in the knowledge base, questions that invite the model to extrapolate. A test set full of easy, well-covered questions will report a hallucination rate that looks great and means very little.
Ignoring extrapolation because it "isn't technically false." This is the mistake I flag most often in reviews. An answer that stretches a true statement past what it actually supports is still a liability, particularly in regulated industries where the gap between "accurate" and "not misleading" is the entire compliance conversation.
No process for what happens after a hallucination is caught. Detection without a feedback loop back into prompt design, retrieval tuning, or guardrail configuration just produces a report nobody acts on. The teams that actually reduce hallucination rates over time treat detected cases as inputs to a fix, not just a metric to track.
Assuming a bigger or newer model solves the problem. Model upgrades sometimes reduce fabrication rates. They rarely fix contradiction, extrapolation, or context blending on their own, because those failure modes are often more about how context is assembled and presented to the model than about the model's raw capability. I've watched teams upgrade models and see their hallucination rate on these three categories stay flat, or occasionally get worse, because a more capable model can construct a more convincing extrapolation.
When to Prioritize This, and When You Can Ease Off
For anything customer-facing, compliance-adjacent, or tied to decisions with real consequences (financial, medical, legal, account-related), hallucination testing across all four categories should be non-negotiable and ongoing, not a nice-to-have.
For internal tools with a small, technically literate user base who understand the system's limitations, and where a wrong answer gets caught and corrected quickly with low downstream cost, you can reasonably scope this down, at least initially. Fabrication and contradiction testing still matter almost everywhere. Extrapolation and blending testing are worth investing in as soon as the system starts being trusted for anything beyond a quick first draft.
A Practical Starting Checklist
- Test cases exist for each of the four hallucination types, not just fabrication
- Source-grounding verification traces individual claims back to retrieved context, where RAG is involved
- At least one test category specifically targets multi-document context blending
- Test set includes ambiguous and out-of-scope questions, not just well-covered ones
- Hallucination testing reruns automatically on model version changes, prompt edits, and knowledge base updates
- Detected hallucinations feed into a documented fix process, not just a dashboard metric
- Human review is used to continually refine the automated test set, not as the sole detection layer
Hallucination rates get reported like a single health metric, and that framing is going to keep causing surprises for teams that treat it that way. The organizations getting this right are the ones breaking hallucination testing down by failure type, building detection methods matched to each one, and treating the whole thing as infrastructure that runs continuously rather than a box checked before launch.
At PrimeQA Solutions, this categorized approach is central to how we structure LLM Testing engagements, because a single pass/fail hallucination score has never once told our clients which of the four problems they were actually shipping.
The uncomfortable truth is that hallucination testing done properly never really finishes. Every model update, every new document added to a knowledge base, every new prompt variant reopens the question. Teams that accept that early build the monitoring habits to catch it. Teams that don't usually find out from a customer, or a compliance review, instead.