23 October 2026, 06:22 PM
AI Testing Services: A Complete Guide to Testing AI Applications
The assistant told a customer that refunds were available for 60 days. The policy had said 30 for eight months.
Every automated test was green. The API returned 200. The response was well formed, polite, fast, and cited a document. The document was real, too. It was an archived policy page that nobody had removed from the knowledge base, and the retrieval step had found it, ranked it first and handed it to the model, which did exactly what it was asked to do.
I'm describing a composite of a few cases I've seen, but I'd bet you could finish the story yourself. The code wasn't broken. The model wasn't "hallucinating" in the dramatic sense. The failure lived in the data, passed through retrieval, and came out looking like a confident answer. Nothing in a conventional test suite was positioned to notice.
That's the core difficulty of testing AI applications. The behavior doesn't live in the code alone. It lives in the code, the prompt, the model, the retrieved context, the tools it can call, the data underneath all of that, and the way users actually phrase things. Test one layer and you've tested a fraction of the system.
So this guide is organized around that stack. I'll show where each layer fails, how to test it, and where specialist help is worth paying for.
A six layer map of an AI application, from guardrails down to data sources and runtime, with the typical failure and the test approach for each layer
Three Assumptions That Quietly Break
Classic testing rests on a few assumptions so basic we rarely say them out loud. AI applications violate all three.
The same input gives the same output. Not with sampling, not across model versions, and not when the retrieved context changes. You can reduce variation, but you can't assume it away.
A specification defines the correct answer. For a summary, a support reply or a recommendation, there isn't one correct string. There's a range of acceptable answers and a range of unacceptable ones, and the boundary is a judgment.
A test passes or fails. An AI test is more often a rate. "Gets the refund policy right in 9 of 10 runs" is a different kind of statement from "returns 30."
Once you accept that, the work changes shape. You stop writing assertions about exact outputs and start building evaluation sets, scoring them, and tracking rates over time.
The Evaluation Set Is the Real Asset
If you take one idea from this guide, take this one. The most valuable testing artifact for an AI application isn't a script. It's a curated set of realistic cases with agreed expectations, kept under version control and run against every change.
Where the cases come from. Real questions from support tickets, search logs and user interviews, with personal data removed. Add the awkward ones: ambiguous phrasing, typos, questions the system should decline, questions in a different language, and questions about things the documentation doesn't cover. Add adversarial cases too. Real users are a much better source than a brainstorm in a meeting room.
How big. Start with fifty good cases rather than five thousand mediocre ones. Grow it every time production teaches you something. Hold out a portion that nobody tunes against, so you can tell improvement from memorization.
What "expected" means. For some cases it's a required fact, such as "mentions 30 days." For others it's a forbidden claim, a required refusal, a format, or a rubric a human can apply consistently. Write the expectation down before you see the output, because it's too easy to rationalize afterward.
A Small Harness You Can Start With
Here's a minimal pytest example. It reads cases from a file, runs each one several times because outputs vary, and applies simple rule checks plus a citation requirement. The names are placeholders for your own application entry point.
import json
import pytest
from assistant import ask # your application entry point
RUNS = 10 # repeat each case: outputs vary
PASS_RATE = 0.90 # agreed threshold per case
def load_cases(path="cases.jsonl"):
with open(path) as f:
return [json.loads(line) for line in f]
def check(case, answer):
text = answer["text"].lower()
ok = all(s.lower() in text for s in case["must_include"])
ok &= not any(s.lower() in text for s in case["must_not_include"])
if case["needs_citation"]:
ok &= len(answer["sources"]) > 0
return ok
@pytest.mark.parametrize("case", load_cases(), ids=lambda c: c["id"])
def test_case(case):
passes = sum(check(case, ask(case["question"])) for _ in range(RUNS))
assert passes / RUNS >= PASS_RATE, f"{case['id']}: {passes}/{RUNS} passed"
A single case in the file looks like this:
{"id": "refund-01", "question": "Can I get a refund after 45 days?",
"must_include": ["30 days"], "must_not_include": ["60 days"],
"needs_citation": true}
This is deliberately simple. It won't judge tone or reasoning. But it catches the expensive failures, like a retired policy, and it runs in any pipeline.
Why Repeat Runs Matter, and How Much Confidence They Buy
Suppose a case passes 18 of 20 runs. Is it a 90 percent case? Perhaps. But with only 20 runs, the true rate could plausibly be anywhere from about 70 to 97 percent (a standard 95 percent Wilson interval). With 200 runs and the same 90 percent observed, the plausible range tightens to roughly 85 to 93 percent.
That's why I'm wary of reports that say "the new prompt improved accuracy from 88 to 91 percent" on a small set. Differences of a few points on small samples are often noise. Use enough cases and enough repeats that your comparisons mean something, and compare distributions across runs, not single outputs. Where your provider lets you pin a model version and reduce sampling randomness, do it for regression testing, while still testing at the settings you use in production.
Retrieval Deserves Its Own Tests
For a retrieval-augmented system, test the retrieval step directly, before the model ever sees the context. Does the right passage appear in the top few results for each question? Are the results relevant, or merely similar-looking? Are retired or conflicting documents excluded? How does it behave when nothing relevant exists, and does the model then say so, or invent something?
Also test the boundary between retrieval and generation. A model can ignore good context, or lean on its own memory instead of the documents. Groundedness checks, which ask whether each claim in the answer is supported by the retrieved text, catch this. Most of the "hallucination" I'm asked to investigate turns out to be one of three things: the right passage wasn't retrieved, the wrong passage was, or the model went beyond what the passage said.
Agents: Testing Actions, Not Just Words
Once a model can call tools, the risk changes from a wrong sentence to a wrong action. An assistant that can look up an order is one thing. An assistant that can issue a refund or send an email is another.
Test whether it picks the right tool, supplies correct arguments, and stops when it should. Set hard limits on steps and spending so a confused loop ends cheaply. Check that every action respects the permissions of the person asking, not the permissions of the service account behind the agent. And require human approval for actions that are expensive or hard to undo. In agent tests I look at the whole trace, not only the final answer, because a correct answer reached through a risky path is still a finding.
Security: The Part Most Teams Underestimate
AI applications introduce attack paths that traditional security testing wasn't designed for. The OWASP Top 10 for LLM Applications is a useful checklist, and a few items deserve emphasis.
Prompt injection. Direct, where a user tells the system to ignore its instructions, and indirect, where malicious instructions are hidden in a document, web page or email that the system retrieves or reads. The indirect form is the more worrying one, because the user never typed anything suspicious.
Data leakage. Can a user coax the system into revealing its instructions, other customers' data, or internal documents they shouldn't see?
Excessive agency. Tools and permissions broader than the feature needs.
Unsafe output handling. Model output passed straight into a database query, a shell or a web page without the same validation you'd apply to user input.
Build an injection corpus, run it regularly, and treat guardrails as one layer of defense, not the whole defense. Guardrails reduce risk. They don't remove it. The stronger controls are architectural: least privilege, separation between instructions and untrusted content, and approvals for sensitive actions.
Bias and Fairness: Test With Pairs
For features that touch people, such as screening, recommendations or support prioritization, use counterfactual pairs. Take the same case and change only an attribute that shouldn't matter, like a name or a pronoun, then compare the outputs. Systematic differences are findings. Keep the limits in mind: a pair test shows the problem exists in the cases you tried, not that the system is free of it elsewhere. And what counts as fair depends on context and regulation, so this is a place to involve legal and domain experts, not just engineers.
Speed and Cost Are Quality Attributes Too
An answer that arrives in twenty seconds is often a failed answer. For model-backed features, measure time to first token and total generation time, tokens per second for streaming, cost per completed task (not per request, since agents may make many calls), and what happens when the provider throttles you. Load test with a realistic mix of short and long prompts, since a handful of very long ones can use far more capacity than their share of traffic would suggest. Check the fallback too: when the primary model is slow or unavailable, what does the user see?
Every Change Is a Release
In a conventional product, code changes trigger regression tests. In an AI product, a change to the prompt, the model version, the embedding model, the chunking strategy, the retrieval settings or the tool descriptions can change behavior as much as any code change, often with no code change at all.
Treat each of those as a release. Run the evaluation set before and after, compare against a stored baseline, and look at which cases moved and why, not only the aggregate. Introduce changes gradually where you can, with a small share of traffic or shadow comparison before everyone sees the new behavior. And be aware that providers update models, so the system can change underneath you. Version pinning helps when it's available, and scheduled re-evaluation covers you when it isn't.
After Launch: Observability and Drift
Production is the most honest test environment, and it's where drift appears. Questions change, documents change, providers change, and the world changes. A system that was accurate in March can be wrong in September without anyone touching it.
Log enough to reconstruct any interaction: the user input, the retrieved passages, the prompt as assembled, the tool calls, the output, latency and cost, subject to your privacy rules. Sample real interactions for human review every week. Watch leading signals such as thumbs-down rates, escalations to humans, retrieval misses, refusal rates and shifts in the kinds of questions arriving. Feed the failures back into the evaluation set. That loop, production to evaluation set to release gate, is the most valuable habit in this whole discipline.
Governance for the Enterprise
At enterprise scale, testing is also evidence. A few practices help.
Tier by risk. A summarizer for internal notes and a system that influences credit or care decisions don't need the same rigor. Decide the tiers up front and tie required testing to them.
Keep records. Store evaluation results, versions, approvals and known limitations with each release. Frameworks such as the NIST AI Risk Management Framework, and regulations such as the EU AI Act, ask for documentation of this kind, and your auditors will too.
Define human oversight. Say who reviews what, how often, and what they can do when they find a problem.
Assign ownership. Every knowledge source, prompt and evaluation set needs a named owner. Unowned data is how the retired refund policy stays in the index.
Limits Worth Admitting
No test suite proves an AI system is safe. You're sampling a very large space of inputs, and the failures that matter most are often the ones nobody thought to include. Judge models disagree with humans in ways that are hard to predict. Evaluation sets age, and a set that was representative last year may not be now. And the more you test, the more it costs, so you'll make tradeoffs on coverage. Say so in your reports. A clear statement of what was and wasn't tested builds more trust than a clean-looking score.
Mistakes That Repeat
Testing only the model and not retrieval or data. Using a handful of impressive demo prompts as the test suite. Treating a single run as a result. Trusting a judge model without comparing it with human grades. Giving agents broad permissions because it was easier. Skipping the "what happens when it doesn't know" cases. Shipping prompt changes without regression runs. And launching without a plan for who reviews production failures.
A Closing Thought
Here's a recommendation you can act on this week. Gather fifty real questions your users have asked, write down what a good answer and a bad answer look like for each, and run your current system against them ten times. Don't build anything fancy. Just look at the cases that fail inconsistently, because those are your real risks and also your best argument for investing in the rest.
I expect evaluation sets to become the closest thing AI products have to a specification: owned, versioned, reviewed and negotiated between product, engineering and risk teams, the way API contracts are today. Teams that build the habit early will be able to change models and prompts quickly, because they'll know within an hour what a change did. Teams that don't will either move slowly or learn from their customers. If you'd like help building that capability, from evaluation design to red teaming and governance evidence, our team at PrimeQA Solutions provides AI Testing Services for engineering and product leaders who need to ship AI features with confidence.