17 August 2026, 01:54 PM
[attachment=8952]
The Model Was Fine. Our Own Integration Code Parsed It Wrong.
A client's support summarization feature started producing blank summaries for a specific ticket category, and the first assumption, reasonably, was a model regression. It wasn't. The LLM provider had made a minor, documented change to how it structured a nested field in its response for a particular content type, and the client's own parsing code, written against the old shape, silently failed to extract the summary text and fell back to an empty string instead of raising an error anyone would notice. The model had done its job correctly the entire time. The integration code reading its output was the actual point of failure.
This is the category of bug that API testing for LLM integrations exists to catch, and it's a different testing surface than either testing the LLM API's own functional contract or testing the model's output quality. It's specifically about the glue code, the adapters, parsers, retry wrappers, and provider abstractions sitting between your application and whichever LLM API it calls, and it's a surface I see under-tested constantly, usually because teams assume it's covered by either their unit tests or their end-to-end tests when it's actually neither. Here are the specific mistakes behind that gap.
Mistake One: Mocks That Are Too Shallow to Catch Anything Real
Most integration test suites mock the LLM API call, for good reason, hitting a real API in every CI run is slow, costly, and non-deterministic. The mistake isn't mocking. It's mocking with a single, trivial, hand-crafted response that never resembles the actual range of shapes a real API returns: missing optional fields, unusually nested structures for certain content types, empty arrays where content was expected, or the specific documented-but-rare response variants that only show up under particular conditions.
A shallow mock gives your integration tests a green checkmark while validating almost nothing about how your parsing code handles real-world response variation. Build your mock response library from actual captured API responses across a genuinely varied set of real requests, including the edge cases and documented alternate response shapes, not a single idealized example written from memory of what the docs said the response should look like.
Mistake Two: No Contract Tests Running Against the Real API
This is the mistake behind the incident that opened this piece. If every single test in your suite runs against a mock, nothing in your testing process will ever notice when the real provider's API drifts away from what your mocks assume, a field gets renamed, a nested structure gets flattened, a previously guaranteed field becomes optional. Your mocks stay comfortably in sync with themselves while silently diverging from reality.
The fix is a small, deliberately separate layer of contract tests that do hit the real API, on a much lower frequency than your full mocked suite, specifically to verify the actual response shape still matches what your parsing code assumes. These don't need to run on every commit. Running them on a schedule, daily or with each deployment, is usually the right balance between catching drift early and not incurring real API cost and latency on every test run.
Mistake Three: Testing Only the Happy Path of Response Parsing
Integration tests frequently validate that parsing works when the response looks exactly like the documented example. They're much less likely to test what happens when a field is missing entirely, when a value is null instead of the expected type, when an array is empty instead of populated, or when the response includes an error or partial-completion indicator nested inside what looks like an otherwise successful response.
Your parsing and extraction code needs explicit test coverage for each of these degraded shapes, not just the clean documented case, because production traffic reliably produces the messy version eventually, and the client's blank-summary incident is exactly what happens when that messy version arrives and nothing was built or tested to handle it gracefully.
Mistake Four: Assuming a Shared Interface Means Shared Test Coverage
Platforms supporting multiple LLM providers behind a common internal interface, so the rest of the application doesn't need to know or care which provider is actually handling a given request, often make a subtle testing mistake: treating the abstraction layer's uniform interface as evidence that testing one provider adequately covers the others. It doesn't. Each underlying provider has its own response quirks, its own error formats, its own rate-limit behavior, and its own edge cases in how it handles things like function calling or structured output, and the adapter code translating each provider's specific behavior into your common interface needs its own independent test coverage per provider, not a single shared test suite run once against whichever provider happens to be configured in the test environment.
Mistake Five: Not Testing Your Own Retry and Fallback Logic as Code
This is distinct from infrastructure-level load and resilience testing, which validates how the system behaves under real concurrent load and real provider degradation. This mistake is about the actual code path: does your retry wrapper correctly distinguish a retryable error from a non-retryable one, does it back off appropriately rather than retrying immediately, and if you support fallback to a secondary provider, does that fallback logic actually trigger under the conditions it's supposed to, and does it correctly translate the request into the secondary provider's expected format rather than assuming the primary provider's request shape works unchanged everywhere.
Test this with deliberately simulated failure responses, timeouts, rate-limit errors, malformed responses, injected directly into your integration layer's test harness, and verify the retry and fallback code actually does what it's supposed to in each specific scenario, rather than only testing it once informally during initial development and never revisiting it.
Mistake Six: Not Testing Prompt Template Rendering Under Edge-Case Input
If your integration layer builds prompts from templates with variable substitution, user input, retrieved context, conversation history, inserted into a template before the request goes out, that template rendering step deserves its own test coverage. Empty variables, unusually long input that might need truncation, special characters or formatting that could break template syntax, and multi-language input if your application supports it are all realistic conditions that template rendering code needs to handle correctly, and they're easy to miss if template testing only ever uses clean, short, English-language example inputs during development.
A Visual Breakdown of the Integration Testing Layers
[attachment=8953]
A Practical Checklist
Where This Leaves Enterprise Teams
The incidents that actually reach production in LLM-integrated applications are rarely about the model getting something wrong. They're about the code translating between your application and the model getting something wrong, quietly, in a layer that felt too mechanical to need dedicated testing. That layer is exactly where the real risk concentrates once you stop assuming a clean mock and a passing end-to-end demo add up to real coverage.
This layered approach to testing the integration boundary is central to how PrimeQA Solutions structures AI Integration Testing for enterprise clients connecting applications to LLM providers, because the bug that breaks a feature in production is almost never the one anyone thought to blame first.
The Model Was Fine. Our Own Integration Code Parsed It Wrong.
A client's support summarization feature started producing blank summaries for a specific ticket category, and the first assumption, reasonably, was a model regression. It wasn't. The LLM provider had made a minor, documented change to how it structured a nested field in its response for a particular content type, and the client's own parsing code, written against the old shape, silently failed to extract the summary text and fell back to an empty string instead of raising an error anyone would notice. The model had done its job correctly the entire time. The integration code reading its output was the actual point of failure.
This is the category of bug that API testing for LLM integrations exists to catch, and it's a different testing surface than either testing the LLM API's own functional contract or testing the model's output quality. It's specifically about the glue code, the adapters, parsers, retry wrappers, and provider abstractions sitting between your application and whichever LLM API it calls, and it's a surface I see under-tested constantly, usually because teams assume it's covered by either their unit tests or their end-to-end tests when it's actually neither. Here are the specific mistakes behind that gap.
Mistake One: Mocks That Are Too Shallow to Catch Anything Real
Most integration test suites mock the LLM API call, for good reason, hitting a real API in every CI run is slow, costly, and non-deterministic. The mistake isn't mocking. It's mocking with a single, trivial, hand-crafted response that never resembles the actual range of shapes a real API returns: missing optional fields, unusually nested structures for certain content types, empty arrays where content was expected, or the specific documented-but-rare response variants that only show up under particular conditions.
A shallow mock gives your integration tests a green checkmark while validating almost nothing about how your parsing code handles real-world response variation. Build your mock response library from actual captured API responses across a genuinely varied set of real requests, including the edge cases and documented alternate response shapes, not a single idealized example written from memory of what the docs said the response should look like.
Mistake Two: No Contract Tests Running Against the Real API
This is the mistake behind the incident that opened this piece. If every single test in your suite runs against a mock, nothing in your testing process will ever notice when the real provider's API drifts away from what your mocks assume, a field gets renamed, a nested structure gets flattened, a previously guaranteed field becomes optional. Your mocks stay comfortably in sync with themselves while silently diverging from reality.
The fix is a small, deliberately separate layer of contract tests that do hit the real API, on a much lower frequency than your full mocked suite, specifically to verify the actual response shape still matches what your parsing code assumes. These don't need to run on every commit. Running them on a schedule, daily or with each deployment, is usually the right balance between catching drift early and not incurring real API cost and latency on every test run.
Mistake Three: Testing Only the Happy Path of Response Parsing
Integration tests frequently validate that parsing works when the response looks exactly like the documented example. They're much less likely to test what happens when a field is missing entirely, when a value is null instead of the expected type, when an array is empty instead of populated, or when the response includes an error or partial-completion indicator nested inside what looks like an otherwise successful response.
Your parsing and extraction code needs explicit test coverage for each of these degraded shapes, not just the clean documented case, because production traffic reliably produces the messy version eventually, and the client's blank-summary incident is exactly what happens when that messy version arrives and nothing was built or tested to handle it gracefully.
Mistake Four: Assuming a Shared Interface Means Shared Test Coverage
Platforms supporting multiple LLM providers behind a common internal interface, so the rest of the application doesn't need to know or care which provider is actually handling a given request, often make a subtle testing mistake: treating the abstraction layer's uniform interface as evidence that testing one provider adequately covers the others. It doesn't. Each underlying provider has its own response quirks, its own error formats, its own rate-limit behavior, and its own edge cases in how it handles things like function calling or structured output, and the adapter code translating each provider's specific behavior into your common interface needs its own independent test coverage per provider, not a single shared test suite run once against whichever provider happens to be configured in the test environment.
Mistake Five: Not Testing Your Own Retry and Fallback Logic as Code
This is distinct from infrastructure-level load and resilience testing, which validates how the system behaves under real concurrent load and real provider degradation. This mistake is about the actual code path: does your retry wrapper correctly distinguish a retryable error from a non-retryable one, does it back off appropriately rather than retrying immediately, and if you support fallback to a secondary provider, does that fallback logic actually trigger under the conditions it's supposed to, and does it correctly translate the request into the secondary provider's expected format rather than assuming the primary provider's request shape works unchanged everywhere.
Test this with deliberately simulated failure responses, timeouts, rate-limit errors, malformed responses, injected directly into your integration layer's test harness, and verify the retry and fallback code actually does what it's supposed to in each specific scenario, rather than only testing it once informally during initial development and never revisiting it.
Mistake Six: Not Testing Prompt Template Rendering Under Edge-Case Input
If your integration layer builds prompts from templates with variable substitution, user input, retrieved context, conversation history, inserted into a template before the request goes out, that template rendering step deserves its own test coverage. Empty variables, unusually long input that might need truncation, special characters or formatting that could break template syntax, and multi-language input if your application supports it are all realistic conditions that template rendering code needs to handle correctly, and they're easy to miss if template testing only ever uses clean, short, English-language example inputs during development.
A Visual Breakdown of the Integration Testing Layers
[attachment=8953]
A Practical Checklist
- Mock response libraries are built from real, varied captured API responses, including edge cases, not a single idealized example
- A separate, lower-frequency contract test suite runs against the real API to catch provider-side drift before it reaches production
- Response parsing is explicitly tested against missing fields, null values, empty arrays, and partial or error-nested responses
- Each supported provider has its own independent adapter test coverage, not a single shared suite run against whichever provider is configured
- Retry, backoff, and fallback logic is tested with deliberately injected failure scenarios, not just observed informally during development
- Prompt template rendering is tested against empty, oversized, special-character, and multi-language input
Where This Leaves Enterprise Teams
The incidents that actually reach production in LLM-integrated applications are rarely about the model getting something wrong. They're about the code translating between your application and the model getting something wrong, quietly, in a layer that felt too mechanical to need dedicated testing. That layer is exactly where the real risk concentrates once you stop assuming a clean mock and a passing end-to-end demo add up to real coverage.
This layered approach to testing the integration boundary is central to how PrimeQA Solutions structures AI Integration Testing for enterprise clients connecting applications to LLM providers, because the bug that breaks a feature in production is almost never the one anyone thought to blame first.