Skip to content
Multigrid

Testing AI systems in practice

The specific tests worth writing — timeouts, budgets, schemas, thread safety, redaction — each one a failure somebody has already shipped.

The advice that a language model application needs tests is easy to agree with and hard to act on, because the first test anyone writes asserts on a sentence, and the sentence changes. The useful question is narrower: given an output you cannot predict, what is left that you can still assert on? The answer is usually a great deal — a schema, a call count, a tool name, a field that must survive a summary, a threshold, a redaction, a spend cap, an ordering. None of those are the model’s prose, and all of them are things that have broken in production.

These pages are the specific layer under the general advice. Each one starts from a failure that has been shipped, names the property that would have caught it, and shows the test that checks that property without a live model in the loop where a live model is avoidable. Where the code depends on a library surface that moves, the page says which version of which documentation it was read from.

A Compatibility Test Suite for Ollama's OpenAI-Compatible Endpoint

A table-driven suite that probes which OpenAI request fields Ollama's /v1 endpoint honours, ignores silently, or rejects.

9 min read

A Compatibility Test Matrix Across OpenAI-Compatible Providers

A provider-by-feature matrix generated from probes and committed to the repo, so a capability change shows up as a diff instead of a support ticket.

9 min read

Testing Retry and Backoff Without Waiting for the Backoff

Assert a four-minute retry ladder in milliseconds by injecting the clock and the randomness, then checking the delay sequence, the jitter bounds and the give-up condition.

10 min read

Testing That an Agent's Memory Persists Correctly Across Turns

Drive a scripted multi-turn conversation against a recording stub and assert on the memory store, so the test never depends on what the model said.

9 min read

Testing a Multi-Turn Conversation for Context Loss Between Turns

Intercept the outbound request and assert that a fact established in turn one is still in the payload at turn five, instead of asking the model whether it remembers.

9 min read

Testing That the System Prompt Isn't Overwritten Mid-Conversation

The assistant abandons its instructions partway through a session; the cause is usually a second system message in the history, and one invariant catches it.

9 min read

Testing That JSON Mode Actually Returns Valid JSON Every Time

A corpus of adversarial inputs, three assertions per case, and a failure rate rather than a pass or fail — because JSON mode fails on specific inputs, not randomly.

10 min read

Testing max_tokens Truncation Doesn't Break JSON Output

Unexpected end of JSON input from a cap that cut the object mid-string, why JSON mode does not prevent it, and the guard that turns it into a clean error.

9 min read

Fuzzing a JSON Schema to Find Where Structured Output Breaks

Generate schemas along one axis at a time — depth, width, enum size — and find the size at which a provider stops accepting or stops honouring them.

10 min read

Testing Backward Compatibility When You Add a Field to an Output Schema

Adding an optional field breaks strict consumers and, under strict structured outputs, is not optional at all — here is the two-direction test that proves a change is safe.

9 min read

A Docker Compose Test Environment for LLM-Calling Code

A compose file that runs a stub inference server and a real local runtime beside the app, so the whole suite works with the network unplugged.

9 min read

Using a Local Model in Tests Instead of Paying for API Calls

What a small local model can and cannot stand in for, and why it belongs between mocks and the real provider rather than replacing either.

8 min read

A Staging Configuration That Mirrors Production Model Settings

Which inference settings must be byte-identical between staging and production, which must differ, and the test that proves nothing else drifted.

8 min read

Why Your Staging Tests Pass but Production Fails With a Different Model

The suite is green, production is wrong, and the two environments are running different weights — how to confirm that in one request and find which layer did it.

9 min read

Testing That an Embedding Model Returns Stable Vectors for the Same Input

Repeated calls with identical input do not return identical vectors; assert cosine similarity against a stability threshold, and test batch position too.

9 min read

Testing That Cosine Similarity Thresholds Don't Drift After a Model Update

A hardcoded 0.82 stops meaning what it did when the embedding model changes; derive the threshold from a labelled set and assert the separation instead.

9 min read

Mutation Testing a Prompt Test Suite to Check It Actually Catches Failures

Deliberately damage the prompt, the model choice and the sampling config, and score your suite on how many of those mutants it turns red.

9 min read

Testing the Tests: Injecting a Known Bad Prompt to Verify Your Suite Catches It

A quarterly drill: one person breaks the prompt in a way the team has actually shipped before, and everyone else finds out whether CI notices.

8 min read

Testing Your App's Behaviour Under a Simulated 429 From Every Provider

One parameterised harness that returns each provider's own 429 body and headers, asserting your handling collapses to a single behaviour.

9 min read

Testing That a Rate Limiter Queues Requests Instead of Dropping Them

Fire more work than the limiter allows and assert every request eventually resolves, in order, at the configured rate, with a bounded queue.

9 min read

Regression Testing a Fine-Tuned Model Against Its Base Model's Test Suite

Run the base model's existing suite against the fine-tune and gate on the set of newly failing cases, not on an aggregate score.

9 min read

Testing for Catastrophic Forgetting After a Fine-Tuning Run

Why narrow fine-tuning degrades capabilities nobody trained on, and how a held-out general set run before and after makes the damage visible.

9 min read

Testing Prompt Output Consistency Across Languages

Run one semantic case in several languages and assert on the invariants that should not vary — schema, enum, tool, language of the reply.

9 min read

Testing That a Translated Prompt Still Triggers the Same Tool Calls

Assert the tool name and argument keys across languages, normalise the values that are allowed to differ, and catch the model answering instead of calling.

9 min read

Testing That Your Token-Counting Code Matches the Provider's Actual Bill

Your estimate says 1,842 and the invoice says 2,106 — the eight reasons a local tokenizer desyncs, and the test that catches the next one.

10 min read

A Test That Fails When Estimated Cost Diverges From Actual Cost by More Than 5%

Build the divergence assertion on an aggregate ratio, derive the 5% threshold from what the estimate is used for, and test the price table separately.

9 min read

Unit Testing Prompt Template Rendering With Every Variable Combination

Test the renderer as a pure function against empty, huge, delimiter-bearing and template-bearing inputs, with properties that hold for all of them.

9 min read

Testing That a Prompt Template Doesn't Leak an Unfilled Placeholder

A raw template variable reached the model — why the renderer allowed it, why the output looked fine anyway, and where to put the guard.

9 min read

Testing Few-Shot Example Selection Logic Independent of the Model Call

Freeze the embeddings, treat the selector as a pure ranking function, and assert on ids, order, budget and leakage rather than on output quality.

9 min read

Testing That Output Length Stays Within a Product's Character Budget

max_tokens does not bound characters — count graphemes, test the truncator as well as the assertion, and decide what over-budget means.

9 min read

Testing That a Circuit Breaker Opens After the Right Number of Failures

Unit-test the breaker's own state machine: the threshold, what resets it, fail-fast while open, single-probe half-open, and which errors count at all.

9 min read

Testing Cache Invalidation After a Prompt Version Changes

Make the cache key a total function of everything that affects the answer, then prove a prompt edit misses and an unchanged prompt hits.

9 min read

End-to-End Testing an AI Chat Widget With Playwright

Intercept the streaming response at the network layer, then assert on the transformation the widget performs rather than on the model text you supplied.

9 min read

End-to-End Testing an AI Chat Widget With Cypress

What cy.intercept can and cannot do with a chunked SSE body, and the window-level transport stub that covers the rest.

9 min read

Mocking a WebSocket-Based AI Chat in an End-to-End Test

Playwright's routeWebSocket gives you both sides of the socket, which makes the reconnection and deduplication assertions that SSE tests cannot reach.

9 min read

Building a Test Report That Shows Which Prompt Version Broke What

Carry the prompt's content hash into the test report as a property, diff against a stored baseline, and triage by prompt file instead of by test name.

9 min read

Alerting the Right Person When a Specific Prompt Test Fails

Derive the owner from the prompt file the failing test loaded, not from the test file, and send one message per cause instead of one per case.

8 min read

Testing That a Guardrail Actually Blocks What It Claims To Block

Turn each documented claim into a row with positives and near-miss negatives, and assert on the guardrail's decision object rather than its refusal text.

9 min read

Testing a Guardrail's False-Positive Rate on Legitimate Requests

How to derive a false-positive rate from a labelled corpus, and why a 500-example corpus cannot distinguish one per cent from three.

9 min read

Testing Behaviour at the Exact Edge of a Model's Context Window

Find the real boundary empirically from the provider's own error, then assert the overflow is a clean typed error rather than silent truncation or a retried 400.

9 min read

Testing That Retrying a Failed Tool Call Doesn't Duplicate a Side Effect

A timeout tells you nothing about whether the effect happened, so the test is about the key that makes the second attempt a no-op.

9 min read

Testing That Upgrading the Provider's SDK Doesn't Change Output Silently

Snapshot the outbound HTTP request rather than the model's answer, because the request is deterministic and the answer is not.

9 min read

Naming Conventions for a Prompt Test Suite That Scales Past 500 Cases

A three-segment scheme — feature, failure mode, case — that makes selectors, grouping and ownership routing fall out of the name.

8 min read

Testing That Fallback Providers Are Tried in the Configured Order

Assert on the recorded sequence of attempts under forced failures, including the case where a non-retryable error must not fall through at all.

9 min read

Testing a Timeout on a Stream That Never Sends Its Final Chunk

Why a read timeout never fires on a stream that keeps sending keepalives, and how to test the total deadline and the missing terminal frame separately.

9 min read

Golden Master Testing for a Legacy Prompt You're Afraid to Touch

Characterization testing applied to a prompt: capture the projection of the output that is actually stable, and let the unstable parts stay unasserted.

9 min read

Test-Driven Development for a Prompt: Writing the Test Before the Prompt

The red step has a specific meaning here — the case must fail with an empty prompt — and that one rule is what stops the suite from testing the model.

9 min read

Scripting a Synthetic User to Drive an Agent Through a Test Scenario

A second model plays the user against a written script, and the assertions land on the agent's tool calls and terminal state rather than on the transcript.

9 min read

Testing How Often a Model Calls a Tool It Was Not Given

A membership assertion on the emitted tool name, plus the dispatcher guard that turns an invented tool into a recoverable turn instead of a 500.

9 min read

Why Rotating an API Key Broke Every Test in CI

The 401 after a routine key rotation has about six causes; a short diagnosis ladder tells you which one you have without printing the key.

9 min read

A Pre-Deploy Check That a Prompt Still Fits the Context Window

Assemble the prompt the way production assembles it, count it with the provider's own tokenizer, and fail the deploy on a budget you wrote down.

9 min read

Testing Strict Mode for Structured Outputs the Way You Can Actually Test It

Three assertions you own — request-time schema rejection, independent re-validation, and the refusal branch — instead of trying to test the provider's grammar.

9 min read

Load Testing an Embeddings Endpoint Under Batch Requests

Embeddings load is measured in items and tokens per second across batch sizes, not requests per second, and the limits you hit first are payload size and tokens per minute.

9 min read

Testing Whether a Prompt Is Portable Before You Switch Model Families

Run the same invariant assertions against every candidate model and read the result as a matrix, so a migration fails in a test rather than in production.

10 min read

Testing That an Agent Respects Its Maximum Step Budget

Mock a model that never stops asking for tools and assert the loop halts at exactly the configured count, with a typed stop reason the caller can act on.

9 min read

Unit Testing a Custom Output Validator Before You Wire It to a Model

A table of known-good and known-bad outputs, assertions on the error list rather than on a boolean, and the edge rows that decide whether the retry prompt is useful.

9 min read

Recording Real Provider Error Responses as Fixtures

Guessed error payloads have the wrong field names and no headers; capture the real ones once, strip the credentials, and replay them from a mock server.

10 min read

Testing SSE Reconnect Logic After a Dropped Connection

The invariant is that the text assembled across a drop equals the text from an unbroken stream, and the test needs a server that cuts the socket mid-event.

10 min read

Keeping Prompt Content Out of CI Logs When a Test Fails

Assertion failures print both sides of the comparison, which is how a customer document ends up in a build log; assert on a projection instead.

9 min read

Testing That a Cost Alert Actually Fires at the Threshold You Set

Feed the alerting logic a synthetic cost stream derived from named token counts and a labelled price, and assert it fires once, at the right event, in the right units.

10 min read

Pinning an API Version in Tests So a Provider Update Cannot Break You Silently

Assert the version header on the wire where a provider dates its API, and where it does not, snapshot the serialised request so an SDK bump that changes a default fails a test.

9 min read

Testing That a Prompt Cache Hit Behaves Like a Cache Miss

First prove you actually got a hit, then assert equivalence on invariants rather than on the exact string, and test the cache key as the pure function it is.

10 min read

Testing Webhook Delivery for an Asynchronous LLM Job

Signature verification on raw bytes, idempotency on the delivery id, a fast acknowledgement, and a reconciler for the delivery that never arrives.

10 min read

Testing Partial Failures in a Batch Inference Job

How to test that a batch job whose third row fails still returns the other ninety-nine, and what to assert about the row that did not.

9 min read

Testing That a Routing Rule Sends a Request to the Right Model

Unit-testing routing configuration as a pure function from request conditions to a model target, with a case table that also proves no rule is dead.

9 min read

Regression Testing a Summarisation Prompt Against Must-Keep Facts

Instead of comparing summaries, assert a fixed checklist of facts survives the edit — one assertion per fact, so a failure names what was dropped.

10 min read

Testing That a Sub-Agent's Output Is Validated Before the Parent Uses It

The parent-child boundary in a multi-agent system is a trust boundary; here is how to test that it actually validates instead of concatenating.

10 min read

Testing That Two Semantically Identical Requests Produce the Same Cache Key

Key generation is a pure function, so test it with two sets: pairs that must collide and pairs that must never collide.

10 min read

Testing the Fallback Prompt Your App Uses When the Primary Model Errors

The fallback path only runs during an incident, which is the worst possible time to discover it produces a different output shape.

9 min read

Calculating How Many Users a 5% Prompt Rollout Actually Reaches

The arithmetic connecting a rollout percentage to real user counts, exposure volume and how long it takes to detect a problem.

9 min read

Testing That a Model Never Returns a Value Outside an Enum

Enum fields are the structured-output constraint most likely to be enforced by your parser rather than by the provider, and the test is adversarial.

9 min read

Testing How Your App Handles an Empty Completion From the Model

The crash is a null content field, and there are at least six distinct ways a provider can return one; here is the fixture set that reproduces each.

10 min read

Testing Thread Safety of a Shared LLM Client Instance

A singleton client with a request-scoped header is a cross-tenant leak waiting for enough concurrency; the test forces the interleaving instead of hoping for it.

10 min read

Testing That Long Inputs Are Truncated Consistently

Truncation is a pure function with boundary cases and invariants, and testing it without a model catches the silent corruption a model call would hide.

9 min read

Testing for Default Parameter Changes After a Major SDK Version Bump

Snapshot the HTTP request your code produces, not the response you get back, and the changed default the changelog omitted shows up as a one-line diff.

9 min read

Testing an Image-Plus-Text Prompt Without a Real Image Every Run

Generate a tiny deterministic image once, keep it as a fixture, and assert on the request payload rather than on what the model says it sees.

9 min read

Testing That a Tool-Calling Loop Stops Before It Exceeds a Cost Budget

Drive a mocked loop toward a runaway and assert the ceiling halts it before the call that would breach, not after.

10 min read

Testing That a User Stays in the Same Prompt Variant Across a Session

Assignment has to be a pure function of a stable key, and the tests that matter are about process restarts, identity changes and conversations that outlive a deploy.

10 min read

An Override Process for When an Eval Gate Blocks an Urgent Fix

How to ship past a failing eval gate during an incident without the override becoming the normal path.

9 min read

Detecting When a Recorded Cassette No Longer Matches the Live API

A scheduled job that re-records your cassettes against the live API and diffs the response shape, so fixture rot is caught before production is.

10 min read

What a Full Regression Suite Costs to Run on Every Commit

A complete worked derivation from case count and token counts to a monthly bill, with every assumption labelled and substitutable.

10 min read

Testing That a Retried Async LLM Job Doesn't Fire Its Webhook Twice

How to force the crash window between a completed model call and an acknowledged webhook, and assert exactly one delivery survives the retry.

9 min read

A Contract Test for Error Response Shape Across Providers

Pin every provider's error envelope to one normalised result, and fail the build when a provider grows a code your handler has never seen.

10 min read

Testing an Agent's Behaviour When a Tool Call Times Out Mid-Loop

Assert that a tool exceeding its deadline becomes a recoverable tool result the model can react to, rather than an exception that ends the conversation.

9 min read

Testing That Log Redaction Runs Before a Failed Test Prints the Prompt

Why a test framework dumps raw prompt content on the failure path specifically, and a meta-test that proves the redaction fires there.

9 min read

Testing That a Redacted Fixture Still Reproduces the Original Bug

Why scrubbing a fixture usually destroys the property that triggered the bug, and how to redact so the reproduction survives.

9 min read

When to Record a Cassette and When to Hand-Write a Fixture

A decision rule based on whether the thing you are testing is the provider's behaviour or your own reaction to it.

9 min read

Testing That Concurrent Requests to One Cache Key Don't Stampede

Reproduce many simultaneous misses on one key and assert the model was called once, including the failure case that poisons a naive fix.

9 min read

A Smoke Test That Runs Against the Real API Before Every Deploy

Two live calls that prove the environment is wired correctly, and a precise list of what they must not try to check.

9 min read

Testing Correct Handling of a Provider's Announced Maintenance Window

Simulate a provider in maintenance mode and assert the app degrades or fails over deliberately, including what happens when the window ends.

9 min read

A Test That Fails When Provider Latency Passes a Regression Threshold

How to put a latency assertion in a suite without it flaking, by comparing a percentile against a control run rather than against a fixed number.

10 min read

Testing a Model Swap Doesn't Change Your Function-Calling Schema Contract

Run the same tool-calling cases against both models and validate every call against one schema, asserting structure and never argument prose.

10 min read

Testing That a Cache Doesn't Serve a Stale System Prompt After a Deploy

Where the old system prompt actually survives a deploy, why it is not the provider's cache, and the key-derivation test that stops it.

10 min read

A Dead-Letter Queue Test for Tool-Call Failures, Not Message Failures

Why a failed tool execution belongs in a different queue from a failed message, and how to test that the routing between them is correct.

10 min read

Other topics