Skip to content
Multigrid

Regression suites for prompts

Sizing, scoping and running a suite against a prompt that changes weekly, and why a green suite still misses regressions.

A regression suite for ordinary code answers a question with a yes or a no: given this input, is the output still what it was. A prompt has no such question available to it. The output is drawn from a distribution, so the same input run twice is two different strings, and the assertion everybody reaches for first — that the model still says what it said — fails on days when nothing is wrong and passes on days when something is. The suite that results is a red build people learn to ignore, which is worse than no suite, because it costs money and buys distrust.

These pages are about the two decisions that actually make the suite work: what you assert on when you cannot assert on the text, and how big the thing is allowed to get before it stops being run. Both are arithmetic more than taste. The rest — organising cases by failure mode, tagging them to the prompt revision they were written against, running only the ones a change can affect, and keeping a control run going so you can tell a provider’s change from your own — follows from getting those two right.

Sizing a Regression Suite for a Prompt That Changes Weekly

The suite size that survives a weekly prompt change is set by your triage budget, not by your CI budget, and the arithmetic goes the opposite way to intuition.

10 min read

Structuring a Prompt Regression Suite by Failure Mode

Organising cases by what they would catch rather than by which feature they exercise, so a red build tells you what broke before you open it.

10 min read

Running a Regression Suite Only on Changed Prompts

Making the dependency from a prompt to its tests visible to the test runner, so a one-prompt commit runs a handful of cases instead of all of them.

10 min read

Tagging Regression Tests to a Specific Prompt Version

A three-field convention that makes a failing case say which prompt revision it was written against, and one number it lets you compute about the whole suite.

9 min read

When a Passing Regression Suite Still Missed a Regression

The five ways a green prompt suite is compatible with a broken system, and why sampling once per case is the one that catches nearly everybody.

10 min read

Baseline Drift: Why Old Regression Tests Stop Meaning Anything

Stored outputs, stored thresholds and stored judgements each decay in a different direction as the model and the prompt move underneath them.

10 min read

Writing a Regression Test From a Support Ticket

Turning a customer complaint about model output into a permanent case: recovering the exact input, minimising it safely, and choosing the assertion the ticket actually implies.

10 min read

Parallelising a Slow Prompt Regression Suite

Choosing the concurrency number from your token-per-minute limit rather than from your CPU count, and finding the shared state that breaks when you do.

11 min read

Running the Same Regression Suite Against Two Model Versions

Using an existing suite as a migration gate: parameterising the model, running both versions paired in one job, and reading the four buckets the results fall into.

10 min read

Regression Testing After a Provider Silently Updates a Model

How a suite goes red on a commit that changed nothing, how to establish it was the provider rather than you, and why only a scheduled run on a frozen tree can tell you.

10 min read

Other topics