Sizing a Regression Suite for a Prompt That Changes Weekly

“As many cases as you can afford” is not an answer, because the thing you cannot afford is not the one people assume. Run the numbers and the CI clock turns out to have enormous headroom while the human triage budget runs out at around seventy-five cases.

Three budgets, only one of which binds

A prompt regression suite spends three separate budgets, and sizing it means finding which one runs out first.

• Wall-clock time. How long a developer waits between pushing a prompt change and getting a verdict. Past about ten minutes people stop waiting and start merging on a hunch.
• Money. Every case is a real inference call, billed per token, multiplied by how often the suite runs.
• Triage. Every red build costs a person time to look at. Not every red build is a real regression, and the ones that are not are the expensive ones, because they buy nothing.

The first two are the ones teams instinctively optimise. They are almost never what stops you.

Working the wall-clock number

Take a set of assumptions and state each one, because the answer is only as good as they are. Assume a median completion latency of 3.5 seconds per case — that is a number you should replace with your own p50 from production logs, not a figure to take on faith. Assume eight concurrent in-flight requests, chosen to sit under your provider’s token-per-minute limit rather than under your CPU count. Assume a six-minute budget for the whole suite, which is 360 seconds.

rounds        = 360 s / 3.5 s per case  ~= 102
executions    = 102 rounds x 8 concurrent = ~822
usable        = 822 x 0.4 (retries, cold start, CI queueing)
              = ~330 cases

Three hundred and thirty cases inside a six-minute budget, and that is with a deliberately pessimistic 40 per cent utilisation factor to absorb retries and runner start-up. Very few prompt suites are anywhere near three hundred cases. Wall-clock time is not why your suite is small.

The money is similarly undramatic. Assume 1,200 input tokens and 300 output tokens per case — a realistic shape for a case that includes a system prompt, a few-shot block and a short structured answer. At Anthropic’s published Claude Sonnet 5 rate of $2 per million input tokens and $10 per million output tokens, a 300-case run costs 0.36 MTok of input at $0.72 plus 0.09 MTok of output at $0.90, which is $1.62. Five gating runs a week is $8.10. A thousand-case nightly is $5.40 a night, or $37.80 a week. Under fifty dollars a week for a suite most teams would call large.

The constraint that actually binds

Now the third budget. A case that fails for a reason that is not a regression — the model sampled an unusual continuation, a provider hiccuped, a judge disagreed with itself — still stops a build and still costs somebody the walk from the notification to the diff to the conclusion that nothing was wrong. Call that a triage. Set the arithmetic up as an inversion: decide the weekly triage budget first, and let it tell you the suite size.

gating_cases = triage_minutes_per_week
               / ( minutes_per_triage
                   x runs_per_week
                   x spurious_failure_rate )

Assume thirty minutes a week of somebody’s attention for suite maintenance, four minutes to investigate one red case, five gating runs a week (one prompt change per weekday), and a two per cent spurious failure rate per case per run. That last figure is an assumption, not something measured here — it is the number you should estimate from your own history by counting how many red cases in the last month turned out to need no change.

gating_cases = 30 / ( 4 x 5 x 0.02 )
             = 30 / 0.4
             = 75

Seventy-five. Against a wall-clock ceiling of three hundred and thirty and a cost ceiling well past a thousand. The suite is bounded by people, at roughly a quarter of the size the machine budget allows, and every case you add past that point converts directly into someone learning to ignore the build.

Splitting the suite to raise the ceiling

Look at where the spurious rate sits in that formula. It is a multiplier in the denominator, which means it is a lever with far more travel than anything else in the expression. Halve it and the suite doubles. Take it to a tenth and the suite grows tenfold.

You cannot make a model deterministic, but you can choose assertions whose answer does not depend on which continuation was sampled. A schema check either parses or does not. A check that the tool named lookup_order was called either fired or did not. A check that no value matching a credit-card pattern appears in the output is exact. An assertion that the model’s summary “mentions the refund policy” is a judgement, and judgements are where the noise lives.

So split the suite in two. The gating tier holds only assertions with a near-zero spurious rate and runs on every prompt change. Put the spurious rate at 0.2 per cent — the residual is provider errors, not model variance — and the same formula gives 30 / (4 × 5 × 0.002), which is 750 cases. The nightly tier holds the judged cases, the rubric-scored ones and the expensive ones; it runs on a schedule against a tree nobody just changed, and its failures go into a queue rather than onto a pull request.

The practical form of “how big should my suite be” is therefore “how much of it can I state as an invariant”. Moving a case from judged to deterministic is worth roughly ten cases of extra headroom. Organising the suite by failure mode makes that move visible, because each mode has its own assertion shape and its own noise floor.

What a weekly cadence changes

Everything above is a snapshot. A prompt that changes weekly adds a second cost that a one-off sizing exercise misses entirely: cases do not just cost their run, they cost their carrying. Every rewrite of the prompt forces a decision about every case that no longer applies, and at a weekly cadence that decision arrives fifty times a year.

Which gives a usable admission test. A case earns permanent residence if it would still be worth writing after the next rewrite. Cases pinned to phrasing fail that test immediately — rewrite the prompt and they are all wrong at once, and the honest response to fifty simultaneous failures is to delete them, which is what people do, which is how a suite dies in one afternoon. Cases pinned to a contract survive: the output must parse as this schema, the total must equal the sum of the line items, no identifier may appear that was not in the input. Those hold across rewrites because they were never about the words.

Then grow it by event rather than by plan. Add a case when something reaches production — a support ticket is the best source there is — and enforce the ceiling by removal, not by refusing additions. When the gating tier passes its number, the question is which existing case has been green for six months against a prompt it no longer describes. That case has stopped being a test and become a bill.