LLM regression testing
Ship prompt changes with evidence,not vibes.
When you change a prompt or bump a model version, something that used to work breaks. Lesharness runs your own cases against the change, scores every output against a rubric, and names the failure mode — then exits non-zero so CI refuses the merge.
$ lesharness run --prompt-version v2 --baseline main --gate dataset cases/triage.yaml (32 cases) model cx/gpt-5.6-luna judge rubric · 6 criteria · failure taxonomy
run 20261008-111323 24/32 75.0% mean 0.978 exit 0 run 20261008-111525 0/32 0.0% mean 0.400 exit 1 GATE FAIL x pass rate 0.0% below floor 95.0% (32 failing of 32) x 24 regression(s), budget 0: triage-cftc-chair-says-new-crypto-rulemaking triage-ofac-sanctions-tren-de-aragua-atm triage-metamask-staking-exits-lido-validators … 21 more x mean score dropped -0.578 (budget -0.050) dominant failure mode format_violation × 32 report reports/20261008-111525.md
The problem
Prompt regressions are silent
A prompt is source code with no compiler. You edit a sentence, the output shape shifts, one class of input starts getting a confident but wrong answer — and nothing fails. No stack trace, no type error, no red build. You find out when a customer tells you.
Benchmark suites don’t catch this. They measure how a model ranks against academic tasks, not whether your application still does your job. The unit of testing has to be your own cases.
So the harness is built the way you’d build a test suite: cases versioned in the repository, a runner that executes them, an assertion layer, and an exit code that a CI system understands.
# both runs "succeeded" finish_reason: stop ← 412 chars, answered finish_reason: length ← 31,204 chars, no answer
What it gives you
Four things, each one a file you can read
No hidden orchestration, no service to run. Every stage is a module in the repository, and the interesting decisions are visible in the code.
Answers, not scores
A weighted rubric says how much went wrong. A fixed failure taxonomy says what went wrong, in a vocabulary you can count across runs.
Failure modes →Ordered, cheap first
Deterministic checks — JSON shape, truncation, regex, length — run before the judge. You never spend tokens proving malformed JSON is malformed.
How it works →A gate, not a dashboard
Three independent brakes and an exit code. A dashboard tells you after the fact; a gate stops the merge in the first place.
The gate →Reproducible offline
A self-test drives the real engine with scripted providers and zero network calls. If you can’t trust it during an outage, you can’t trust it.
Self-test →Worked example
A plausible prompt edit, caught in one run
Not a synthetic example. One prompt revision a human would really make — relax the output contract, add “be enthusiastic” — against the same 32 cases and the same model.
| Measure | prompt v1 | prompt v2 |
|---|---|---|
| Cases passed | 24 / 32 | 0 / 32 |
| Pass rate | 75.0% | 0.0% |
| Mean rubric score | 0.978 | 0.400 |
| Regressions vs baseline | — | 24 |
| CI exit code | 0 | 1 |
Exit codes are a contract: 0 the run is usable and passed, 1 the gate failed, and 2 nothing was measured because every case errored. The third one exists so a provider outage can never be mistaken for a passing run in an unattended script.
Full walkthrough →Failure modes, by run
Each bar is all 32 cases, so the widths are directly comparable and the segments are counts of cases, not severity. v1 fails in three different ways; v2 collapses into a single mode covering every case — which is the signal you want, because a uniform result points at the contract rather than at the model.
Every prompt change is a release. Treat it like one.
The app shows the live run registry, the baseline diff, and every case with its rubric detail — no clicking required to see what broke.