Six stages, no magic in any of them

Every stage is a file you can read. The ordering is deliberate: cheap deterministic checks run before the judge, so you never spend tokens proving that malformed JSON is malformed.

Golden dataset cases/*.yaml

Your own cases — an input, a weighted rubric, expected values, and tags. Built from what your system actually processed, never invented rows. This one is 32 cases drawn from 8,309 real headlines.

input: the exact payload the system receives
criteria: id, description, weight — the rubric the judge scores against
checks: deterministic assertions evaluated before any judge call
tags: slices for reporting, e.g. grounding, adversarial

Versioned prompts prompts/<id>/vN.txt

A prompt change is a release, so treat the file like one. Never edit in place — add a version and move a pointer. Each run records the version plus the content hash, so two runs are only comparable when the content really matches.

prompts/triage/v1.txt, v2.txt — immutable history
prompts/triage/current.txt — the pointer runs resolve
a changed pointer with an unchanged hash is flagged

Deterministic checks checks.py

JSON shape, required keys, regex, length bounds — free and unambiguous. Plus truncation detection that catches what providers under-report: a length-limit finish reason, or an unbalanced-brace scan for outputs cut off mid-structure.

json · contains · not_contains · regex · one_of
max_words · min_words · length
these run first because they cost no tokens

LLM judge judges.py

Scores each output against the weighted rubric and returns a failure mode from the fixed vocabulary. Free-text “why it failed” is not aggregatable across runs; an enum is. That is what lets you chart hallucination: 4 → 11 this week and act on it.

one score per criterion, weighted into a single case score
one mode from a closed set of 12
reasoning text retained for humans, never parsed

Diff against baseline report.py

Classifies every case as stable, regressed, fixed, or still broken. Emits Markdown for humans and JSON for machines, keyed on output hashes so that unchanged cases are provably unchanged rather than merely similar.

stable · regressed · fixed · still broken
markdown report plus machine-readable JSON
output hashes make “nothing changed” a checkable claim

The gate gate.py

The exit code is the contract. Three independent brakes, plus a warning when the candidate output is byte-identical to the baseline — that means your prompt change never reached the model, which is a false green that looks exactly like a pass.

floor · regression budget · mean-score drop
exit 0 pass, exit 1 fail — consumable by any CI
byte-identical output raises a deployment warning

Why the order matters

Cheap and certain, then expensive and judgemental

Most prompt regressions are not subtle. The output stops being valid JSON, or a required field disappears, or the answer is cut off. Those are decidable without a model, so deciding them with a model is waste — and worse, it is unreliable, because a judge asked to assess malformed input may still score the prose it can read.

The judge is reserved for the part only judgement can settle: whether a summary is grounded, whether a conclusion follows, whether the answer is relevant. Running the deterministic layer first means the judge sees only well-formed outputs, and its budget is spent on real ambiguity.

checks.py — the kinds that run first

# structural
json          parses as a single JSON object
one_of        value is one of an allowed set
regex         matches a pattern

# textual
contains      substring must appear
not_contains  substring must not appear
max_words     brevity ceiling
min_words     substance floor

# transport
length        output length within bounds
unbalanced-brace scan  catches silent truncation

Provider-reported truncation is not enough. Some gateways return finish_reason: stop on an output that was plainly cut off. The brace scan catches those independently, which is how the truncated_response mode stays trustworthy.

Reproducibility

A run is only comparable to another run when the content matches

Runs record the prompt version and the hash of its content, the dataset hash, the model, the judge model, and the git revision. Comparing across different content isn’t blocked — it’s reported as what it is, instead of being silently presented as a regression.

prompt_sha

Content hash of the rendered prompt. Same pointer, different hash, means someone edited in place.

dataset_sha

Content hash of the case file, so two runs are only compared against identical case sets.

output_sha

Per-case output hash. Unchanged cases are provably unchanged, not assumed.

git_sha

Revision recorded at run time, so a score can be traced back to a commit.

Read the six stages as code, not as slides.

The repository is small on purpose. Every stage named above is one module with a single responsibility.

Open the app Read the docs