The demo, end to end
Everything on this page is the recorded output of the committed demo: 32 cases, one model, two prompt revisions. Nothing here is illustrative.
The setup
One plausible edit to a working prompt
The dataset is a triage prompt that reads a news headline and returns a structured judgement. Version 1 states the output contract explicitly. Version 2 relaxes it and asks for enthusiasm — the kind of edit made by someone trying to make the output nicer to read.
Return ONE JSON object with exactly these keys: summary one sentence, <= 30 words, no numbers absent from the input materiality one of: low | medium | high category one of: regulation | security | market | infrastructure | other so_what one sentence on why it matters No markdown fences. No prose outside the object.
Write back a short description of the item,
then how important it is, then say why it
matters. Be enthusiastic and punchy so the
output is exciting to read. Keep it brief.
# contract relaxed, format unspecified
The result
Same dataset, same model, one revision apart
| Measure | v1 | v2 |
|---|---|---|
| Cases passed | 24 / 32 | 0 / 32 |
| Pass rate | 75.0% | 0.0% |
| Mean rubric score | 0.978 | 0.400 |
| Regressions | — | 24 |
| Mean score delta | — | -0.578 |
| Exit code | 0 | 1 |
Failure modes, by run
The v2 bar is a single segment: every case broke the same way at once, which points at the output contract rather than at model quality. The expected-failure list in the gate confirms it case by case.
The gate output
What CI actually received
$ lesharness run --prompt-version v2 --baseline main --gate dataset cases/triage.yaml (32 cases) model cx/gpt-5.6-luna judge rubric · 6 criteria · failure taxonomy
run 20261008-111323 24/32 75.0% mean 0.978 exit 0 run 20261008-111525 0/32 0.0% mean 0.400 exit 1 GATE FAIL x pass rate 0.0% below floor 95.0% (32 failing of 32) x 24 regression(s), budget 0: triage-cftc-chair-says-new-crypto-rulemaking triage-ofac-sanctions-tren-de-aragua-atm triage-metamask-staking-exits-lido-validators … 21 more x mean score dropped -0.578 (budget -0.050) dominant failure mode format_violation × 32 report reports/20261008-111525.md
What the judge said
Rubric detail on a case that failed at 94%
The clearest argument for rubric-weighted scoring over a pass/fail threshold. This case scored highly and was still marked wrong.
“The output satisfies the format, grounding, relevance, category, and brevity requirements. However, materiality is somewhat overstated as high because the headline reports losses rather than an event that breaks rules, changes markets, or demonstrates a comparable systemic consequence.”
the rubric agreed with everything the output did, and disagreed with the one judgement it made
What v1 did well
24 of 32 cases passed. The contract held, the schema was respected, and the grounding criterion caught no inventions.
24/32 · mean 0.978What v1 did badly
8 failures spread across three modes: reasoning, instruction-following, and one hallucination. Real quality work, not a collapse.
3 modes · small countsWhat v2 did
Removed all variation. Every case failed identically. A uniform result is a strong signal that the change was structural.
1 mode · ×32Run it yourself in about a minute.
The self-test needs no API key and no network. Point the runner at any OpenAI-compatible endpoint when you want scores.