Pass rate alone hides damage
A change can hold the pass rate flat at 75% while trading three fixed cases for three newly broken ones. That is why there are three brakes and not one.
Absolute floor
Never ship below a minimum pass rate, regardless of what the diff says. This is the brake that catches a wholesale collapse.
Regression budget
How many previously-passing cases are allowed to fail now. Default is zero — a prompt revision that breaks a working case is a regression, not a trade.
Mean-score drop
Catches decay that never crosses the pass threshold. “Still passing, but barely” is a leading indicator of the next outage.
The false green
A pass that proves nothing
The most dangerous gate result is the one that looks healthy. If a prompt edit silently failed to apply — a stale pointer, an environment variable that didn’t propagate, a cache that served the old template — the candidate run will be byte-identical to the baseline and every case will pass.
That reads as a clean bill of health. It is actually evidence that nothing was tested. So the gate compares output hashes against the baseline and raises a warning when the candidate is identical: not a failure, because genuinely identical output is possible, but never a silent pass either.
The same reasoning applies to prompt hashes. A run whose pointer changed while its content hash did not is flagged, because the version you believe you tested is not the version that ran.
$ lesharness run --baseline main --gate PASS 32/32 100.0% → exit 0 $ lesharness run --prompt-version v2 --baseline main --gate FAIL 0/32 0.0% → exit 1 x pass rate 0.0% below floor 95.0% x 24 regression(s), budget 0 x mean score dropped -0.578 (budget -0.050) $ lesharness gate 20261008-111525 # re-evaluate a stored run FAIL unchanged verdict → exit 1
Wiring it in
The gate is a command, not an integration
Anything that can read an exit code can run it. The workflow below is the one shipped in the repository, trimmed to the part that matters.
jobs: selftest: # always runs, no secrets steps: - run: pip install pyyaml - run: python -m lesharness.cli selftest gate: # needs a model endpoint steps: - run: python -m lesharness.cli run --dataset cases/triage.yaml --baseline main --gate # exit 1 fails the build
Two jobs, on purpose
The self-test job needs nothing: no API key, no network, no model. It proves the engine itself works. It should never be skipped, and it should never be the reason a build is flaky.
The gate job needs an endpoint, so it carries a secret and may be skipped on forks. Splitting them means a contributor without credentials still gets a meaningful green check for their change.
On thresholds. The defaults are deliberately strict. A project adopting the harness for the first time will fail the floor immediately — which is the correct outcome, because the first job is to establish where you actually are, promote that as the baseline, and tighten from there.
Three brakes, one exit code.
Run it against the committed demo dataset and watch a plausible prompt edit fail the build in a single run.