Documentation — Lesharness

Install

Two commands

Python 3.10 or newer. The only runtime dependency is pyyaml, used for the case format. Everything else — the HTTP client, the report writer, the dashboard server — is standard library.

The self-test runs the real engine against scripted providers with no network access, so it works on a plane and in a locked-down CI container.

shell

# python 3.10+
uv venv --python 3.11 .venv

# installs the package, so the `lesharness` command exists
uv pip install --python .venv -e .

# prove the engine works, offline, no API key
.venv/bin/lesharness selftest

expected

ok   taxonomy has 'instruction_miss'
ok   taxonomy has 'refusal'
ok   truncation detected from finish_reason
ok   gate exits 1 on a collapsed run
SELFTEST PASSED — engine verified offline

Reference

Command line

Eight subcommands. The first three are the ones you will use daily.

Command	What it does	Exits
run	Execute a dataset against a model and prompt version; optionally gate it.	0 / 1 with --gate
gate	Re-evaluate a stored run against current thresholds, without calling the model.	0 / 1
baseline	Promote a stored run to a named baseline that later runs compare against.	0
diff	Compare two stored runs: stable, regressed, fixed, still broken.	0
rubric	Draft a weighted rubric for a new case from a specification.	0
ls	List recorded runs from the registry.	0
serve	Serve the dashboard locally (default port 8787).	0
selftest	Offline proof that the engine works. No network, no keys.	0 / 1

Common flags

--dataset

Path to a case file. YAML, one or more files.

--model

Model identifier to evaluate, passed through to the provider.

--base-url

OpenAI-compatible endpoint. Any gateway, any local server.

--prompt-version

Version of the prompt under test, e.g. v2.

--baseline

Named baseline to compare against. Diff is reported against it.

--gate

Apply the three brakes and exit non-zero on failure.

--promote

Record this run as the new baseline once you trust it.

--concurrency

Parallel case execution. Default 4.

--tag

Restrict the run to cases carrying a tag.

--case

Restrict the run to named cases. Useful when iterating on one failure.

Format

A case is an input, a rubric, and optional checks

The rubric is the part people underestimate. A vague criterion produces a vague score, and a vague score produces a gate that either never fires or fires constantly.

cases/example.yaml

id: triage-new-crypto-rulemaking
input: |
  CFTC Chair Says New Crypto Rulemaking
  (source: The Block)

criteria:
  - id: valid_json
    weight: 3.0
    description: >
      a single parseable JSON object with
      summary, materiality, category, so_what

  - id: summary_grounded
    weight: 3.0
    description: >
      every claim in summary appears in the
      headline; no invented numbers or actors

checks:
  - kind: json
  - kind: one_of
    field: materiality
    values: [low, medium, high]

tags: [regulation, grounding]

Write criteria you could argue about

A criterion is only useful if two reasonable people could disagree on whether it was met. “The output is good” fails that test and produces noise. “Every claim in the summary appears in the input” passes it and produces a signal.

Weights are not percentages. They are relative. A weight-3 criterion contributes three times as much as a weight-1 criterion to the case score. Keeping them as small integers makes the aggregate interpretable without arithmetic.

Checks are free, criteria cost tokens. Anything decidable without judgement belongs in checks. Moving work from the rubric into a check makes runs faster, cheaper, and more deterministic at the same time.

The lesson that cost the most. The first rubric on this dataset scored summary and so_what with the same “no claims beyond the input” criterion. But so_what is the field whose entire job is inference — so every correct output was penalised. The baseline read 38%. The model was fine; the rubric was wrong. Splitting the criterion moved the same model and prompt to 75%.

Providers

Anything OpenAI-compatible

The runner speaks one protocol. Where that protocol is served is your choice: a hosted vendor, a gateway in front of several vendors, or a local inference server.

environment

export EVAL_API_KEY=...         # bearer token
export EVAL_BASE_URL=...        # optional; --base-url wins

# a full run against a local gateway
lesharness run \
  --dataset cases/triage.yaml \
  --model cx/gpt-5.6-luna \
  --base-url http://127.0.0.1:20128/v1 \
  --concurrency 4 \
  --promote

Model and judge are separate

The model under test and the judge are configured independently, and a run records both. Using a stronger model as the judge than the one being evaluated is normal and intended.

Judge cost is tracked per run. Every run reports what the judging cost, so a large dataset cannot quietly become expensive. A 32-case run with a mid-size judge typically lands in the cents.

Transport failures are separated. A provider error is recorded as runtime_error and never counted as a quality regression. Infrastructure noise should not move your quality score.

Transient failures are retried. A 429, a 5xx, or a dropped connection is retried up to three times with exponential backoff, honouring Retry-After when the provider sends it. Permanent failures such as 401 are not retried, because they will not fix themselves. And if every case in a run errors, the gate says nothing was measured rather than reporting a pass-rate drop — a configuration problem should never read as a quality decline.

For the Claude endpoint specifically, what is already wired, what is not, and what a judge swap would cost: see Claude integration.

Verification

The harness is tested like anything else

A test harness that only works when the network works is not a safety net. The self-test drives the real runner, judge, gate, baseline, and diff code paths against scripted providers that return fixed strings.

Offline by construction

No sockets opened, no keys read. Deterministic, so a failure means the engine changed, not that a vendor had a bad minute.

selftest.py

Unit tests on the parts that must not drift

Truncation detection, check evaluation, gate arithmetic, registry round-tripping. 24 tests, no network.

pytest tests -q

Fixture-based judging

The scripted judge reads the criterion ids out of the judging prompt, so the fixtures exercise scoring and aggregation rather than a stub.

ScriptedJudge

Layout

What lives where

lesharness/schema.py

Case, criterion, case result, run result. The data model.

lesharness/cases.py

Loading and validating case files.

lesharness/prompts.py

Versioned prompt store, content hashing, pointer resolution.

lesharness/providers.py

OpenAI-compatible client, retries, cost accounting.

lesharness/checks.py

Deterministic assertions and truncation detection.

lesharness/judges.py

Rubric scoring and the failure-mode taxonomy.

lesharness/runner.py

Execution, concurrency, result assembly.

lesharness/report.py

Markdown and JSON reports, baseline diffing.

lesharness/registry.py

Run history and baseline promotion.

lesharness/gate.py

The three brakes and the exit code.

lesharness/cli.py

The eight subcommands.

lesharness/selftest.py

Offline proof, scripted providers and judge.

Install, run, and wire it into CI