One adapter, because the harness speaks one protocol

The runner talks the OpenAI chat-completions shape. Claude exposes the same shape at its own endpoint, so pointing the harness at Claude is a configuration change rather than a code change.

The status, stated plainly

Not enabled yet — and here is exactly what that means

Every score on this site was produced with a different model, and the site says so wherever a model is named. The runs carry their model identifier in the report data, and the dashboard prints it. Nothing here claims otherwise, because a regression harness that misreports which model produced a result has no reason to exist.

What is already true is the integration path. The provider layer is a single OpenAI-compatible client, chosen deliberately so that the judge backend is a setting rather than a rewrite. Claude is reachable through it today, with no new dependency:

So this page is a plan with a cost, a command, and a config diff — not a feature list. When the runs are re-judged on Claude, the numbers on the demo page get replaced by the new ones and this banner comes down.

the entire integration, as it stands

# no new dependency, no adapter to write
export EVAL_API_KEY=$ANTHROPIC_API_KEY

lesharness run \
  --dataset   cases/triage.yaml \
  --model     claude-sonnet-5-5 \
  --base-url  https://api.anthropic.com/v1/ \
  --baseline  main \
  --gate

Why the OpenAI-compatible endpoint and not the native Messages API. One protocol means one client, one retry path, one cost model, and one place for a transport bug to live. A native adapter would buy marginal fidelity and cost a second code path that only ever runs in production.

Where Claude goes

Five places in the loop, in the order they matter

The harness has one hard dependency on a model: the judge. Everything else is a convenience, and each of them degrades gracefully if the model is unavailable.

Judge

Ship-blocking. Scores each output against the weighted rubric and returns one failure mode from the fixed taxonomy. This is the core of the product and the first thing to move.

Rubric drafting

Turns a written specification into weighted criteria, so a new case starts from a draft instead of a blank file. Output is reviewed and committed by a human.

Failure classification

Assigns the mode for outputs that never reach the judge, such as malformed JSON caught by the deterministic layer. Cheap, high volume, and the highest token spend after judging.

Patch suggestions

Proposes a prompt revision from a failing case and its judge reasoning. Advisory only — a suggestion is never applied automatically and never counted as a pass.

Dataset curation

Keeps the case set from drifting: spots duplicate cases, over-represented slices, and gaps where a failure mode has no coverage. Runs offline, not in CI.

Why the judge is the one that matters

The rest of the pipeline is deterministic on purpose

Structure checks, truncation detection, and the gate itself run without a model. That is a design commitment, not an oversight: the parts that decide whether a build fails should produce the same answer every time.

Which leaves the judge as the single point where model quality actually changes what the harness concludes. A stronger judge scores more accurately and classifies failures more precisely. A weaker judge produces confident nonsense that gets aggregated into a number someone then trusts. So the judge is exactly the place to spend on the best available model, and the only place where a swap is worth a re-run of the whole dataset.

There is a second, quieter argument. A judge and a model under test drawn from the same family will share failure modes, and a shared blind spot is invisible to a rubric scored by its own relative. Judging across vendors is a deliberate hedge against a correlated error that no amount of prompt tuning would surface.

Cost

What the re-judge would cost

32 cases × 2 prompt versions = 64 judged calls, plus the candidate generation pass. Each call carries a rubric and one short output.

cents, not dollars

Effort

What would change in the code

Two flags and one environment variable. No new module, no adapter, no schema change — the report already records the judge model per run.

config only

Blast radius

What would need re-baselining

Every stored score is model-dependent, so the baseline is re-promoted once and all comparisons are dated from that point forward.

1 re-promote

The honest caveat. A new judge re-scores the same outputs. It does not magically fix the eight cases the current baseline fails — and if it does, that is evidence of a rubric problem to investigate, not a win to promote. A judge that only ever agrees with you is not measuring anything.

Blast radius

What changes if the judge moves

Written out so the swap is a decision rather than an experiment.

Unchanged

The case format, the check kinds, the failure taxonomy, the gate arithmetic, the report schema, the registry, and the CLI. The engine does not care which model is judging.

no code change

Re-run

Every judged score, and therefore the baseline. Past runs stay in the registry as history but stop being comparable, because a score is only meaningful next to the judge that produced it.

1 re-promote

New fields in the report

Two already exist: the judge model per case and the judge cost per run. A cross-vendor judge makes both load-bearing rather than decorative.

already recorded

Read the six stages, then see where a model is actually required.

Only one stage needs a model. The other five produce identical output every time, which is the point.

See how it works Read the docs