A score tells you something broke. This tells you what.

The taxonomy is fixed in code, so failures stay comparable across runs and across models. It is the part that turns a number into a decision.

format_violation

Broke the required output shape — JSON, markdown, or plain text.

What it usually means: Make the contract explicit in the prompt, or parse tolerantly. If it is JSON, say so and show a shape.

hallucination

Stated something not supported by the input.

What it usually means: Ground the prompt in “use only the provided text” and add a negative case that punishes invention.

truncated_response

Cut off by a token or length limit.

What it usually means: Raise the ceiling, or shorten the task. Check for an unbalanced-brace scan hit if the provider reports a clean stop.

incomplete

Stopped early or left the task unfinished.

What it usually means: Usually an instruction ordering problem. Ask for the last field first, or split the task.

instruction_miss

Ignored an explicit instruction in the prompt.

What it usually means: Instructions compete. Move the ignored one earlier, or make it structural rather than advisory.

reasoning_error

Conclusion does not follow from the given facts.

What it usually means: The most interesting mode. The output is well-formed and plausible, and still wrong.

off_topic

Answered a different question than the one asked.

What it usually means: Often a retrieval or context-assembly bug rather than a prompt bug.

refusal

Declined the task.

What it usually means: Check for an accidental safety-adjacent trigger, or an ambiguity the model resolved conservatively.

verbosity

Correct, but far longer than the task needed.

What it usually means: Set an explicit length. Brevity instructions without a number are not instructions.

empty_response

Produced no usable content.

What it usually means: Usually transport or a hard filter. Confirm against the provider response before blaming the prompt.

runtime_error

Provider or transport failure — not a model-quality failure.

What it usually means: Never counted as a quality regression. Separated deliberately so infrastructure noise doesn’t move the score.

unknown

Unclassified. Kept honest rather than forced into a bucket.

What it usually means: A growing unknown count is the signal to extend the taxonomy, not to guess.

Why a closed vocabulary

“Hallucination: 4 → 11” is actionable. “Quality got worse” is not.

An LLM judge asked for free-text criticism will produce something different every run. That text is genuinely useful to read once. It is useless as a metric, because you cannot count it, chart it, or compare it across two prompt versions without reading everything again.

So the judge returns two things: a weighted score along the rubric, and a mode drawn from a closed set. The score tells you the magnitude of a change. The mode tells you the kind. Both are numbers, both aggregate, and the free-text reasoning is still retained for the case you actually want to investigate.

Keeping unknown in the vocabulary matters as much as the other eleven. A taxonomy with no escape hatch gets abused: real failures get shoved into the nearest bucket to avoid an error, and the distribution stops meaning anything. A growing unknown rate is a measurable signal that the taxonomy needs work — which is the point.

“The output satisfies the format, grounding, relevance, category, and brevity requirements. However, materiality is somewhat overstated as high because the headline reports losses rather than an event that breaks rules, changes markets, or demonstrates a comparable systemic consequence.”

judge verdict · case triage-certik-tallies-7664-million-in-stolen-funds
failure mode reasoning_error · score 0.94 — a case that still failed the rubric at 94% of its weighted points

Why that example matters. A pass/fail threshold on the aggregate score would have waved this through. The rubric found it, and the taxonomy gave it a name — which is the difference between a number you report and a fact you can act on.

Reading a distribution

One mode dominating is a different bug from five modes appearing

A single mode, suddenly

Almost always structural: a contract changed, a field was renamed, a length limit was hit. Fix is usually mechanical and obvious once named.

format_violation ×32

Many modes, slowly

The hard case. Quality drifting across several dimensions at once, none of them crossing a pass threshold. This is what a mean-score brake is for.

4 modes · small counts

Runtime errors

Not a quality signal at all. Separated out so that a flaky gateway cannot masquerade as a prompt regression and send you chasing the wrong fix. Transient failures are retried before they are ever recorded; if every case errors, the run reports that nothing was measured and exits non-zero rather than reporting a pass-rate drop.

runtime_error · excluded

One caveat worth knowing before you trust a number

Model endpoints are not deterministic, including at temperature 0. Measured on this harness: three identical requests to the same endpoint at temperature 0 returned three different completions, and a --repeats 3 run over six cases flagged 6 of 6 as flaky.

That is a property of the models, not a defect in the harness — but it decides how you read a result. A single sample per case cannot separate a real regression from sampling noise, so a gated run with no repeats now warns that its regression verdict may be noise. If a number matters, run it again with --repeats 3 and look at the flakiness count before you act on it.

See the taxonomy applied to a real dataset.

The worked example shows all four modes a single prompt revision can introduce, with the raw gate output.

See the demo Read the docs