LLM judges vs decision models¶
The arena can judge each round two ways. An LLM panel prompts cheap chat models for a verdict and a written reason. A decision panel asks decision models one question through the orq.ai Decisions API: is response A better, response B, or neither? Each answers with a probability per option and no prose.
This page compares the two on real arena rounds and says where each one belongs. Every number here comes from a run made for this page, and the commands that produced it are listed at the end so it can be regenerated. Dated 2026-10-13, evaluatorq 1.55.1.
Short version. On two pools of eight models, the shipped decision panel (jev-latest, pplx-decider-v1-27b, gpt-6-luna) agreed with a strong reference jury better than the LLM trio did on both, decided about a third more rounds, leaned toward longer answers no more than the reference does, and cost about an eleventh as much per round. It loses the written reasons on judge cards. The panel was picked using these same two pools, so the margin is an upper estimate; the second-best panel tried still beat the LLM trio on one pool and tied or trailed it on the other (section 2.2).
1. How the two work¶
LLM panel (judge_api: llm) |
Decision panel (judge_api: decisions) |
|
|---|---|---|
| What the judge is asked | A prompt with both answers and the criteria; it writes JSON {explanation, value} |
One choice question with options A, B, tie; the criteria are the question, both answers are the material |
| What comes back | A verdict and a paragraph of reasoning | A probability for A, B and tie; the vote is the most likely option |
| Calibration | None: a verdict, no strength | Probabilities; native decision models are trained to make them meaningful (section 2.4) |
| Price | Input and output tokens at chat rates; output is the reasoning | Input tokens only, at about $0.035 to $0.10 per million |
| Seat order | Both orders, a flip abstains | The same; the flip rule is unchanged |
| Context | The judge model's window | Model specific and smaller: decider-4b takes 32k, jev-latest 32k of state plus question |
| Failure modes | Off-contract JSON, refusals, a reasoning model running out of tokens | Refusal (counts as abstain), 4xx/5xx; no parse failures, the answer is typed |
| What the reader sees | The judge's reason on every card | Three probability bars per card, no reason |
Both run through evaluatorq's pairwise jury, so stand-in judges, the quorum and the plurality
rule are the same in both modes. Self-judge exclusion is the arena's own step, taken before the
jury is built, so it too is the same in both. The arena pins every judge to the
endpoint judge_api names, because the catalog marks some chat models as decision-capable too
(section 3.1).
2. Measured on the arena¶
Setup¶
- Pool 1, quickstart: the committed
examples/quickstart/battles.jsonl, eight frontier and mid-tier models, 140 rounds, recorded in July with the then-default LLM panel (claude-haiku-4-5, gemini-2.5-flash-lite, gpt-5.4-nano). - Pool 2, budget: two fresh runs of the
configs/budget_8.yamlpool onprompts/starter.jsonl(eight small models, 140 rounds each), one judged live by the LLM panel that file shipped with then (claude-haiku-4-5, gemini-3.5-flash-lite, gpt-5.4-nano) and one live by a decision panel. Each was then re-judged by the other kind of panel. Both configs are kept indocs/research/configs/, sinceconfigs/budget_8.yamlnow ships the decision panel. - Reference jury: there is no human ground truth for these rounds, so each pool was also
re-judged by three strong chat models (grok-4.6, kimi-k3, glm-5.3, both seat orders, about
$10 per pool). "Agreement" below means agreement with this jury on the rounds both decided.
It is a better yardstick than the cheap LLM panel, not a truth. It is independent of pool 1
but not of pool 2:
budget_8seatsxai/grok-4-1-fast, so grok-4.6 judges its own family there, and the pool 2 figures (κ 0.92, Spearman 0.93) may be flattered by it. Re-running pool 2 with kimi-k3 and glm-5.3 alone would settle it. - Decision models tried: typesafe/jev-latest, orq/decider-4b, openai/gpt-6-luna, perplexity/pplx-decider-v1-27b, alibaba/decision-model-preview, berget/Cloudflare/clef-flash, berget/convaiinnovations/laya, berget/Qwen/Qwen3.5-2B: every native decision model the catalog lists (chat models that also have a Decisions endpoint are not included, see section 8).
2.1 Pool 1: quickstart, against the reference jury¶
| Panel | Decided | Ties | Agreement | Cohen's κ | Spearman | Longer answer won | Jury $ per 1k rounds | Seconds per round (p50) |
|---|---|---|---|---|---|---|---|---|
| LLM trio, as recorded | 54% | 3% | 91% | 0.83 | 0.79 | 80% | n/a | n/a |
| LLM trio, same three re-run | 59% | 0% | 90% | 0.79 | 0.81 | 83% | $7.09 | 8.5 |
| Decisions: jev-latest, pplx-decider, gpt-6-luna (shipped) | 78% | 8% | 93% | 0.87 | 0.95 | 67% | $0.65 | 0.5 |
| Decisions: jev-latest, pplx-decider, decider-4b | 80% | 14% | 94% | 0.90 | 0.83 | 72% | $0.41 | 1.1 |
| Decisions: jev-latest, decider-4b, gpt-6-luna (the plan's panel) | 73% | 17% | 94% | 0.89 | 0.95 | 69% | $0.63 | 1.2 |
| Reference jury itself | 79% | 10% | 68% | $74.17 | 94.5 |
Decided is the share of rounds that produced a verdict; the rest were inconclusive and never reach the rating. Agreement and κ are measured on the rounds both the panel and the reference decided. Spearman compares the eight-model ranking with the reference jury's ranking; with eight models one swapped pair moves it by about 0.05, so read differences under 0.1 as noise.
What it says:
- The LLM trio decides barely half the rounds. Its judges flip between seat orders 30% to 50% of the time (gpt-5.4-nano half the time), and a flip throws the vote out. jev-latest and pplx-decider flip 10% to 17% across the two pools; gpt-6-luna about a third of the time.
- On the rounds both decide, every decision panel agrees with the reference at least as well (κ 0.87 to 0.90 against 0.79 to 0.83), while deciding a third more of them.
- The LLM trio favours the longer answer more than the reference does. The longer answer won 80% to 83% of its decided rounds; the reference jury's figure is 68%, the shipped decision panel's 67%. The arena prices length out of the rating separately either way.
- The shipped panel costs about an eleventh as much and answers in half a second.
2.2 Pool 2: budget, two live runs¶
Two fresh runs of configs/budget_8.yaml (gpt-5.4-mini, gemini-3.7-flash, deepseek-flash,
mistral-small, grok-4-1-fast, command-a, qwen3.5-flash, claude-haiku-4-5), 140 rounds each.
Live, as a user would see them. One run judged live by the LLM trio, the other live by the plan's decision panel (jev-latest, decider-4b, gpt-6-luna, pplx-decider as stand-in). The responses differ between the two runs, so this compares whole runs, not verdicts:
| Live run | Rated rounds | Jury $ | Jury seconds per round (p50) | Run wall time | Interval width, top model |
|---|---|---|---|---|---|
| LLM trio | 78 | $0.88 | 3.4 | 14 min 19 s | 1525 to 2164 |
| Decision panel | 106 | $0.08 | 0.34 | 11 min 55 s | 1157 to 1602 |
Against the reference, on the LLM-judged run's responses:
| Panel | Decided | Ties | Agreement | Cohen's κ | Spearman | Longer answer won | Jury $ per 1k rounds | Seconds per round (p50) |
|---|---|---|---|---|---|---|---|---|
| LLM trio, live | 55% | 3% | 94% | 0.89 | 0.69 | 79% | $6.32 | 3.4 |
| Decisions: jev-latest, pplx-decider, gpt-6-luna (shipped) | 77% | 7% | 96% | 0.92 | 0.93 | 69% | $0.59 | 0.6 |
| Decisions: jev-latest, pplx-decider, decider-4b | 70% | 16% | 88% | 0.79 | 0.76 | 83% | $0.37 | 1.4 |
| Reference jury itself | 76% | 8% | 64% | $69.33 | 68.5 |
The second pool is where the choice of the third judge shows: with decider-4b in the third seat the panel trails the LLM trio on agreement (κ 0.79 against 0.89) and leans to the longer answer as much as it does; with gpt-6-luna it leads on every column. decider-4b alone agrees with the reference at κ 0.56 on this pool against 0.71 on the first. Seat-swapped re-judges ran the other way too (the LLM trio over the decision-judged run): the two kinds of panel ranked that run's models with Spearman 0.90 between them.
2.3 Each decision model alone, and panel size¶
Every decision model judged all rounds of both pools once, so any sub-panel can be rebuilt offline from their votes: a judge's vote does not depend on who else sits on the panel.
| Judge | κ alone, pool 1 / pool 2 | Decided, pool 1 / 2 | Flip rate | Picks the first seat | Seconds per round, uncontended |
|---|---|---|---|---|---|
| jev-latest | 0.85 / 0.84 | 84% / 83% | 16% / 17% | 55% / 57% | 0.29 |
| pplx-decider-v1-27b | 0.76 / 0.76 | 86% / 90% | 13% / 10% | 49% / 51% | 0.38 |
| gpt-6-luna | 0.67 / 0.73 | 66% / 64% | 33% / 36% | 65% / 67% | 0.15 |
| clef-flash | 0.71 / 0.67 | 68% / 78% | 32% / 22% | 44% / 45% | 1.6 |
| decider-4b | 0.71 / 0.56 | 70% / 70% | 29% / 30% | 43% / 49% | 0.22 |
| decision-model-preview | 0.45 / 0.57 | 34% / 49% | 65% / 51% | 28% / 33% | n/a |
| laya | 0.11 / 0.01 | 26% / 25% | 73% / 75% | 36% / 29% | n/a |
| Qwen3.5-2B | n/a | 0% / 0% | 100% | 100% | n/a |
Qwen3.5-2B picks whichever answer is shown first, every time, so the flip rule discards all of its votes. laya and decision-model-preview are close to noise on this task. jev-latest and pplx-decider are the two best on both pools and the most self-consistent. gpt-6-luna is weaker alone and leans to the first seat, but its errors fall on different rounds from the other two, which is what a third judge is for.
Panel size, over every sub-panel of the eight (pool 1 / pool 2):
| Judges | Sub-panels | Decided (median) | κ, median |
|---|---|---|---|
| 1 | 8 | 67% / 67% | 0.71 / 0.67 |
| 3 | 56 | 49% / 52% | 0.87 / 0.82 |
| 5 | 56 | 78% / 79% | 0.84 / 0.82 |
| 7 | 8 | 90% / 88% | 0.84 / 0.82 |
Three judges decide fewer rounds than one (the panel needs a plurality, and a flip or a split costs the round), but agree better with the reference (κ 0.87 / 0.82 against 0.71 / 0.67). Five and seven judges decide more rounds again with κ flat around 0.82 to 0.84, so a bigger panel buys coverage, not agreement. The median three-judge decision panel does not beat the LLM trio on pool 2 (κ 0.82 against 0.89); the shipped one does because it was picked for it. Among three-judge panels that decide at least 60% of rounds on both pools and seat no degenerate judge, jev-latest, pplx-decider and gpt-6-luna has the best combined agreement (κ 0.87 and 0.92) and the highest decided share. It ships, with decider-4b as the stand-in (fast, run by orq, only called when a primary fails). gpt-6-luna is an OpenAI model, so the preflight flags the family overlap on pools with OpenAI candidates; the LLM trio overlapped on all three seats.
2.4 Do the probabilities mean anything?¶
Agreement with the reference jury, by how sure the judge said it was (shipped panel, pool 1 then pool 2):
| Judge | under 0.60 | 0.60 to 0.80 | 0.80 to 0.95 | 0.95 and over |
|---|---|---|---|---|
| jev-latest | 72% / 50% | 93% / 92% | 96% / 96% | 95% / 100% |
| pplx-decider-v1-27b | 72% / 69% | 100% / 96% | 100% / 100% | (2 votes) |
| gpt-6-luna | (4 votes) | 62% / 80% | 79% / 67% | 93% / 97% |
For jev-latest and pplx-decider the confidence means something: a vote under 0.60 agrees with the reference half to three quarters of the time, one over 0.60 more than nine times in ten. gpt-6-luna says 0.80 or more on most votes and is right less often than that in the middle band.
That makes judge_abstain_below a real dial (shipped panel, pool 1 / pool 2):
| Floor | Decided | Agreement | κ |
|---|---|---|---|
| off | 78% / 77% | 93% / 96% | 0.87 / 0.92 |
| 0.5 | 69% / 68% | 96% / 95% | 0.92 / 0.91 |
| 0.6 | 48% / 58% | 100% / 99% | 1.00 / 0.97 |
| 0.7 | 40% / 44% | 100% / 98% | 1.00 / 0.97 |
A floor of 0.6 buys near-perfect agreement at the price of a third of the rounds, which widens every confidence interval. The default stays off; raise it when a decided round must be right more than the ranking must be tight.
2.5 Seat swap¶
The plan was to turn seat swapping off for decision panels if their judges flipped under 2% of the time, halving judge calls. They flip 10% to 36%. Swap stays on.
3. What it buys and what it costs the arena¶
Buys. An eleventh of the jury spend. Rounds judged in about half a second instead of three to eight. A third more decided rounds from the same responses, so tighter intervals for the same model spend. A length lean no stronger than the reference jury's. A measured jury cost: decision calls report their own price, so the report's jury card is exact rather than an estimate (prompted judges are now priced per call too). A confidence per vote that can be thresholded.
Costs. The judge cards lose their written reasons; a reader auditing a surprising verdict sees "A 71%, B 18%, tie 11%" instead of why. Criteria nuance is compressed into one question. The Decisions API is in beta and carries no evaluators or guardrails yet.
3.1 Found along the way¶
- The shipped LLM panel had quietly become a mixed one. evaluatorq 1.55.1 sends a judge to
the Decisions API whenever the authenticated catalog marks it decision-capable, and the catalog
marks claude-haiku-4-5 (and gpt-oss, gemini-3.8-flash, glm-5.3-flash and others) that way. From
the evaluatorq 1.55.1 bump until this change, the default panel's haiku seat answered with
probabilities, not a prompted verdict.
judge_apinow decides the endpoint and the arena pins every judge to it. - Decision judges can hit the workspace rate limit. A re-judge at concurrency 8 with three
decision judges and both seat orders wants 48 requests in flight, and a provider behind the
router can answer 429. evaluatorq does not retry a 429: the vote is lost. Since evaluatorq 1.74
it also caps in-flight judge requests at 10 per process. We measured that cap on 2026-10-21
(evaluatorq 1.74.0, arena
3f5a894): the 140-round quickstart log re-judged by this panel (jev-latest, pplx-decider-v1-27b, gpt-6-luna) at--concurrency 8, twice at the default cap and twice with it raised to 48 throughscripts/rejudge_cap.py. Wall time was the same within noise (35 and 48 s at 10, 41 and 44 s at 48). Lost votes were not: at 10, pplx-decider lost 6 and 4 of its 140 votes to 429s; at 48 it lost 57 and 62, and the panel decided 93 and 95 rounds instead of 109 and 113. So the cap costs no time and protects the votes; we keep it. The commands and raw numbers are inrejudge-cap-timing.json. Live runs stay under the cap (generation is the bottleneck), and a re-judge that loses votes can be repeated at a lower--concurrency. - jev-latest's catalog window (64k) is not its limit: state plus question must fit 32k. Prompt plus two capped responses stays well under that on every shipped config.
4. LLM-as-jury in general¶
A decision model is a drop-in judge when the verdict is a choice from a short fixed list (A or B or tie, pass or fail, a label), a yes/no with a probability, or a bounded score, and the material fits its context. Pairwise preference, guardrail checks and rubric levels all fit.
It is not a drop-in when the output has to be a reason (an audit trail, feedback to a writer), when the rubric has many clauses that interact (it gets one question, not a checklist to reason through), when the material is long (multi-turn transcripts or agent traces past 32k to 64k), when judging means computing or quoting (checking arithmetic, citing the faulty line), or when the verdict depends on tool calls and their results that would have to be serialized into the state.
5. Applications: evaluation¶
- Pairwise preference at arena scale, as measured above.
- Pass/fail guardrails and CI gates: a
noulquestion per rule, a threshold on the probability, and the threshold means something because the probability is calibrated. - Rubric scores: a
scorequestion over ordered levels returns a probability-weighted level. - Online sampling: at input-token prices, judging every production call instead of a 1% sample becomes a budget line rather than a project.
- Evaluator alignment: where an LLM judge needs prompt surgery to move its pass rate, a calibrated judge has a threshold to move, and the confidence says which items a human should look at.
6. Applications: data generation¶
Decision models do not generate; they filter and label. A pilot: 192 benchmark prompts generated by gpt-5.4-mini for the four starter categories, 72 of them with a planted flaw (off category, unanswerable, or a near-duplicate of a starter prompt), each labelled usable or not.
| Labeller | Flaws caught | Flags that were flaws | Good prompts kept | Seconds per label | $ per 1k labels |
|---|---|---|---|---|---|
| jev-latest (decision) | 89% | 97% | 98% | 0.26 | $0.025 |
| gpt-6-luna (decision) | 71% | 100% | 100% | 0.14 | $0.044 |
| decider-4b (decision) | 65% | 57% | 71% | 0.11 | $0.013 |
| gemini-3.5-flash-lite (LLM) | 88% | 94% | 97% | 0.56 | $0.105 |
| gpt-5.4-nano (LLM) | 85% | 79% | 87% | 0.58 | $0.075 |
jev-latest matched the better LLM filter at a quarter of the price and twice the speed. Every labeller caught every unanswerable prompt; off-category prompts were the hard case for all of them, and decider-4b missed nearly every near-duplicate. Labels here are the planted intent, not a human check.
Where this goes: rejection sampling of synthetic data, dedup and on-topic gates, preference-pair labelling for DPO (an arena round is exactly a labelled pair, with a confidence), and routing the low-confidence items to people.
7. When to use which¶
| Situation | Use | Why |
|---|---|---|
| Ranking models on your prompts, arena style | Decision panel | Cheaper, faster, more decided rounds, better agreement; which three judges sit on it matters (section 2.2) |
| You must read why a judge decided | LLM panel | Decision models return no reasons |
| Judging long transcripts or agent traces | LLM panel | Decision context is 32k to 64k |
| Rubric with many interacting clauses | LLM panel, or one decision question per clause | One question compresses the rubric |
| Guardrail or CI gate on a clear rule | Decision model | Calibrated probability, typed answer, input-only price |
| Judging all production traffic | Decision model | Price makes 100% sampling affordable |
| Filtering synthetic data | Decision model (jev-latest) | Matched the better LLM filter at a quarter of the cost |
| A judge must not change identity mid-run | Either; the arena pins endpoints |
8. Open questions and follow-ups¶
- Human anchors. The reference here is a strong LLM jury. Fifty human-annotated rounds per
pool (
orq-arena annotate, thenorq-arena anchor) would say how both panels track people. - Soft Bradley-Terry. Every vote now stores its probabilities, so a rating fit on them rather than on hard votes can be tried offline on saved runs.
- evaluatorq: retry 429s with the router's wait hint; expose per-vote usage, so the report can
show each judge's cost; let a caller pass the Decisions API
cacheoption, so a repeated re-judge costs nothing. - Emulated decision models. The catalog's chat models with a Decisions endpoint (haiku, gemini flash, gpt-oss) report probabilities they were not trained to calibrate; they were not measured here.
- Calibration plot in the report, from the data already in
judge_votes.
Reproduce¶
All re-judges read recorded responses, so they cost judge tokens only. Tables come from
scripts/judge_study.py; the pilot from scripts/data_filter_pilot.py.
# Pool 1: reference jury, then each panel (repeat --judge per model)
orq-arena rejudge examples/quickstart/battles.jsonl \
--config docs/research/configs/quickstart_reference_jury.yaml --judge-api llm \
--judge xai/grok-4.6 --judge moonshotai/kimi-k3 --judge zai/glm-5.3 \
--output qs.reference.jsonl
orq-arena rejudge examples/quickstart/battles.jsonl \
--config docs/research/configs/quickstart_decisions.yaml --judge-api decisions \
--judge typesafe/jev-latest --judge perplexity/pplx-decider-v1-27b --judge openai/gpt-6-luna \
--concurrency 3 --output qs.decisions.jsonl # the study ran at 3; see 3.1 on concurrency
uv run python scripts/judge_study.py --ref qs.reference.jsonl \
"recorded LLM panel=examples/quickstart/battles.jsonl" "decisions=qs.decisions.jsonl"
# Pool 2: two live runs, then the same re-judges over b8_llm.jsonl
orq-arena run --config docs/research/configs/budget_8_live_llm.yaml \
--prompts prompts/starter.jsonl -y --output b8_llm.jsonl
orq-arena run --config docs/research/configs/budget_8_live_decisions.yaml \
--prompts prompts/starter.jsonl -y --output b8_dec.jsonl
# Panel sweep: all eight decision models in one panel, then every sub-panel offline
orq-arena rejudge examples/quickstart/battles.jsonl \
--config docs/research/configs/quickstart_decisions.yaml --judge-api decisions \
--judge typesafe/jev-latest --judge orq/decider-4b --judge openai/gpt-6-luna \
--judge perplexity/pplx-decider-v1-27b --judge alibaba/decision-model-preview \
--judge berget/Cloudflare/clef-flash --judge berget/convaiinnovations/laya \
--judge berget/Qwen/Qwen3.5-2B --output qs.all8.jsonl
uv run python scripts/judge_study.py --ref qs.reference.jsonl "all 8=qs.all8.jsonl" --sweep "all 8"
# Data filtering pilot
uv run python scripts/data_filter_pilot.py --out pilot/
A round where a judge call failed for an infrastructure reason (a 429 burst) was re-judged with
the same panel and spliced back (scripts/repair_failed_rounds.py); 1 to 16 rounds per log were
affected.