Methodology¶
How orq-arena turns judged pairwise rounds into a ranking you can audit. This page is the overview of the principles; the implementation itself is open source and is the full detail.
How it works¶
For every round, one prompt, two candidates, in a round-robin tournament:
- Both candidates answer the same prompt through the same orq.ai router gateway.
- An evaluatorq jury compares the two responses. By default every judge sees the pair
twice, once in each seat order (
judge_swap: falseshows it once); a judge that contradicts itself between orders abstains rather than being trusted on a coin flip. - If fewer than two judges cast a decisive vote, the round is inconclusive and never reaches the rating.
- Every judged round (win, loss, or tie) feeds a Bradley-Terry fit over all rounds so far.
- The fit produces one rating per candidate with a bootstrap 95% confidence interval, plus, when the jury's length preference is statistically identified, a length-controlled rating that prices that preference out.
- The run also reports how much the judges agreed with each other and how often each one flipped between seat orders, public evidence of how much to trust the verdicts.
Judging¶
Both seat orders, by default. LLM judges measurably favor whichever answer they see first (Zheng et al., 2023), so every judge scores each pair twice with the sides swapped:
flowchart TD
o1["verdict in seat order 1 (A,B)"] --> rec{"same verdict<br/>both orders?"}
o2["verdict in seat order 2 (B,A)"] --> rec
rec -- yes --> floor{"judge_abstain_below set and<br/>top probability under it?"}
rec -- no --> flip["abstain, flipped=True"]
rec -- "a call failed" --> fail["abstain, not a flip"]
floor -- yes --> low["abstain, not a flip"]
floor -- no --> vote["reconciled vote"]
vote --> quorum{"≥ 2 decisive votes?"}
quorum -- no --> inc["inconclusive,<br/>never reaches the rating"]
quorum -- yes --> plur{"strict plurality<br/>among them?"}
plur -- yes --> verdict["round verdict: A / B / tie"]
plur -- no --> inc
judge_swap: false sends one order only: half the judge calls, no flip rule, and position
bias goes unmeasured. A judge that gives a different verdict in each order abstains for that round, and the flip is
recorded. A round needs at least two decisive votes (configurable) to produce a verdict, and
that quorum is necessary but not sufficient: the surviving votes still have to reach a strict
plurality. Two decisive votes that disagree one-all produce no plurality, so the round is
dropped from the rating rather than decided by a jury that could not agree with itself.
judge_abstain_below (decision panels only, off by default) adds one more way to abstain: a
reconciled vote whose winning option has a lower probability than the floor is set aside, the
same as a flip, and the quorum and plurality are then counted without it.
Self-preference. LLM judges recognize and favor their own family's prose (Panickssery et al., 2024). A judge that is also a contestant is excluded from its own matches, and the preflight warns whenever a judge shares a provider family with any candidate. The clean setup is a jury drawn entirely from families outside the pool.
All judges score the same free-text criteria for the whole run (configurable, and swappable
after the fact with rejudge).
Two kinds of judge. With judge_api: llm each judge is a chat model prompted for a
verdict and a written reason. With judge_api: decisions each judge is a decision model asked
one question through the orq.ai Decisions API: is A better, B better, or neither? It answers
with a probability for each option and no prose; the vote is the most likely option, and its
probability is the judge's confidence. Everything above holds for both: both seat orders, the
flip rule, the quorum and the plurality. A decision model bills input tokens only, which is
why a decision jury costs a fraction of a prompted one. How the two compare on real arena
rounds is in LLM judges vs decision models.
Ratings¶
Bradley-Terry, not incremental Elo. Ratings come from a Bradley-Terry fit (Bradley & Terry, 1952) over every judged round, refit as the tournament progresses; ties count as half a win for each side. This is the same statistical core as Chatbot Arena's leaderboard (Chiang et al., 2024). Ratings are anchored so the field's average sits at 1000. Each model also plays one virtual tie against a fixed 1000-rated anchor. Without it, a model that loses (or wins) every round has no finite rating: on a small run it printed as exactly -3000. One tie pulls the extremes of a small run toward the middle; with many rounds per model it barely moves anything. (In statistical terms the fit is a posterior mode with a weak prior, not a pure maximum-likelihood fit.)
Confidence intervals. A 1000-resample bootstrap gives each candidate a 95% CI. On a small run the intervals are wide and overlapping; that is the honest output. Those intervals are marginal, so read one at a time: two of them overlapping does not mean the two models are tied, or even that they are indistinguishable. Both are drawn from the same resamples and share the anchoring, so they move together.
Separation. Whether the top two actually differ is a question about their difference, bootstrapped within each resample: the top spot is separated when that difference interval excludes 0. When it does not, the run reports the measured gap and how often the leader came out ahead ("ahead in 71% of resamples") rather than declaring a tie, which the data cannot support either way.
Length control. Seat-swapping fixes position bias, not verbosity bias: a jury that likes
longer answers likes them in both orders. The headline claim is deliberately a plain count,
how often the longer answer (in characters) won a decisive round, because a count has no
estimator behind it and nothing to converge. Behind it, following length-controlled
AlpacaEval (Dubois et al., 2024) and LMArena's style control, the fit is repeated with a
length term: a joint logistic fit (Newton with a line search, stopped on the gradient norm,
never on an iteration budget) yields the jury's length coefficient, and a second rating
column reports each model with that preference priced out (length-adj. on the report page,
len-ctrl in the live TUI). The coefficient and the column appear only when the preference
is identified: the fit converged and a bootstrap interval over the same rows excludes 0.
Resamples where one side always wins have no finite coefficient; they are counted as
separated and widen the interval rather than being dropped. The length fit has no virtual
tie: when every round has the same length ratio, a tie would decide the coefficient on its
own. So when a model wins or loses every round, that fit has no finite answer, and the column
stays hidden even though the main rating is finite. A big gap between a model's two
ratings means verbosity, not quality, is doing the separating. Length is the only style axis
controlled today; markdown formatting is not.
Per-category slices. Ratings are also refit per prompt category, but only for categories with enough comparisons to mean anything; thin slices are dropped instead of shown with misleading precision.
Fixed settings. These are constants, not config keys: 1000 bootstrap resamples for the rating intervals and 400 for the length coefficient, a category slice needs 20 judged comparisons, a model enters a rater's human-anchor ranking with at least 3 qualifying rounds, and the run's schedule, prompt draws and resamples use seed 42.
Auditing the jury¶
Every run reports the numbers needed to challenge its own ranking:
- Agreement: mean modal-vote share per round, plus chance-corrected Fleiss' and Cohen's kappa with the standard Landis-Koch labels (poor to almost perfect). Fleiss' kappa counts only the rounds where every primary judge cast a decisive vote, and the report says how many that was; Cohen's kappa uses each pair's rounds where both were decisive. Votes from stand-in judges are left out of both.
- Per-judge flip rates: how often each judge contradicted itself between seat orders, published per judge rather than hidden in an average.
- Jury swap:
rejudgere-scores the recorded responses under a different panel (judge tokens only, zero regeneration) and reports the Spearman rank correlation against the original ranking, the direct test that the ranking is not an artifact of which judges were picked.
Reliability¶
- Failed streams void the round. One retry per side; a second failure voids the round, which is logged and shown but never judged or rated. The rating reflects the candidates' words, never the network's mood.
- Timeouts measure silence, not duration. A stream only dies after a long gap with no new chunk, so a model that thinks for minutes before its first token is not penalized.
- Truncation is judged, not hidden. A response cut off by its token cap is judged as-is and flagged visibly; the jury sees exactly what a reader would see.
- A voided round keeps what it received. The partial response is recorded alongside the error rather than discarded. The round is still never judged and never rated; the text is there so a failure can be diagnosed from the log instead of guessed at.
Reproducibility¶
Every run is seeded (schedule and prompt draws) and writes a manifest next to the battle log with the config it ran, content hashes of that config and the prompt set, the prompts path, the host the run actually reached, the judge panel, and the closing agreement stats, so two runs are provably comparable and any rating can be re-derived from its log. The prompt hash covers each prompt's category as well as its text, so two banks that differ only in how they label prompts are not mistaken for the same set.
Rounds are written to the battle log as they resolve, not batched to the end of a match, so a run
interrupted partway keeps every round it already paid for. Each record carries the full router id
of both models alongside the short display name, because short names collide across providers
(openai/gpt-oss-120b and groq/gpt-oss-120b). Every view the run and its report produce is keyed
on that full id, not just the leaderboard: the rating, the length-controlled rating, verbosity, cost
and speed, so two providers of one model can never be apart in one panel and merged in another.
When two candidates would share a display name, both fall back to their full ids and the preflight
says so.
orq-arena report <log> rebuilds from that manifest, not from the live YAML: model names, the
judge panel and the reasoning flags come from the run itself, so regenerating a page after the
config moved on cannot relabel a model or compute κ against a panel that never judged the
run. Runs recorded before the full config was stored still pin all three from the manifest's
candidate and judge entries. When identity has to come from a config file instead (no readable
manifest, or an explicit --config that disagrees with the recorded config_sha256), the page
says so in its Confidence stats, so a forwarded report carries the caveat with it.
Human anchor: does the panel agree with people?¶
The jury metrics above measure the panel against itself; the human-anchor workflow measures
it against people. annotate renders a recorded run into a blind annotation page (no model
names, no jury votes, sides swapped per round) you can send to raters; anchor merges their
votes back and reports each rater's kappa against the panel majority and the rank correlation
between the human ranking and the panel's. Both rankings behind that correlation are fit on
the rounds the rater actually voted: models below a minimum comparison count are left out
rather than seated at the default rating, and the model count prints beside every
correlation. Usage: cli.md.
Lessons from The Leaderboard Illusion¶
In 2025 a study of Chatbot Arena, the best-known public LLM leaderboard, found that its rules let a few big labs look better than they were (Singh et al., 2025). Arena disputed parts of the account. The study closes with five fixes. orq-arena is a private arena you run yourself, not a public leaderboard, but the same traps apply, so here is where each one stands:
| What the study found | What orq-arena does |
|---|---|
| Labs tested many versions in secret and hid the scores they did not like. | Every round is written to the battle log the moment it ends, including failed and inconclusive ones. The report covers the whole pool; there is no option to drop a model from it. |
| Some labs got far more secret tests than others. | Nobody submits models to your arena. You choose the pool. |
| Models were quietly removed, mostly open ones, which skews the ratings of the rest. | The pool is fixed for the whole run and every pair of models meets, so no comparison is missing. |
| Some models were shown to voters far more often. | Every pair is judged on the same number of prompts, in a seeded random order. |
| Outsiders could not see what was tested, removed, or how often. | The battle log, the run manifest and the report hold every answer, every judge vote and how much the judges agreed. |
| Labs trained on Arena data to climb the board. | You rank on your own prompts, which no lab has trained on. |
One trap no tool can close for you: running again (a new seed, or rejudge with another
panel) until you like the result, then sharing only that run. If you publish a ranking, publish
every run behind it. Two smaller notes: each pair is judged on its own random draw of prompts
when max_rounds is below the size of your prompt set (the preflight says so), and pairs are
not sampled adaptively, since a full round-robin already compares every pair equally.
Current limitations¶
- The shipped 30-prompt bank is a smoke test, sized to exercise every mechanism, not to defend a ranking. A ranking you intend to defend takes your own prompt set (hundreds of rounds) and judges from families outside the pool.
References¶
- Bradley & Terry (1952). Rank Analysis of Incomplete Block Designs: The Method of Paired Comparisons. doi:10.2307/2334029
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685
- Chiang et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132
- Dubois et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475
- Panickssery et al. (2024). LLM Evaluators Recognize and Favor Their Own Generations. arXiv:2404.13076
- Singh et al. (2025). The Leaderboard Illusion. arXiv:2504.20879