evaluatorq¶

Helper function/decorator to create a named job that ensures the job name is preserved even when errors occur during execution.

This wrapper:

• Automatically formats the return value as {"name": ..., "output": ...}
• Attaches the job name to errors for better error tracking
• Can be used as a decorator (@job("name")) or function (job("name", fn))

Parameters:

Returns:

# As a decorator:
@job("text-analyzer")
async def analyze_text(data: DataPoint, row: int):
    return {"length": len(data.inputs["text"])}

# As a function wrapper:
my_job = job("my-job", async_function)

# With lambda for simple cases:
uppercase_job = job("uppercase", lambda data, row: data.inputs["text"].upper())

Creates an evaluator that checks if the output contains the expected output. Uses the data.expected_output from the dataset to compare against.

Parameters:

Returns:

# Basic usage
evaluator = string_contains_evaluator()

# With case-sensitive matching
strict_evaluator = string_contains_evaluator(case_insensitive=False)

# With custom name
my_evaluator = string_contains_evaluator(name="my-contains-check")

Invoke an Orq deployment and return just the text content. This is a convenience wrapper around deployment() for simple use cases.

Parameters:

Returns:

# In a job
@job("my-job")
async def my_job(data, row):
    return await invoke("summarizer", inputs=data.inputs)

Build a pairwise (A-vs-B) LLM jury that reuses the shared panel machinery.

Judges compare two responses and pick a winner ('A'/'B'/'tie'). With swap on (default) each judge is run in both orderings to correct for position bias (see ADR-24). Panel/orchestration params mirror llm_jury. prompt overrides the built-in Mustache-style template (which exposes the response_a.*/response_b.* namespace via _side_to_namespace); leave it None to use the default. Returns a PairwiseComparator; call compare per A/B pair, and roll many comparisons up with evaluatorq.pairwise.build_report.

max_concurrency caps TOTAL in-flight judge LLM calls across all concurrently running compare calls on the returned comparator (each pair fans out judges x orderings x repetitions). None (default) keeps the fan-out unbounded.

assignment="cyclic" (CyclicJudge, arXiv:2603.01865) gives each comparison exactly one judge, cycling through the panel so every judge covers an equal share of the run. Judge bias cancels in expectation over many comparisons at single-judge cost, and the assigned judge still runs both orderings when swap is on. Per-pair winners are single-judge opinions — roll them up with build_report and read run-level rates. The rotation runs over the deduplicated panel (duplicate judge entries add weight only under "all"), repetitions still applies to the single assigned judge, and the cursor lives on the comparator: a reused comparator continues where the previous run stopped, so exact balance holds per freshly built comparator, not per run. Shuffle your pairs first if their order is meaningful. Requires min_successful_judges=1.

Usage:

from evaluatorq import llm_jury_pairwise

comparator = llm_jury_pairwise(
    criteria="The answer is accurate, complete, and directly addresses the question.",
    judges=["anthropic/claude-sonnet-4-6", "openai/gpt-5.4-mini"],
)
comparison = await comparator.compare(
    question="What is the capital of France?",
    response_a="The capital of France is Paris.",
    response_b="The capital of France is Berlin.",
)
print(comparison.winner)

Fit hard BT-sigma over a run's reconciled votes and re-derive winners.

The two "items" are the run's A and B sides; every decisive reconciled vote is one comparison between them. Reconciliation has already symmetrised position bias (both orderings per judge), satisfying the model's commutativity requirement. Categorical votes make hard BT-sigma the natural variant, which is also the paper's most robust one under inconsistency.

Identical votes are collapsed into weighted records before the fit (a judge has at most three distinct judgements here: A, B, tie), so the fit cost stays flat no matter how many comparisons the run holds.

Judges whose decisive votes are unanimous are excluded from the reliability weighting: in the two-item collapse their sigma measures one-sidedness, not reliability (see BTSigmaAggregation). They vote with the median weight of the remaining judges (or uniformly when no judge remains), and fit_warnings names them.

Roll a set of pairwise comparisons up into headline and per-judge metrics.

aggregation='plurality' (default) keeps the existing uniform plurality consensus. aggregation='bt-sigma' additionally fits hard BT-sigma over the run and attaches the reliability-weighted rollup (report.bt_sigma) plus each judge's discriminator (JudgeStats.sigma); the headline plurality rates are unchanged so the two aggregations stay comparable.

Creates an evaluator that checks if the output exactly matches the expected output. Uses the data.expected_output from the dataset to compare against.

Parameters:

Returns:

# Basic usage (case-sensitive)
evaluator = exact_match_evaluator()

# With case-insensitive matching
loose_evaluator = exact_match_evaluator(case_insensitive=True)

Fit the Bradley-Terry family by MLE on the given comparisons.

judge_sigma adds the per-judge discriminator (BT-sigma). hard binarizes preferences first (hard BT / hard BT-sigma) - the paper's most robust variant when judges are highly inconsistent.

Degradations (recorded in warnings, never raised):

• a single judge with judge_sigma=True falls back to plain BT - a lone sigma is absorbed into the skill scale and carries no information (paper section 3.3);
• a tiny ridge keeps disconnected graphs / perfect separation finite.

Per-judge within-datapoint repetition consistency, in [0, 1] (RES-1251).

The unit of analysis is one (judge, comparison, ordering) group: repeated passes of the SAME prompt. A group needs >= 2 decisive repetitions; its consistency is the mean pairwise agreement among them, then discounted by the share of that vote's passes that errored or came back off-contract (repetition_failures) so a flaky judge scores below a clean one; a genuine abstention is not counted as a failure and does not penalise. A comparison contributes at most ONE observation per judge (its groups averaged), so repetition count never multiplies a judge's evidence, and different datapoints are never compared to each other - which is exactly why this is interpretable as judge reliability where the global two-item fit was not.

Empty dict when no judge has any qualifying group (e.g. repetitions=1 or a legacy run without observations).

This is the SHRUNK reliability weight, not raw self-agreement: at the recommended R=2 a group's agreement is only 0.0 or 1.0, so a judge with one lucky group would otherwise carry the same weight as a judge with many. Each judge's mean is shrunk toward the panel mean by its evidence count (empirical-Bayes, _SHRINKAGE_PSEUDO_OBS pseudo-observations), so thin evidence is pulled to neutral and cannot dominate a run. A perfectly self-consistent judge therefore reads below 1.0 unless the whole panel is; repetition_consistency_raw publishes the un-shrunk mean beside it (RES-1251).

Per-judge RAW within-datapoint self-agreement mean (RES-1251).

1.0 = a judge that always agreed with itself on repeated passes of the same prompt. This is PURE agreement, not failure-adjusted and not shrunk: a judge that agreed with itself on every completed pass reads 1.0 even if some passes failed (the failure discount and the shrinkage both live in the repetition_consistency weight, not here). Published beside that weight so the raw signal stays visible. Empty dict when no judge has a qualifying group.

Run a panel over one A-vs-B comparison and reconcile it into a verdict.

Each judge runs through the shared run_jury. When swap is on (default) every judge is also run with A and B exchanged; the two verdicts are un-swapped and passed through reconcile_pair, so a judge that just follows slot order abstains and is recorded as a flip.

Both orderings run concurrently. Replacement judges are promoted at the pair level — a stand-in is run in both orderings so it can cast a real reconciled vote, and a primary that fails in either ordering is what it stands in for. The winner is the plurality of the reconciled votes, or 'inconclusive' when fewer than min_successful_judges cast a decisive reconciled vote.

max_concurrency caps how many judge calls run at once across the whole comparison (judges x orderings x repetitions, replacements included). Pass an existing asyncio.Semaphore to share one budget across several comparisons. None (default) keeps the fan-out unbounded.

Tracing: the whole comparison is ONE orq.pairwise_jury span (RES-985). Each ordering drives _run_jury_core rather than run_jury so it doesn't mint a second jury span whose aggregates describe half a comparison; every judge span hangs off this one, tagged judge.label_swapped.