evaluatorq¶

Helper function/decorator to create a named job that ensures the job name is preserved even when errors occur during execution.

This wrapper:

• Automatically formats the return value as {"name": ..., "output": ...}
• Attaches the job name to errors for better error tracking
• Can be used as a decorator (@job("name")) or function (job("name", fn))

Parameters:

Returns:

# As a decorator:
@job("text-analyzer")
async def analyze_text(data: DataPoint, row: int):
    return {"length": len(data.inputs["text"])}

# As a function wrapper:
my_job = job("my-job", async_function)

# With lambda for simple cases:
uppercase_job = job("uppercase", lambda data, row: data.inputs["text"].upper())

Creates an evaluator that checks if the output contains the expected output. Uses the data.expected_output from the dataset to compare against.

Parameters:

Returns:

# Basic usage
evaluator = string_contains_evaluator()

# With case-sensitive matching
strict_evaluator = string_contains_evaluator(case_insensitive=False)

# With custom name
my_evaluator = string_contains_evaluator(name="my-contains-check")

Invoke an Orq deployment and return just the text content. This is a convenience wrapper around deployment() for simple use cases.

Parameters:

Returns:

# In a job
@job("my-job")
async def my_job(data, row):
    return await invoke("summarizer", inputs=data.inputs)

Build a pairwise (A-vs-B) LLM jury that reuses the shared panel machinery.

Judges compare two responses and pick a winner ('A'/'B'/'tie'). With swap on (default) each judge is run in both orderings to correct for position bias (see ADR-24). Panel/orchestration params mirror llm_jury. prompt overrides the built-in Mustache-style template (which exposes the response_a.*/response_b.* namespace via _side_to_namespace); leave it None to use the default. Returns a PairwiseComparator; call compare per A/B pair, and roll many comparisons up with evaluatorq.pairwise.build_report.

A classify judge (typesafe/jev-latest) is seated the same way it is on a pointwise panel: it answers an A/B/tie choice question whose question is criteria and whose material is every placeholder the template renders except criteria itself. state_fields narrows that material to the paths you name. Position-bias swapping is unchanged — the question is rebuilt per ordering, so each call says which response sits in which seat.

Concurrency is bounded by the run-scoped LLM ceiling (see common.llm_limit): every judge call routes through common.llm_call and takes a slot, so judge fan-out across concurrent compare calls is capped run-wide.

assignment="cyclic" (CyclicJudge, arXiv:2603.01865) gives each comparison exactly one judge, cycling through the panel so every judge covers an equal share of the run. Judge bias cancels in expectation over many comparisons at single-judge cost, and the assigned judge still runs both orderings when swap is on. Per-pair winners are single-judge opinions — roll them up with build_report and read run-level rates. The rotation runs over the deduplicated panel (duplicate judge entries add weight only under "all"), repetitions still applies to the single assigned judge, and the cursor lives on the comparator: a reused comparator continues where the previous run stopped, so exact balance holds per freshly built comparator, not per run. Shuffle your pairs first if their order is meaningful. Requires min_successful_judges=1.

Usage:

from evaluatorq import llm_jury_pairwise

comparator = llm_jury_pairwise(
    criteria="The answer is accurate, complete, and directly addresses the question.",
    judges=["anthropic/claude-sonnet-4-6", "openai/gpt-5.6-luna"],
)
comparison = await comparator.compare(
    question="What is the capital of France?",
    response_a="The capital of France is Paris.",
    response_b="The capital of France is Berlin.",
)
print(comparison.winner)

Fit hard BT-sigma over a run's reconciled votes and re-derive winners.

The two "items" are the run's A and B sides; every decisive reconciled vote is one comparison between them. Reconciliation has already symmetrised position bias (both orderings per judge), satisfying the model's commutativity requirement. Categorical votes make hard BT-sigma the natural variant, which is also the paper's most robust one under inconsistency.

Identical votes are collapsed into weighted records before the fit (a judge has at most three distinct judgements here: A, B, tie), so the fit cost stays flat no matter how many comparisons the run holds.

Judges whose decisive votes are unanimous are excluded from the reliability weighting: in the two-item collapse their sigma measures one-sidedness, not reliability (see BTSigmaAggregation). They vote with the median weight of the remaining judges (or uniformly when no judge remains), and fit_warnings names them.

Roll a set of pairwise comparisons up into headline and per-judge metrics.

aggregation='plurality' (default) keeps the existing uniform plurality consensus. aggregation='bt-sigma' additionally fits hard BT-sigma over the run and attaches the reliability-weighted rollup (report.bt_sigma) plus each judge's discriminator (JudgeStats.sigma); the headline plurality rates are unchanged so the two aggregations stay comparable.

JudgeStats.consistency / consistency_raw are filled under BOTH aggregations: they are read straight off the recorded repetition observations and need no Bradley-Terry fit, so gating them on bt-sigma would report "no usable repeats" for a plurality run that has them.

Creates an evaluator that checks if the output exactly matches the expected output. Uses the data.expected_output from the dataset to compare against.

Parameters:

Returns:

# Basic usage (case-sensitive)
evaluator = exact_match_evaluator()

# With case-insensitive matching
loose_evaluator = exact_match_evaluator(case_insensitive=True)

Fetch Orq traces and normalize each one to a canonical trace.

One unreadable trace becomes a Trace with import_error set; other traces in the batch remain available. No trace is silently discarded. Partition the result with partition_traces rather than filtering by hand — every surface is meant to report import failures the same way.

The HTTP calls retry through common.retry.with_retry — including the 429s and 5xx that raise_for_status raises as httpx.HTTPStatusError — and the shared httpx.AsyncClient adds no retry layer of its own. Raw httpx rather than the Orq SDK because traces.search requires a from_/to window that TraceInput does not, and its span models would need a second parser.

Fit the Bradley-Terry family by MLE on the given comparisons.

judge_sigma adds the per-judge discriminator (BT-sigma). hard binarizes preferences first (hard BT / hard BT-sigma) - the paper's most robust variant when judges are highly inconsistent.

Degradations (recorded in warnings, never raised):

• a single judge with judge_sigma=True falls back to plain BT - a lone sigma is absorbed into the skill scale and carries no information (paper section 3.3);
• a tiny ridge keeps disconnected graphs / perfect separation finite.

Look a preset up by name, listing the alternatives when there is no match.

Hold one slot from every active finite LLM budget.

Wrap only the request itself — holding a slot across parsing or judging shrinks the budget without reducing load on the provider.

Create an evaluatorq scorer backed by an Orq evaluator.

evaluator_id is required and keyword-only. It identifies an evaluator that already exists in Orq. The optional model is an invocation-time override for LLM-based evaluators; it is not needed for deterministic built-in evaluators, and a configured LLM evaluator can use its Orq model when this is omitted. When omitted, model is not sent to the SDK at all.

The evaluatorq result name is derived, never hand-named, from everything that varies which judge actually runs: orq:<evaluator_id> when model is omitted, orq:<evaluator_id>@<model> when it is set. This keeps two orq_evaluator calls that differ only by model — the standard way to compare judges — distinguishable in the results table and the uploaded experiment; a single fixed orq:<evaluator_id> name would have merged them into one indistinguishable stream.

client accepts an already-built Orq SDK client (or a test double shaped like one). When omitted, one is resolved from api_key or ORQ_API_KEY — and base_url or ORQ_BASE_URL, for a self-hosted deployment — the first time the scorer actually runs, and reused for every later datapoint scored by this same orq_evaluator(...) call. Resolving lazily — inside the scorer, not at factory time — preserves the property that constructing an evaluator that is never invoked does no work and raises nothing; caching that resolution after the first call keeps evaluate()'s default datapoint_parallelism (10 concurrent rows, or more across a large run) from building one Orq(...) client and one unclosed connection pool per row, and from deferring a missing ORQ_API_KEY or missing [orq] extra until after every row has already run and billed.

The scorer sends the latest user turn as query, evaluatorq's job output as output, expected_output as the optional human/reference answer, and preserves prior assistant/tool trajectory messages. Orq invocation trace IDs remain in EvaluationResult.raw_output for local use; evaluatorq's Experiment uploader intentionally strips raw evaluator output. The Orq SDK owns retries for this call; evaluatorq does not add another retry layer.

Per-judge within-datapoint repetition consistency, in [0, 1] (RES-1251).

The unit of analysis is one (judge, comparison, ordering) group: repeated passes of the SAME prompt. A group needs >= 2 decisive repetitions; its consistency is the mean pairwise agreement among them, then discounted by the share of that vote's passes that errored or came back off-contract (repetition_failures) so a flaky judge scores below a clean one; a genuine abstention is not counted as a failure and does not penalise. A comparison contributes at most ONE observation per judge (its groups averaged), so repetition count never multiplies a judge's evidence, and different datapoints are never compared to each other - which is exactly why this is interpretable as judge reliability where the global two-item fit was not.

Empty dict when no judge has any qualifying group (e.g. repetitions=1 or a legacy run without observations).

This is the SHRUNK reliability weight, not raw self-agreement: at the recommended R=2 a group's agreement is only 0.0 or 1.0, so a judge with one lucky group would otherwise carry the same weight as a judge with many. Each judge's mean is shrunk toward the panel mean by its evidence count (empirical-Bayes, _SHRINKAGE_PSEUDO_OBS pseudo-observations), so thin evidence is pulled to neutral and cannot dominate a run. A perfectly self-consistent judge therefore reads below 1.0 unless the whole panel is; repetition_consistency_raw publishes the un-shrunk mean beside it (RES-1251).

Per-judge RAW within-datapoint self-agreement mean (RES-1251).

1.0 = a judge that always agreed with itself on repeated passes of the same prompt. This is PURE agreement, not failure-adjusted and not shrunk: a judge that agreed with itself on every completed pass reads 1.0 even if some passes failed (the failure discount and the shrinkage both live in the repetition_consistency weight, not here). Published beside that weight so the raw signal stays visible. Empty dict when no judge has a qualifying group.

Call /classify once for all questions in request and validate the reply.

The caller owns retry policy. This helper owns the classify span, timeout, wire response validation, and mapping provider failures into ClassifyOutcome.

Run a panel over one A-vs-B comparison and reconcile it into a verdict.

Each judge runs through the shared run_jury. When swap is on (default) every judge is also run with A and B exchanged; the two verdicts are un-swapped and passed through reconcile_pair, so a judge that just follows slot order abstains and is recorded as a flip.

Both orderings run concurrently. Replacement judges are promoted at the pair level — a stand-in is run in both orderings so it can cast a real reconciled vote, and a primary that fails in either ordering is what it stands in for. The winner is the plurality of the reconciled votes, or 'inconclusive' when fewer than min_successful_judges cast a decisive reconciled vote.

Concurrency is bounded by the run-scoped LLM ceiling (see common.llm_limit): every judge call routes through common.llm_call and takes a slot, so both orderings and the replacement pass share one run-wide cap rather than a per-comparison one.

Tracing: the whole comparison is ONE orq.pairwise_jury span (RES-985). Each ordering drives _run_jury_core rather than run_jury so it doesn't mint a second jury span whose aggregates describe half a comparison; every judge span hangs off this one, tagged judge.label_swapped.