Skip to content

Evaluation Reference

Everything evaluatorq() accepts, and the patterns built on top of it. If you have not run an evaluation yet, start with Getting Started.

evaluatorq()

async def evaluatorq(
    name: str,
    params: EvaluatorParams | dict[str, Any] | None = None,
    *,
    data: DatasetIdInput | ExperimentInput | TraceInput | Sequence[Awaitable[DataPoint] | DataPointInput] | None = None,
    jobs: list[Job] | None = None,
    evaluators: list[Evaluator] | None = None,
    datapoint_parallelism: int = 10,
    llm_parallelism: int | None = None,
    print_results: bool = True,
    description: str | None = None,
    path: str | None = None,
    inference: bool = True,
    single_trace: bool = False,
    on_datapoint_complete: DataPointComplete | None = None,
) -> EvaluatorqResult
Parameter Type Default Description
data list[DataPoint \| dict] | list[Awaitable[DataPoint]] | DatasetIdInput | ExperimentInput | TraceInput required Data to evaluate — local rows (a DataPoint or a plain dict with the same keys), an Orq dataset, an existing experiment, or recorded trace output
jobs list[Job] | None required when inference=True Jobs to run on each data point; omitted and ignored when inference=False
evaluators list[Evaluator] | None None Evaluators that score job outputs
datapoint_parallelism int (≥1) 10 Number of concurrent datapoints. The former name parallelism still works, deprecated
llm_parallelism int (≥1, or -1) | None None Ceiling on in-flight LLM requests for the whole run. None keeps an enclosing limit or applies the default of 10; -1 adds no cap and disables the default only at the top level
print_results bool True Display the progress and results table
description str | None None Optional evaluation description
path str | None None Path for organizing results on the Orq dashboard (e.g. "Project/Category")
inference bool True Run the jobs; set False to score data that already has outputs
single_trace bool False Open one span for the whole run so every row shares a trace — see One trace per run
on_datapoint_complete DataPointComplete | None None Callback run once per data point as it finishes — see Reacting to each row as it finishes

Parameters can also be passed positionally as an EvaluatorParams model or a plain dict — the three forms below are equivalent:

await evaluatorq("my-eval", data=[...], jobs=[...], datapoint_parallelism=5)
await evaluatorq("my-eval", {"data": [...], "jobs": [...], "datapoint_parallelism": 5})
await evaluatorq("my-eval", EvaluatorParams(data=[...], jobs=[...], datapoint_parallelism=5))

Full type signatures live in the API Reference.

Jobs

The @job() decorator

@job() names a job. The name shows up in the results table, in traces, and as JobResult.job_name, which is how you tell which job failed:

from evaluatorq import job

@job("risky-job")
async def risky_operation(data: DataPoint, row: int):
    return await potentially_failing_operation(data)

# A raise is recorded as job_name="risky-job", error="<error details>".
# Without @job the error text is the same, but job_name is the generic "job".

It also wraps plain callables, which is handy for one-liners:

uppercase_job = job("uppercase", lambda data, row: data.inputs["text"].upper())
word_count_job = job("word-count", lambda data, row: len(data.inputs["text"].split()))

Multiple jobs per data point

Every job runs against every data point, which is how you compare variants (two prompts, two models, preprocessing on and off) on identical inputs:

await evaluatorq(
    "multi-job-eval",
    data=[...],
    jobs=[preprocessor, analyzer, transformer],
    evaluators=[...],
)

What a job can return

A job's return value is its output, typed as Output:

Output = str | int | float | bool | dict[str, Any] | list[Message] | AgentResponse | None

A job that produces structured data can return a dict instead of flattening it into a string. Scorers receive it unchanged as params["output"], so they can read fields directly:

@job("extract")
async def extract(data: DataPoint, _row: int) -> dict:
    return {"city": data.inputs["text"].split()[-1], "confidence": 0.9}


async def city_correct(params: ScorerParameter) -> EvaluationResult:
    output = params["output"]  # the dict above, not a string
    return EvaluationResult(value=output["city"] == params["data"].expected_output)

On the way out (the Orq upload and JSON result dumps) a dict output is serialized with json.dumps, so the platform shows it as a JSON string. The exception is a dict with "object": "response", which is an OpenResponses payload and is sent as-is. Pydantic models inside the dict are dumped to dicts first; anything else that is not JSON-serializable becomes its str().

Reporting a failure the job handled

A job that lets a failure raise needs nothing: process_job records it and the row counts as failed. A job that catches its failure to keep the rest of the batch alive — a target that answered 401, a simulation the runner ended in error — must say so, because a returned dict looks like a clean run:

async def resilient_job(data: DataPoint, row: int) -> dict:
    try:
        answer = await call_my_agent(data.inputs["text"])
    except MyAgentError as exc:
        # Keeps the partial output for diagnosis, and still fails the row.
        return {"name": "my-agent", "output": None, "error": str(exc)}
    return {"name": "my-agent", "output": answer, "error": None}

Emit error on every path, None on success: an omitted key and a clean run are indistinguishable, so a job that forgets the key on one branch reports a dead target as a passing one. A row with a non-empty error keeps its output for diagnosis, but its evaluators are skipped — scoring a transcript you already know is dead buys nothing and costs an LLM judge call per row. Where a failure lives covers how it is counted and how to gate CI on it.

This is the raw-dict job contract. @job() wraps a function's return value into {"name", "output"}, so an error key returned from a decorated function lands inside output, where a judge reads it as the target's failure rather than the runner reading it as the row's. A decorated job reports a row failure by raising.

Calling an Orq deployment from a job

When the thing you are evaluating is an Orq deployment, call it from inside the job. evaluatorq.deployment wraps the Orq SDK in two async functions: deployment() returns a DeploymentResponse carrying both content (the extracted text) and raw (the untouched SDK response), and invoke() is the same call returning just the text. One Orq client is created lazily on first use and reused for the process.

This needs ORQ_API_KEY. A missing key raises ValueError at the first call, naming the variable.

from evaluatorq import DataPoint, job
from evaluatorq.deployment import invoke


@job("orq-deployment-job")
async def my_job(data: DataPoint, _row: int) -> str:
    return await invoke("my-deployment", inputs=data.inputs)

inputs fills the deployment's template variables, so a summarizer whose prompt references {{text}} takes inputs={"text": "Long article..."}.

For a chat-style deployment, pass messages — and reach for deployment() when you want the raw response as well:

from evaluatorq.deployment import deployment

response = await deployment("chatbot", messages=[{"role": "user", "content": "Hello!"}])
print(response.content)  # extracted text
print(response.raw)      # full SDK response object

thread={"id": "conversation-123"} groups several calls into one conversation on the platform. context (routing attributes) and metadata are also accepted — see deployment() in the API reference for the full signature.

To red-team or simulate that same deployment rather than evaluate it, name it as a target instead: Targets › Orq-hosted.

Data sources

data accepts inline DataPoints, an Orq dataset, or awaitables that resolve to DataPoints — the last of which lets you stream rows in from a slow source without blocking the run:

async def get_data_point(i: int) -> DataPoint:
    await asyncio.sleep(0.01)  # e.g. a network fetch
    return DataPoint(inputs={"value": i})

await evaluatorq(
    "async-eval",
    data=[get_data_point(i) for i in range(1000)],
    jobs=[...],
)

For Orq-hosted datasets, pass data=DatasetIdInput(dataset_id="..."). That path requires ORQ_API_KEY — see Configuration. To score responses a past Orq experiment already produced, rather than generating new ones, pass ExperimentInput — see Replaying a past experiment.

Replaying a past experiment

Sometimes you want to score responses that an Orq experiment already produced instead of generating fresh ones — to try new evaluators against a past run, or to re-grade without paying for another round of generation. That is what no-inference mode does: pass inference=False and evaluators run against the recorded response in each row rather than calling any job.

The response source is chosen by the data argument to evaluatorq():

data value What it loads
list[DataPoint] In-memory datapoints.
DatasetIdInput(dataset_id=...) Rows from an Orq dataset (you supply or generate the responses).
ExperimentInput(experiment_id=..., run_id=...) The recorded responses from a past experiment run. Requires inference=False.

ExperimentInput sits alongside DatasetIdInput in the data union — it is not a dataset, it is a completed experiment run whose outputs get replayed.

Finding the IDs

Both IDs are read off the Orq UI:

  • experiment_id — the ID in the experiment URL, /experiments/<experiment_id>. The same ID appears in the /v3/experiments/<id> API routes.
  • run_id — optional. Every execution of an experiment creates a new run. Open a run from the experiment's run history to read its ID from the URL, or list them with GET /v3/experiments/<id>/runs. Omit it to replay the latest run.

Example

from evaluatorq import ExperimentInput, evaluatorq


async def run():
    await evaluatorq(
        "replay-past-experiment",
        data=ExperimentInput(experiment_id="<experiment_id>"),  # latest run
        evaluators=[my_evaluator],
        inference=False,
    )

Pin a specific run with run_id:

data=ExperimentInput(experiment_id="<experiment_id>", run_id="<run_id>")

ORQ_API_KEY must be set — the recorded rows are fetched from the Orq API. When inference=False, jobs is optional and ignored. Any row whose recorded response is missing or blank fails loudly rather than being silently skipped.

Evaluate recorded trace output

TraceInput is the async request for recorded trace data, and it has two mutually exclusive modes. Query mode selects a bounded batch by limit, time bounds (start_time/end_time), search text, or filters — the mode you reach for when you do not already have a trace ID. Trace mode takes one trace_id, with an optional exact span_id; span_id cannot be used without its trace, and limit cannot be combined with trace_id because trace mode never reads it. The shared importer returns normalized Trace objects and Trace.to_datapoint() supplies the evaluatorq row shape.

Query mode is the common path — score whatever ran recently without looking up an ID first:

from datetime import datetime, timedelta, timezone

from evaluatorq import TraceInput, evaluatorq, orq_evaluator

results = await evaluatorq(
    name='production-quality',
    data=TraceInput(
        limit=50,
        start_time=datetime.now(timezone.utc) - timedelta(days=1),
        search='checkout',
    ),
    evaluators=[orq_evaluator(evaluator_id='eval-1')],
)

filters is a list[dict[str, Any]] passed straight through to the platform's advanced trace filters — the same shape the Orq Traces UI produces. Each entry takes a field (the attr. prefix over the response's attributes path), an op, and values as a list of strings, even for a single value:

data=TraceInput(limit=50, filters=[{"field": "attr.orq.billing.cache_read_cost", "op": "gt", "values": ["0"]}])

Once you have a specific trace — from the Orq UI's trace URL, or orq traces search --from 24h --to now --query checkout -o json — trace mode replaces the whole query:

results = await evaluatorq(
    name='production-quality',
    data=TraceInput(trace_id='trace_123', span_id='span_456'),
    evaluators=[orq_evaluator(evaluator_id='eval-1')],
)

Evaluatorq skips jobs and sends the trace's recorded output to your evaluators — TraceInput resolves inference=False on its own, so there is no need to pass it. An exact span is parsed on its own; a trace or query selection starts from the latest eligible non-evaluator span and follows parents until it finds messages. Chat Completions, Responses, and OpenTelemetry GenAI messages are accepted from top-level span input/output, flat or nested span attributes, and span event attributes. A query that matches no traces logs a warning, and evaluatorq() raises rather than running a green zero-row evaluation.

orq_evaluator needs the Orq SDK, which the evaluatorq[orq] extra installs (uv add "evaluatorq[orq]") — a base evaluatorq install already carries it, but name the extra explicitly rather than relying on that. evaluator_id is required and keyword-only. It identifies an evaluator that already exists in Orq and determines the evaluatorq result name (orq:eval-1 when model is omitted, orq:eval-1@<model> when you set one); there is no separate name= override. model= is optional and overrides the configured model only for model-backed evaluators; deterministic built-ins do not need it. A missing trace response is an error, not a clean score, so check the returned results before trusting a rate.

orq_evaluator also scores agent simulation runs, for example evaluatorq(..., jobs=[wrap_simulation_agent(...)], evaluators=[orq_evaluator(evaluator_id='eval-1')]). When the datapoint inputs have no messages and no string query, it reads the conversation from the simulation output: the final simulated-customer turn is sent as query, every earlier turn as messages, and the agent's final turn, tool calls included, as output. When the simulated conversation ends on a customer turn (error or timeout), output is empty because the agent never replied.

Built-in evaluators

from evaluatorq import exact_match_evaluator, string_contains_evaluator

string_contains_evaluator()                        # case-insensitive by default
string_contains_evaluator(case_insensitive=False)  # case-sensitive
string_contains_evaluator(name="my-contains-check")  # custom name in the table
exact_match_evaluator()                            # case-sensitive by default

Both compare the job output against the data point's expected_output. For LLM-graded evaluators see LLM as a Jury; for structured, multi-dimensional scores see Structured Results.

Custom evaluators

An evaluator is a {"name": ..., "scorer": ...} pair whose scorer receives the data point and the job output and returns a score:

async def accuracy_scorer(params):
    data, output = params["data"], params["output"]
    score = calculate_score(output, data.expected_output)
    return {"value": score, "explanation": "High accuracy match" if score > 0.8 else "Partial match"}


await evaluatorq(
    "dataset-evaluation",
    data=DatasetIdInput(dataset_id="your-dataset-id"),
    jobs=[processor],
    evaluators=[{"name": "accuracy", "scorer": accuracy_scorer}],
)

Where a failure lives

evaluatorq() returns one DataPointResult per row. Each holds a job_results list of JobResult, one per job, and each of those holds an evaluator_scores list of EvaluatorScore, one per evaluator. A failure is recorded in the error field at the level where it happened, and the levels above it stay None. Anything that reads results — a CI gate, a per-row callback, your own report — has to check all three levels, or it reports a broken row as healthy.

Field Set when What is left Summary table row
DataPointResult.error the row never reached its jobs — an awaitable datapoint that raised job_results is None, and if the awaitable never resolved, data_point.inputs is a {"row_index": N} placeholder instead of your own keys Failed Data Points
JobResult.error the job raised, or reported its own failure its evaluators are skipped, so evaluator_scores is empty; output is None if the job raised and kept if it reported the failure Failed Jobs
EvaluatorScore.error the job succeeded but an evaluator could not produce a verdict the job's output is intact; that score's value is empty and its pass_ is None none — it is listed under Errors in the printed report and left out of that evaluator's average

The last one is the quiet one: a dead judge leaves the row and the job both reporting success, and no summary count includes it. check_pass_failures(results, treat_errors_as_failure=True) reads all three fields — see Pass/fail and CI.

Pass/fail and CI

An evaluator that returns pass_ turns the run into a gate:

async def quality_scorer(params):
    score = calculate_quality(params["output"])
    return {
        "value": score,
        "pass_": score >= 0.8,
        "explanation": f"Quality score: {score}",
    }

When any evaluator returns pass_: False, evaluatorq() returns the results; the library never exits the process. To make a script a CI gate, inspect the results and exit explicitly:

from evaluatorq.evaluatorq import check_pass_failures

results = await evaluatorq(...)
if check_pass_failures(results, treat_errors_as_failure=True):
    raise SystemExit(1)

treat_errors_as_failure=True also gates on rows that errored, at any of the three levels in Where a failure lives. It defaults to False, which gates on evaluator pass_ alone, so a run whose target was dead throughout can pass a gate that leaves it off.

The results table gains a pass rate row — Pass Rate | 75% (3/4).

Controlling the run

Parallelism

await evaluatorq("parallel-eval", data=[...], jobs=[...], datapoint_parallelism=10)

datapoint_parallelism counts tasks, and the bounds nest: at most datapoint_parallelism datapoints run at once, and within each one a separate budget of the same size covers its jobs and then its evaluators. Ten datapoints each running ten evaluators is a hundred concurrent tasks, not ten.

Bounding LLM requests

Against a provider concurrency limit, size the request ceiling instead:

await evaluatorq("bounded-eval", data=[...], jobs=[...], llm_parallelism=20)

Unset, the ceiling is 10 concurrent requests, and that default also covers calls made outside evaluatorq(), such as a standalone run_pairwise(). Two unconfigured runs in the same event loop share that default cap of 10, protecting the provider from their combined load; a run with an explicit llm_parallelism= has its own budget. Pass -1 at the top level to disable the default ceiling. To bound code that is not an entry point, wrap it in async with evaluatorq.llm_concurrency_limit(n):. Nested limits stack: a block set to 5 inside a run set to 10 has a ceiling of 5, while a block set to 20 or -1 inside that run remains under the run's ceiling of 10.

This counts requests, not tasks, so it holds however the fan-out nests. It is a concurrency bound rather than a rate limit — ten slots against 10s calls is about 60 requests/minute, but the same ten slots become 300/minute if the provider speeds up to 2s.

Requests evaluatorq issues itself (judges, juries, simulation agents, the red-team pipeline) take a slot automatically. A job that calls a provider SDK directly is invisible to the budget unless you wrap it:

from evaluatorq import llm_slot

async def my_job(data_point, row_index):
    async with llm_slot():
        response = await client.chat.completions.create(...)
    return {"name": "my-job", "output": response.choices[0].message.content}

Wrap only the request — holding a slot across parsing shrinks the budget without reducing load on the provider.

red_team(), simulate(), generate_and_simulate() and generate() take the same argument, with the same meaning.

Reacting to each row as it finishes

on_datapoint_complete is a callback evaluatorq() invokes once per data point, as soon as that row's jobs and evaluators are done. Reach for it when the run is long enough that waiting for the returned list is too late — you want each row streamed into a log, a queue, or your own UI, or you want a failing row to surface while the run is still going. Skip it when you only need the finished list; the returned results carry everything the callback sees.

Running the example below with ORQ_API_KEY set uploads a four-row experiment named ticket-triage to your workspace, as any evaluatorq() run does. Unset the key to run it locally and upload nothing.

import asyncio

from evaluatorq import (
    DataPoint,
    DataPointComplete,
    DataPointResult,
    evaluatorq,
    exact_match_evaluator,
    job,
)


@job("classify")
async def classify(data: DataPoint, _row: int) -> str:
    ticket = data.inputs["ticket"].lower()
    if "refund" in ticket:
        return "billing"
    if "broken" in ticket:
        return "hardware"
    raise RuntimeError("no rule matched this ticket")


async def unavailable_row() -> DataPoint:
    raise RuntimeError("ticket store timed out")


def row_finished(result: DataPointResult) -> None:
    ticket = result.data_point.inputs.get("ticket")
    if result.error is not None:
        print(f"row {result.data_point.inputs.get('row_index')} dropped: {result.error}")
        return
    for job_result in result.job_results or []:
        if job_result.error is not None:
            print(f"{ticket} | {job_result.job_name} failed: {job_result.error}")
            continue
        for score in job_result.evaluator_scores or []:
            if score.error is not None:
                print(f"{ticket} | {score.evaluator_name} could not score: {score.error}")
            else:
                print(f"{ticket} | {score.evaluator_name}={score.score.value} pass={score.score.pass_}")


sink: DataPointComplete = row_finished


async def main() -> None:
    data = [
        DataPoint(inputs={"ticket": "I want a refund"}, expected_output="billing"),
        DataPoint(inputs={"ticket": "My device is broken"}, expected_output="hardware"),
        DataPoint(inputs={"ticket": "Where is my order?"}, expected_output="shipping"),
        unavailable_row(),
    ]
    await evaluatorq(
        "ticket-triage",
        data=data,
        jobs=[classify],
        evaluators=[exact_match_evaluator()],
        on_datapoint_complete=sink,
        print_results=False,
    )


if __name__ == "__main__":
    asyncio.run(main())

The callback receives the row's DataPointResult — the same object the returned list holds, carrying data_point and a job_results list whose entries have job_name, output, error and evaluator_scores. DataPointComplete is the callback's own type, exported from evaluatorq, which is what the example's sink annotation uses.

Agent simulation has a hook of the same name, SimulationHooks.on_datapoint_complete, and it is a different thing: it fires per simulated conversation and receives a SimulationResult. See Hooks receive failed evaluator outcomes for that one.

A plain def and an async def are both accepted, and the choice matters as soon as the callback does something slow. An awaitable is awaited before the row releases its slot in datapoint_parallelism, so an async def that waits on a slow sink costs that row its slot and nothing more. A plain def runs inline on the event loop, so a blocking call inside one stalls every other row as well: five rows at datapoint_parallelism=5 with a 0.2 s wait in the callback spend 0.2 s on callbacks in total under async def, and 1.0 s under a blocking def. Use async def for anything that waits, or hand the work off and return.

Rows run concurrently under datapoint_parallelism, which is 10 unless you set it, so the callback fires in completion order, which is not row order unless that budget is 1 — the example above reports its four rows in whichever order they finish. Read result.data_point to tell which row arrived. The returned list stays in row order.

A callback that checks only result.error misses most failures. A row's failure can sit at any of three levels, and a dead judge leaves both the row and the job reporting success; Where a failure lives lists the three fields. The example's row_finished reads all of them.

In the example, unavailable_row() sets the row-level error, the Where is my order? ticket sets the job-level one, and the two matched tickets come through with their exact-match score.

An exception from the callback does not stop the run. evaluatorq logs it at ERROR level with its traceback, as on_datapoint_complete raised; the run continues, and moves on to the next row. Every row still runs, is returned and is uploaded, whatever the data source. The cost is that a callback failure shows up only in that log line: if the callback's work must not be lost, such as a sink that has to receive every row, catch and record the failure inside the callback yourself.

Only an Orq dataset (DatasetIdInput) streams. Its rows arrive in batches, and the callback still fires per row, when that row finishes rather than when its batch does. Every other source — inline rows, ExperimentInput, TraceInput — is resolved up front and runs from memory. Watching the batched path for yourself takes a live Orq dataset, because nothing local stands in for it.

The example passes print_results=False so the callback's output is the only thing on screen — see Suppressing terminal output. It does not stop the upload: a run still sends its results to Orq whenever ORQ_API_KEY is set.

Organizing results on Orq

await evaluatorq(
    "my-evaluation",
    data=[...],
    jobs=[...],
    path="MyProject/Evaluations/Unit Tests",
)

path groups runs in the Orq dashboard — e.g. "Team/Sprint-42/Feature-X".

Documenting a run

await evaluatorq(
    "model-comparison",
    description="Compare GPT-4o vs Claude on customer support responses",
    data=[...],
    jobs=[...],
)

Suppressing terminal output

results = await evaluatorq("silent-eval", data=[...], jobs=[...], print_results=False)

for result in results:
    for job_result in result.job_results or []:
        print(f"{job_result.job_name}: {job_result.output}")

Where to next