Tracing¶

evaluatorq ships optional OpenTelemetry tracing. When enabled, every evaluation run, job, evaluator, and LLM call becomes a span you can view in the Orq dashboard or any OTLP-compatible backend.

How tracing is enabled¶

Tracing initialises lazily on the first evaluation run. It turns on automatically when either condition is true:

• ORQ_API_KEY is set — the OTLP base endpoint is https://my.orq.ai/v2/otel (or <ORQ_BASE_URL>/v2/otel if ORQ_BASE_URL is set); the exporter appends /v1/traces, so spans POST to …/v2/otel/v1/traces.
• OTEL_EXPORTER_OTLP_ENDPOINT is set — that endpoint is used as the OTLP base.

If neither variable is set, no tracer is created and all span context managers are no-ops.

Set ORQ_DISABLE_TRACING=1 or ORQ_DISABLE_TRACING=true to suppress tracing even when the above variables are present.

Install the OTEL packages¶

Tracing depends on optional packages that are not installed by default:

uv add opentelemetry-api opentelemetry-sdk \
    opentelemetry-exporter-otlp-proto-http \
    opentelemetry-semantic-conventions
# or via the extras bundle:
uv add "evaluatorq[otel]"

Prefer pip? Use python -m pip install "evaluatorq[otel]", which installs into the interpreter you just named rather than whichever pip happens to be first on your PATH.

If these packages are absent the SDK silently skips initialisation — no error is raised.

Minimal enable example¶

import os
import asyncio

os.environ["ORQ_API_KEY"] = "your_orq_api_key"   # tracing auto-enables

from evaluatorq import DataPoint, evaluatorq, job, string_contains_evaluator


@job("echo")
async def echo_job(data: DataPoint, _row: int) -> str:
    return str(data.inputs.get("text", ""))


asyncio.run(
    evaluatorq(
        "my-eval",
        data=[DataPoint(inputs={"text": "hello"}, expected_output="hello")],
        jobs=[echo_job],
        evaluators=[string_contains_evaluator()],
    )
)

To send traces to a custom OTLP endpoint instead:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 uv run my_eval.py

To debug tracing setup:

ORQ_DEBUG=1 uv run my_eval.py

This prints the resolved endpoint, auth header presence, and any initialisation errors to stdout.

OTLP exporter details¶

• Protocol: HTTP/protobuf (OTLPSpanExporter from opentelemetry-exporter-otlp-proto-http)
• Export mode: BatchSpanProcessor (asynchronous batching)
• Timeout: 5 seconds per export request
• Auth: Authorization: Bearer <ORQ_API_KEY> is added automatically when the resolved endpoint's hostname ends in .orq.ai or is exactly orq.ai. For any other endpoint the header is not added; use OTEL_EXPORTER_OTLP_HEADERS to supply auth manually.
• Custom headers: parsed from OTEL_EXPORTER_OTLP_HEADERS as key1=value1,key2=value2.

Span hierarchy¶

Evaluation runner spans¶

orq.job                          # one per DataPoint — root when no ambient trace is active,
  ├── <your job code>            #   otherwise a child of the caller's span
  └── orq.evaluation             # one per evaluator applied to this job

All orq.job spans from a single evaluatorq() call share the same orq.run_id attribute, which ties them together as a logical run without requiring a common parent span. Because there is no common parent, though, an N-row run arrives as N separate traces — one rooted at each orq.job.

One trace per run: single_trace=True¶

Pass single_trace=True to bracket the whole run in one evaluatorq.run span, so every row lands in a single trace:

await evaluatorq("my-eval", data=rows, jobs=[my_job], single_trace=True)

evaluatorq.run                   # one per evaluatorq() call — the root
  └── orq.job                    # one per DataPoint, now a child rather than a root
      └── orq.evaluation

It defaults to False so existing traces keep their shape. Red teaming and simulation do not need the flag — they already open their own root spans (Evaluatorq - Red Teaming / Evaluatorq - Agent Simulation), and orq.job nests under those.

Span attributes on evaluatorq.run:

Span attributes on orq.job:

Span attributes on orq.evaluation:

Red teaming spans¶

Evaluatorq - Red Teaming         # root — one per red_team() call
  ├── orq.redteam.context_retrieval
  ├── orq.redteam.datapoint_generation
  │     ├── orq.redteam.capability_classification
  │     │     ├── chat (llm_purpose=classify_tools)
  │     │     └── chat (llm_purpose=infer_resources)
  │     └── orq.redteam.strategy_planning
  │           └── chat (llm_purpose=generate_strategies)
  ├── orq.job                    # one per attack datapoint
  │     └── orq.redteam.attack
  │           ├── orq.redteam.target_call
  │           └── orq.redteam.attack_turn  (x N turns)
  │                 ├── orq.redteam.adversarial_generation
  │                 │     └── chat (llm_purpose=adversarial)
  │                 └── orq.redteam.target_call
  ├── orq.evaluation             # security evaluator result
  │     └── orq.redteam.security_evaluation
  │           └── chat (llm_purpose=evaluation)
  └── orq.redteam.memory_cleanup # post-run agent memory entity cleanup (only when cleanup is enabled, entities exist, and the target has configured memory stores)

LLM spans (chat ...) carry standard GenAI attributes:

The root Evaluatorq - Red Teaming span additionally carries:

Simulation spans¶

Evaluatorq - Agent Simulation    # root — one per simulate() / generate_and_simulate() call
  ├── chat/responses {model}     # persona/scenario generation calls
  ├── orq.simulation.first_message_generation   # ONE span for the whole persona x scenario sweep
  │     └── chat/responses {model} (x N pairs)
  └── orq.simulation.run         # one per datapoint
        ├── orq.simulation.first_message_generation   # only when no first message was pre-generated
        │     └── chat/responses {model} (orq.llm.purpose="first_message")
        └── orq.simulation.turn  (x N turns)
              ├── orq.simulation.target_call           # calls the agent under test; no span attrs of its own
              ├── orq.simulation.judge_evaluation
              │     └── chat/responses {model} (orq.llm.purpose="judge")
              └── orq.simulation.user_simulator_call
                    └── chat/responses {model} (orq.llm.purpose="user_simulator")

orq.simulation.generate          # root — one per standalone generate() call
  └── chat/responses {model}     # persona/scenario/first-message generation calls

generate_personas() and generate_scenarios() don't open a synthetic root span when invoked standalone. They do create orq.simulation.persona_generation / orq.simulation.scenario_generation spans around their LLM calls. Those generation spans carry the active run metadata when called inside an outer simulation or red-team scope; standalone helpers intentionally have no synthetic run id to stamp.

Span attributes on Evaluatorq - Agent Simulation / orq.simulation.generate:

Span attributes on orq.simulation.run:

Span attributes on orq.simulation.first_message_generation:

Under the root (one span covering the whole persona x scenario sweep):

Under orq.simulation.run (only when a datapoint carried no pre-generated first message):

Span attributes on orq.simulation.turn:

orq.simulation.target_call, orq.simulation.judge_evaluation, and orq.simulation.user_simulator_call carry no span attributes of their own — they exist purely to scope the nested LLM call (and, for target_call, the target's own input/output recording). LLM spans nested under judge_evaluation and user_simulator_call carry the same GenAI attributes as the red teaming LLM spans above, tagged via orq.llm.purpose.

Judge-panel spans¶

llm_jury(), run_jury(), and run_pairwise() (src/evaluatorq/common/jury.py, src/evaluatorq/pairwise.py) run a panel of judges under a shared span hierarchy:

orq.evaluation {evaluator}         # from the core runner, when a jury backs an evaluator
  └── orq.jury                     # one per deliberation (orq.pairwise_jury in comparative mode)
        └── orq.judge              # one per judge (x2 in comparative mode — see below)
              └── chat {model}     # the judge's own LLM call(s), tagged orq.llm.purpose="judge"

The panel opens no span of its own outside orq.jury — it can equally be called standalone (not nested under orq.evaluation), in which case orq.jury is the root. All jury/judge spans are opened via evaluatorq.common.tracing's with_span(), so — like every other span in this document — they are a no-op when tracing is disabled; verdicts and aggregation are unaffected either way.

A judge whose call failed leaves its orq.judge span with OTel status ERROR (via set_span_error), but the failure is swallowed at the panel level — the jury carries on with whatever judges succeeded (or promotes a replacement) and the parent orq.jury span stays OK.

Span attributes on orq.judge:

No token usage or cost here: those are recorded once, on the chat spans underneath, and rolled up by the consumer. Stamping them on every ancestor as well made the same tokens appear three times in one trace.

judge.label_swapped is only ever set (True/False) in comparative mode — in plain run_jury() deliberations it is absent, since each judge votes once.

Span attributes on orq.jury:

pairwise_plurality is a reported value, not an accepted argument — you cannot pass it to aggregator=, and validate_aggregator() rejects it. It names the rule run_pairwise() applies internally: pairwise_consensus(), a strict plurality over reconciled pair votes, run after judges that flipped across the two orderings have already been dropped to abstentions. The six aggregator= keywords reduce raw per-judge votes instead, so labelling this one mode would name it after a function it does not call.

Comparative (pairwise) mode¶

run_pairwise() compares two responses (A vs. B) and, to control for position bias, runs every judge in both label orderings. This changes the span shape from the plain jury case:

• One orq.pairwise_jury span covers the whole comparison — both orderings drive the same span rather than each minting its own; run_pairwise calls the internal _run_jury_core directly (not run_jury) so it doesn't open a second jury span per ordering.
• Each judge appears twice under that one orq.pairwise_jury span — one orq.judge span per ordering, distinguished by judge.label_swapped (False for the A/B ordering, True for the swapped B/A ordering).
• The span is named orq.pairwise_jury, not orq.jury — it aggregates reconciled pair votes rather than raw per-judge votes, so it gets its own name rather than masquerading as a plain jury. Its attributes stay in the jury.* namespace, plus these comparative-only extras:

judge.verdict is already un-swapped in comparative mode. The labels a judge returns there name a position, not a response: a judge that picks the same response both times says A in one ordering and B in the other, which reads as a self-contradiction and is in fact the opposite, a perfectly consistent judge. label_swapped=True spans are mapped back to the canonical frame before the attribute is written, so "how often did this judge pick response A" is answerable from judge.verdict alone, with no join against judge.label_swapped.

There is deliberately no raw-frame twin. The text the verdict was parsed from is one level down, on the chat child's gen_ai.output.messages, so a second attribute here would only restate what the trace already holds — the same reasoning as the alias removal noted above.

Un-swapping is per-ordering and needs nothing but label_swapped. Flip detection is what needs both orderings, and it stays on the parent — jury.flipped_judges names the judges that really did follow slot order.

jury.flipped counts judges that answered in both orderings but disagreed with themselves — that is position bias, not a failure, so a flipped judge is deliberately excluded from jury.judges_failed: judges_failed counts only judges with no reconciled vote and no flip (i.e. one or both orderings raised an error). A judge can be flipped, failed, or a normal decisive vote, but never counted under more than one of those buckets.

orq.evaluation, orq.jury / orq.pairwise_jury, orq.judge, and the nested chat {model} LLM spans follow the ambient OTel context — nothing threads an explicit parent across the orq.evaluation → orq.jury seam, so a jury backing a custom evaluator's scorer nests correctly without extra plumbing.

Run correlation¶

Every LLM invocation issued during a red_team() or simulation run (simulate(), generate_and_simulate(), or generate()) is tagged so an operator can filter Orq's trace UI down to exactly the model calls belonging to one run. The same metadata is inherited by generate_personas() and generate_scenarios() when they are called inside an outer simulation or red-team scope; standalone calls have no synthetic root run id.

A companion key rides the same rail: evaluatorq_pipeline, whose value is "red_teaming" or "agent_simulation". It identifies which surface issued the call and is sent as request metadata alongside evaluatorq_run_id — filter on it to separate red-team traffic from simulation traffic regardless of run. Both evaluatorq_run_id and evaluatorq_pipeline are native request metadata fields on Chat Completions and Responses calls. They are sent to direct OpenAI-compatible endpoints as well as through the Orq router.

How it reaches every call¶

Both red-team and simulation route their datapoints through a nested evaluatorq() call. The run id isn't threaded through function arguments — it's bound to a contextvars.ContextVar (src/evaluatorq/common/thread_context.py) at the run's entrypoint and read back at the call site. Because a ContextVar set in an ancestor scope is visible to nested calls (and copied into child asyncio tasks), every LLM call issued from inside the nested evaluatorq() run automatically carries the SAME evaluatorq_run_id as the outer red-team/sim run — no explicit plumbing required.

Call sites read it back one of two ways, and the difference matters when you are tracking down a missing tag:

• Chat Completions (create / .parse) and Responses calls read the same context and send it as native request metadata.
• The router-specific thread body parameter is separate and remains endpoint- gated: it is included only when the client routes through Orq and a conversation thread is active. It is never required for run correlation.

Separate root invocations receive separate ids: two calls to simulate(), generate_and_simulate(), or generate() each get a distinct evaluatorq_run_id, even if called back-to-back in the same process. Nested evaluatorq() work within one red-team or simulation root receives that root's id, and nested generation helpers inherit it. Standalone generate_personas() and generate_scenarios() do not mint ids of their own. The evaluatorq-core orq.run_id attributes continue to describe evaluatorq evaluation runs and are unchanged by this correlation mechanism.

Using it¶

In Orq's trace UI, filter spans/traces on the evaluatorq_run_id request-metadata value (copy it from the orq.evaluatorq_run_id attribute on the run's root span, or from your own logs/hooks that captured the run id) to see every model call — target, judge, user-simulator, attacker, evaluator, generation — that belongs to one red_team() or simulate()/generate_and_simulate()/generate() invocation, including calls made through the nested evaluatorq() run. Add evaluatorq_pipeline to the filter to scope further to just red-team or just simulation traffic.

Content capture and truncation¶

Two env vars control how much text is stored on spans:

• EVALUATORQ_CAPTURE_MESSAGE_CONTENT (default true): set to false or 0 to keep LLM message content out of traces entirely. Token counts and model name are still recorded.
• EVALUATORQ_SPAN_MAX_TEXT_CHARS (default: no limit): set to a positive integer to truncate span text attributes. Truncated strings end with ... [truncated].

W3C trace context propagation¶

To propagate trace context across service boundaries, inject the active span's W3C traceparent/tracestate headers into your outgoing HTTP requests. Use the OpenTelemetry SDK's public inject() helper — a stable, supported API:

from opentelemetry.propagate import inject

headers: dict[str, str] = {}
inject(headers)          # writes `traceparent` (+ `tracestate`) for the active span
# pass `headers` into your outgoing request, e.g. httpx.get(url, headers=headers)

inject() is a no-op when no span is active, so it is safe to call whenever OpenTelemetry is installed. (The from opentelemetry.propagate import inject import itself requires OTel; if you need code that also runs without it installed, use the internal helper below, which degrades to an empty dict.)

Where to next¶

• Configuration — API keys and environment variables.
• CLI Reference — run evaluations and red-team/sim from the terminal.
• Orq Deployment — trace invocations against an Orq-hosted deployment.