Tracing¶

evaluatorq ships optional OpenTelemetry tracing. When enabled, every evaluation run, job, evaluator, and LLM call becomes a span you can view in the Orq dashboard or any OTLP-compatible backend.

How tracing is enabled¶

Tracing initialises lazily on the first evaluation run. It turns on automatically when either condition is true:

• ORQ_API_KEY is set — the OTLP base endpoint is https://my.orq.ai/v2/otel (or <ORQ_BASE_URL>/v2/otel if ORQ_BASE_URL is set); the exporter appends /v1/traces, so spans POST to …/v2/otel/v1/traces.
• OTEL_EXPORTER_OTLP_ENDPOINT is set — that endpoint is used as the OTLP base.

If neither variable is set, no tracer is created and all span context managers are no-ops.

Set ORQ_DISABLE_TRACING=1 or ORQ_DISABLE_TRACING=true to suppress tracing even when the above variables are present.

Install the OTEL packages¶

Tracing depends on optional packages that are not installed by default:

uv add opentelemetry-api opentelemetry-sdk \
    opentelemetry-exporter-otlp-proto-http \
    opentelemetry-semantic-conventions
# or via the extras bundle:
uv add "evaluatorq[otel]"

Prefer pip? Use python -m pip install "evaluatorq[otel]", which installs into the interpreter you just named rather than whichever pip happens to be first on your PATH.

If these packages are absent the SDK silently skips initialisation — no error is raised.

Minimal enable example¶

import os
import asyncio

os.environ["ORQ_API_KEY"] = "your_orq_api_key"   # tracing auto-enables

from evaluatorq import DataPoint, evaluatorq, job, string_contains_evaluator


@job("echo")
async def echo_job(data: DataPoint, _row: int) -> str:
    return str(data.inputs.get("text", ""))


asyncio.run(
    evaluatorq(
        "my-eval",
        data=[DataPoint(inputs={"text": "hello"}, expected_output="hello")],
        jobs=[echo_job],
        evaluators=[string_contains_evaluator()],
    )
)

To send traces to a custom OTLP endpoint instead:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 uv run my_eval.py

To debug tracing setup:

ORQ_DEBUG=1 uv run my_eval.py

This prints the resolved endpoint, auth header presence, and any initialisation errors to stdout.

OTLP exporter details¶

• Protocol: HTTP/protobuf (OTLPSpanExporter from opentelemetry-exporter-otlp-proto-http)
• Export mode: BatchSpanProcessor (asynchronous batching)
• Timeout: 5 seconds per export request. The separate end-of-run flush timeout is set with ORQ_OTEL_FLUSH_TIMEOUT_MS (see below).
• Auth: Authorization: Bearer <ORQ_API_KEY> is added automatically when the resolved endpoint's hostname ends in .orq.ai or is exactly orq.ai. For any other endpoint the header is not added; use OTEL_EXPORTER_OTLP_HEADERS to supply auth manually.
• Custom headers: parsed from OTEL_EXPORTER_OTLP_HEADERS as key1=value1,key2=value2.

Batching and flush¶

The defaults suit one-off runs. Tune them for a long-lived worker process, or for a CI job where losing spans is unacceptable.

The first three are fixed into the processor when tracing initializes and cannot change for the life of the process. ORQ_OTEL_FLUSH_TIMEOUT_MS is read on every flush, so a long-running process can raise it before a big run — and evaluatorq enforces that bound itself, because the SDK currently ignores the timeout handed to force_flush. Each value must be a positive integer. An empty, non-numeric, zero or negative value is ignored: evaluatorq logs a warning and falls back to the default, so a typo leaves tracing running on a value you can see in the logs.

The TracerProvider lives for the whole process. In a long-lived process that runs many evaluations, red-team or simulation runs back to back, the same span queue serves every run. These settings behave the same whether you export to Orq or to a third-party collector; only endpoint latency differs.

Spans are dropped two ways, and neither one fails the run, so the loss shows up in your logs instead of your exit code:

• Queue overflow — spans arrive faster than the exporter drains them. A burst of parallel jobs (the datapoint_parallelism argument) or a slow endpoint will do it. Once the queue is full the SDK evicts the oldest buffered span to make room, so a trace comes back missing its early spans rather than its last ones. It logs Queue full, dropping Span. through the stdlib logger opentelemetry.sdk._shared_internal. Identical warnings are suppressed within 20-second buckets, so one line means loss started, not that exactly one span was lost.
• Exit before flush — evaluatorq(), red_team(), simulate() and generate_and_simulate() each force-flush in a finally, so a run that raises still flushes. Argument validation that fails before the run's tracing scope opens does not, and the standalone pairwise entry points do not flush at all. The SDK's atexit hook flushes again on a clean shutdown. Neither helps against a flush that hits its timeout, or against a hard SIGKILL such as an OOM kill or a CI job cancelled mid-run, which drops the buffer without a warning. A flush that times out logs OTEL span flush timed out after <ms>ms; some spans may not have been exported. and one that fails outright logs OTEL span flush failed (<error>); some spans may not have been exported.

The queue drains continuously, not at the end of a run, and it does not drain on a fixed cadence. ORQ_OTEL_SCHEDULE_DELAY_MS is the idle timer: it decides how long a partial batch waits before going out. As soon as the queue reaches ORQ_OTEL_MAX_BATCH_SIZE the exporter wakes immediately and keeps exporting while the queue stays above that threshold. Under a burst, lowering the schedule delay changes nothing, because throughput is bounded by how fast the collector accepts batches.

For CI, absorb the burst and allow a longer final flush:

export ORQ_API_KEY="your_orq_api_key"      # enables tracing
export ORQ_OTEL_MAX_QUEUE_SIZE=16384       # absorb a burst of parallel jobs
export ORQ_OTEL_FLUSH_TIMEOUT_MS=30000     # give the final flush up to 30s

In a GitHub Actions workflow, set the same values in the job's env: block:

env:
  ORQ_API_KEY: ${{ secrets.ORQ_API_KEY }}
  ORQ_OTEL_MAX_QUEUE_SIZE: 16384
  ORQ_OTEL_FLUSH_TIMEOUT_MS: 30000

The flush timeout is an upper bound on the wall-clock time added to the job. If the collector stops responding, the run waits the full 30 seconds, logs a warning, and continues.

Span hierarchy¶

Evaluation runner spans¶

orq.job                          # one per DataPoint — root when no ambient trace is active,
  ├── <your job code>            #   otherwise a child of the caller's span
  └── orq.evaluation             # one per evaluator applied to this job

All orq.job spans from a single evaluatorq() call share the same orq.run_id attribute, which ties them together as a logical run without requiring a common parent span. Because there is no common parent, though, an N-row run arrives as N separate traces — one rooted at each orq.job.

One trace per run: single_trace=True¶

Pass single_trace=True to bracket the whole run in one evaluatorq.run span, so every row lands in a single trace:

await evaluatorq("my-eval", data=rows, jobs=[my_job], single_trace=True)

evaluatorq.run                   # one per evaluatorq() call — the root
  └── orq.job                    # one per DataPoint, now a child rather than a root
      └── orq.evaluation

It defaults to False so existing traces keep their shape. Red teaming and simulation do not need the flag — they already open their own root spans (Evaluatorq - Red Teaming / Evaluatorq - Agent Simulation), and orq.job nests under those.

Span attributes on evaluatorq.run:

Span attributes on orq.job:

Span attributes on orq.evaluation:

Red teaming spans¶

This is the dynamic and hybrid shape:

Evaluatorq - Red Teaming         # root — one per red_team() call
  ├── orq.redteam.context_retrieval
  ├── orq.redteam.datapoint_generation
  │     ├── orq.redteam.capability_classification
  │     │     ├── chat (llm_purpose=classify_tools)
  │     │     └── chat (llm_purpose=infer_resources)
  │     └── orq.redteam.strategy_planning
  │           └── chat (llm_purpose=generate_strategies)
  ├── orq.job                    # one per attack datapoint
  │     └── orq.redteam.attack
  │           ├── orq.redteam.target_call
  │           └── orq.redteam.attack_turn  (x N turns)
  │                 ├── orq.redteam.adversarial_generation
  │                 │     └── chat (llm_purpose=adversarial)
  │                 └── orq.redteam.target_call
  ├── orq.evaluation             # security evaluator result, annotated in place
  │     └── responses (llm_purpose=evaluation)
  └── orq.redteam.memory_cleanup # post-run agent memory entity cleanup (only when cleanup is enabled, entities exist, and the target has configured memory stores)

Static mode is single-shot, and its tree is correspondingly shorter: no context_retrieval or datapoint_generation work, and no attack_turn or adversarial_generation spans. One target_call per attack, then the evaluation:

Evaluatorq - Red Teaming
  ├── orq.job
  │     └── orq.redteam.attack
  │           └── orq.redteam.target_call
  ├── orq.evaluation
  │     └── responses (llm_purpose=evaluation)
  └── orq.redteam.memory_cleanup

There is no orq.redteam.security_evaluation span. The OWASP scorer annotates the framework's own orq.evaluation span in place rather than nesting a redundant layer between it and the judge's LLM span.

The judge's span is named for the endpoint that served it. Evaluator configs default to api='responses', so it is usually responses {provider}/{model}; it is chat {model} when the call falls back to Chat Completions — a non-router client, structured_output=False, a model the catalogue cannot qualify as Responses-capable, or api='chat_completions' set explicitly.

LLM spans (chat ... / responses ...) carry standard GenAI attributes:

The root Evaluatorq - Red Teaming span additionally carries:

Simulation spans¶

Evaluatorq - Agent Simulation    # root — one per simulate() / generate_and_simulate() call
  ├── chat/responses {model}     # persona/scenario generation calls
  ├── orq.simulation.first_message_generation   # ONE span for the whole persona x scenario sweep
  │     └── chat/responses {model} (x N pairs)
  └── orq.simulation.run         # one per datapoint
        ├── orq.simulation.first_message_generation   # only when no first message was pre-generated
        │     └── chat/responses {model} (orq.llm.purpose="first_message")
        └── orq.simulation.turn  (x N turns)
              ├── orq.simulation.user_simulator_call   # turns 2+ only — turn 1's user line is the first message above
              │     └── chat/responses {model} (orq.llm.purpose="user_simulator")
              ├── orq.simulation.target_call           # calls the agent under test; no span attrs of its own
              └── orq.simulation.judge_evaluation
                    └── chat/responses {model} (orq.llm.purpose="judge")

orq.simulation.generate          # root — one per standalone generate() call
  └── chat/responses {model}     # persona/scenario/first-message generation calls

generate_personas() and generate_scenarios() don't open a synthetic root span when invoked standalone. They do create orq.simulation.persona_generation / orq.simulation.scenario_generation spans around their LLM calls. Those generation spans carry the active run metadata when called inside an outer simulation or red-team scope; standalone helpers intentionally have no synthetic run id to stamp.

Span attributes on Evaluatorq - Agent Simulation / orq.simulation.generate:

Span attributes on orq.simulation.run:

Span attributes on orq.simulation.first_message_generation:

Under the root (one span covering the whole persona x scenario sweep):

Under orq.simulation.run (only when a datapoint carried no pre-generated first message):

Span attributes on orq.simulation.turn:

orq.simulation.target_call, orq.simulation.judge_evaluation, and orq.simulation.user_simulator_call carry no span attributes of their own — they exist purely to scope the nested LLM call (and, for target_call, the target's own input/output recording). LLM spans nested under judge_evaluation and user_simulator_call carry the same GenAI attributes as the red teaming LLM spans above, tagged via orq.llm.purpose.

Judge-panel spans¶

llm_jury(), run_jury(), and run_pairwise() (src/evaluatorq/common/jury.py, src/evaluatorq/pairwise.py) run a panel of judges under a shared span hierarchy:

orq.evaluation {evaluator}         # from the core runner, when a jury backs an evaluator
  └── orq.jury                     # one per deliberation (orq.pairwise_jury in comparative mode)
        └── orq.judge              # one per judge (x2 in comparative mode — see below)
              └── responses {model}  # the judge's own LLM call(s), tagged orq.llm.purpose="judge"

The leaf span is responses {provider}/{model} on the default api='responses', and chat {model} when the call falls back to Chat Completions.

The panel opens no span of its own outside orq.jury — it can equally be called standalone (not nested under orq.evaluation), in which case orq.jury is the root. All jury/judge spans are opened via evaluatorq.common.tracing's with_span(), so — like every other span in this document — they are a no-op when tracing is disabled; verdicts and aggregation are unaffected either way.

A judge whose call failed leaves its orq.judge span with OTel status ERROR (via set_span_error), but the failure is swallowed at the panel level — the jury carries on with whatever judges succeeded (or promotes a replacement) and the parent orq.jury span stays OK.

Span attributes on orq.judge:

No token usage or cost here: those are recorded once, on the chat spans underneath, and rolled up by the consumer. Stamping them on every ancestor as well made the same tokens appear three times in one trace.

judge.label_swapped is only ever set (True/False) in comparative mode — in plain run_jury() deliberations it is absent, since each judge votes once.

One judge attribute lives a level down, on the chat / responses span that made the call rather than on orq.judge: judge.verdict_coerced. It is set to abstain_with_value when the model returned abstain=true together with a non-null value — a self-contradictory verdict, kept as an abstention with the value dropped. It is absent on a well-formed verdict, so counting it per judge model in the trace store answers "can this model follow the verdict schema". Nothing aggregates it today — it does not reach JuryVote, the run manifest or any report, so a coerced verdict is indistinguishable from a clean abstention once it leaves the judge call. Query the spans, not the run artifact.

Span attributes on orq.jury:

pairwise_plurality is a reported value, not an accepted argument — you cannot pass it to aggregator=, and validate_aggregator() rejects it. It names the rule run_pairwise() applies internally: pairwise_consensus(), a strict plurality over reconciled pair votes, run after judges that flipped across the two orderings have already been dropped to abstentions. The six aggregator= keywords reduce raw per-judge votes instead, so labelling this one mode would name it after a function it does not call.

Comparative (pairwise) mode¶

run_pairwise() compares two responses (A vs. B) and, to control for position bias, runs every judge in both label orderings. This changes the span shape from the plain jury case:

• One orq.pairwise_jury span covers the whole comparison — both orderings drive the same span rather than each minting its own; run_pairwise calls the internal _run_jury_core directly (not run_jury) so it doesn't open a second jury span per ordering.
• Each judge appears twice under that one orq.pairwise_jury span — one orq.judge span per ordering, distinguished by judge.label_swapped (False for the A/B ordering, True for the swapped B/A ordering).
• The span is named orq.pairwise_jury, not orq.jury — it aggregates reconciled pair votes rather than raw per-judge votes, so it gets its own name rather than masquerading as a plain jury. Its attributes stay in the jury.* namespace, plus these comparative-only extras:

judge.verdict is already un-swapped in comparative mode. The labels a judge returns there name a position, not a response: a judge that picks the same response both times says A in one ordering and B in the other, which reads as a self-contradiction and is in fact the opposite, a perfectly consistent judge. label_swapped=True spans are mapped back to the canonical frame before the attribute is written, so "how often did this judge pick response A" is answerable from judge.verdict alone, with no join against judge.label_swapped.

There is deliberately no raw-frame twin. The text the verdict was parsed from is one level down, on the chat child's gen_ai.output.messages, so a second attribute here would only restate what the trace already holds — the same reasoning as the alias removal noted above.

Un-swapping is per-ordering and needs nothing but label_swapped. Flip detection is what needs both orderings, and it stays on the parent — jury.flipped_judges names the judges that really did follow slot order.

jury.flipped counts judges that answered in both orderings but disagreed with themselves — that is position bias, not a failure, so a flipped judge is deliberately excluded from jury.judges_failed: judges_failed counts only judges with no reconciled vote and no flip (i.e. one or both orderings raised an error). A judge can be flipped, failed, or a normal decisive vote, but never counted under more than one of those buckets.

orq.evaluation, orq.jury / orq.pairwise_jury, orq.judge, and the nested LLM spans follow the ambient OTel context — nothing threads an explicit parent across the orq.evaluation → orq.jury seam, so a jury backing a custom evaluator's scorer nests correctly without extra plumbing.

Run correlation¶

Every LLM invocation issued during a red_team() or simulation run (simulate(), generate_and_simulate(), or generate()) is tagged so an operator can filter Orq's trace UI down to exactly the model calls belonging to one run. The same metadata is inherited by generate_personas() and generate_scenarios() when they are called inside an outer simulation or red-team scope; standalone calls have no synthetic root run id.

A companion key rides the same rail: evaluatorq_pipeline, whose value is "red_teaming" or "agent_simulation". It identifies which surface issued the call and is sent as request metadata alongside evaluatorq_run_id — filter on it to separate red-team traffic from simulation traffic regardless of run. Both evaluatorq_run_id and evaluatorq_pipeline are native request metadata fields on Chat Completions and Responses calls. They are sent to direct OpenAI-compatible endpoints as well as through the Orq router.

How it reaches every call¶

Both red-team and simulation route their datapoints through a nested evaluatorq() call. The run id isn't threaded through function arguments — it's bound to a contextvars.ContextVar (src/evaluatorq/common/thread_context.py) at the run's entrypoint and read back at the call site. Because a ContextVar set in an ancestor scope is visible to nested calls (and copied into child asyncio tasks), every LLM call issued from inside the nested evaluatorq() run automatically carries the SAME evaluatorq_run_id as the outer red-team/sim run — no explicit plumbing required.

Call sites read it back one of two ways, and the difference matters when you are tracking down a missing tag:

• Chat Completions (create / .parse) and Responses calls read the same context and send it as native request metadata.
• The router-specific thread body parameter is separate and remains endpoint- gated: it is included only when the client routes through Orq and a conversation thread is active. It is never required for run correlation.

Separate root invocations receive separate ids: two calls to simulate(), generate_and_simulate(), or generate() each get a distinct evaluatorq_run_id, even if called back-to-back in the same process. Nested evaluatorq() work within one red-team or simulation root receives that root's id, and nested generation helpers inherit it. Standalone generate_personas() and generate_scenarios() do not mint ids of their own. The evaluatorq-core orq.run_id attributes continue to describe evaluatorq evaluation runs and are unchanged by this correlation mechanism.

Using it¶

In Orq's trace UI, filter spans/traces on the evaluatorq_run_id request-metadata value (copy it from the orq.evaluatorq_run_id attribute on the run's root span, or from your own logs/hooks that captured the run id) to see every model call — target, judge, user-simulator, attacker, evaluator, generation — that belongs to one red_team() or simulate()/generate_and_simulate()/generate() invocation, including calls made through the nested evaluatorq() run. Add evaluatorq_pipeline to the filter to scope further to just red-team or just simulation traffic.

Content capture and truncation¶

Two env vars control how much text is stored on spans:

• EVALUATORQ_CAPTURE_MESSAGE_CONTENT (default true): set to false or 0 to keep LLM message content out of traces entirely. Token counts and model name are still recorded.
• EVALUATORQ_SPAN_MAX_TEXT_CHARS (default: no limit): set to a positive integer to truncate span text attributes. Truncated strings end with ... [truncated].

W3C trace context propagation¶

To propagate trace context across service boundaries, inject the active span's W3C traceparent/tracestate headers into your outgoing HTTP requests. Use the OpenTelemetry SDK's public inject() helper — a stable, supported API:

from opentelemetry.propagate import inject

headers: dict[str, str] = {}
inject(headers)          # writes `traceparent` (+ `tracestate`) for the active span
# pass `headers` into your outgoing request, e.g. httpx.get(url, headers=headers)

evaluatorq itself injects these headers on every call it makes to a provider — chat completions, Responses, the Orq agent target and the Orq deployment target — so a provider that runs its own tracing nests its server-side spans under the calling span instead of starting a loose root trace. Set EVALUATORQ_PROPAGATE_TRACE_CONTEXT to false or 0 to switch that off — outgoing requests then carry no traceparent, and the receiving side traces independently. The toggle covers evaluatorq's own calls; it does not affect your own inject() calls.

inject() is a no-op when no span is active, so it is safe to call whenever OpenTelemetry is installed. (The from opentelemetry.propagate import inject import itself requires OTel; if you need code that also runs without it installed, use the internal helper below, which degrades to an empty dict.)

Where to next¶

• Configuration — API keys and environment variables.
• CLI Reference — run evaluations and red-team/sim from the terminal.
• Targets › Orq-hosted — trace invocations against an Orq-hosted agent or deployment.