evaluatorq.formats¶
Convert an agent run between chat, Responses, OTel and ATIF.
Bases: BaseModel
An agent run as an ATIF-v1.7 or v1.8 trajectory.
Read a document with from_json, not model_validate_json. Reading accepts ATIF-v1.7 and ATIF-v1.8 only. from_json requires schema_version at the document root and rejects a v1.7 document holding audio; built in code (or embedded as a subagent) the field defaults to ATIF-v1.7. to_json keeps the trajectory's own schema_version, upgrading to v1.8 when any content part in the document (embedded subagents included) is audio.
classmethod ¶
Read an ATIF JSON document (text, bytes or an already-parsed dict).
Unlike model_validate, the document root must carry schema_version, as the ATIF spec requires, and a document stamped ATIF-v1.7 may not contain audio content parts (audio needs ATIF-v1.8).
Raises:
| Type | Description |
|---|---|
ValueError | The document is not a JSON object, has no |
Return True if any content part in this trajectory or an embedded subagent is audio.
Return the JSON-ready dict, stamping one schema_version on the whole document.
Serialise to ATIF JSON. Version defaults to the trajectory's own, or 1.8 when audio is present.
Convert to chat. Lossy: see convert_responses_atif and convert_chat_responses for the losses.
Convert to a Responses transcript (items plus per-call Response metadata).
Convert to one OTel trace (root invoke_agent, chat and execute_tool children).
Bases: BaseModel
A run as contracts.Message turns. The lossy edge: no reasoning, no per-step metrics, no ids beyond tool calls.
Render as Responses input items via messages_to_responses_input.
Lost: tool results with no tool_call_id (warned), non-text assistant parts, Message.name on non-tool rows, and item ids that do not start with fc_.
to_atif(
*,
agent_name: str = 'unknown',
agent_version: str = 'unknown',
session_id: str | None = None,
) -> AtifTrajectory
Convert to an ATIF trajectory through Responses. session_id defaults to a hash of the messages.
Lost: everything lost by ChatConversation.to_responses and ResponsesConversation.to_atif (unlinked tool results, non-text assistant parts, Message.name on non-tool rows, non-fc_ item ids, file parts as markers). Consecutive assistant turns with no tool result between them merge into one agent step.
Convert to one OTel trace through Responses and ATIF.
Lost: everything lost by ChatConversation.to_atif and AtifTrajectory.to_otel, including the fc_ item ids (kept in ATIF step extra, which OTel drops). Chat has no timing, usage or model names, so the spans carry none.
Bases: BaseModel
One span. span_type keeps Orq's raw wrapper type; parsed message attributes leave attributes.
A naive start_time or end_time is read as UTC.
Bases: BaseModel
A list of spans forming one trace (a forest when parents are missing).
Every span that names a trace id must name the same one; spans with an empty trace id are accepted.
Raises:
| Type | Description |
|---|---|
ValueError | The spans carry more than one distinct trace id. |
classmethod ¶
Parse raw Orq span dicts (the full tree, messages kept typed and lossless).
Spans with no parent, or whose parent is not in this trace, in start-time order.
Direct children of span_id in start-time order (no time last, ties by list order).
Convert to an ATIF trajectory (subagent invoke_agent subtrees become subagent trajectories).
Lost: see convert_otel_atif.otel_to_atif for the mapping. Extra output choices (warned), span attributes other than model, usage, finish reasons and error type, and spans that are not chat, execute_tool or invoke_agent.
to_responses(
*, agent_name: str = 'unknown', agent_version: str = 'unknown'
) -> ResponsesConversation
Convert to a Responses transcript through ATIF.
Lost: everything lost by OtelTrace.to_atif and AtifTrajectory.to_responses (span times and ancestry, subagent trees, per-call metrics beyond one Response per agent step).
Convert to chat through ATIF and Responses.
Lost: everything lost by to_atif and ResponsesConversation.to_chat (reasoning, span times and ancestry, per-call metrics, subagent trees, media).
Bases: BaseModel
A run as Responses items plus, optionally, the Response of each model call.
items is the transcript: input and output items in order. Each Response.output holds the typed output items that call produced, and must match the transcript: responses built with an empty output get it filled from items (one response per run of consecutive output items), and responses that name their output must cover every output item in items, in order, none spanning an input item. A response only adds the model, usage, timing and status of its call.
Render as chat messages.
Lost: reasoning items (counted and warned once), media that is not text, image or file parts, status, annotations, and item ids that do not start with fc_.
to_atif(
*,
agent_name: str = 'unknown',
agent_version: str = 'unknown',
session_id: str | None = None,
) -> AtifTrajectory
Convert to an ATIF trajectory: user/system messages become steps, assistant output becomes agent steps.
session_id defaults to a hash of the items, so converting the same items twice gives the same id.
A tool result closes its agent step, and so does the first output item of the next Response when responses are set; each response enriches the agent step it starts. Kept in free-form slots: reasoning tokens and total tokens in metrics.extra; status, error and incomplete_details (non-default only), the response id, fc_ item ids and encrypted-only reasoning in step extra; the developer role in extra.original_role of a system step. A compaction item becomes a system step whose extra holds context_management and the item under evaluatorq.compaction. Lost: encrypted reasoning content, non-text assistant parts, file parts (rendered as [file: name] markers), and item types ATIF has no step for (warned).
Convert to one OTel trace through ATIF.
Lost: everything lost by ResponsesConversation.to_atif and AtifTrajectory.to_otel (encrypted reasoning content, non-text assistant parts, file parts as markers, item types ATIF has no step for).