Agent Simulation¶
Drive your agent through realistic multi-turn conversations without writing test transcripts by hand. Three LLMs are in play:
- Your agent — the target under test (a hosted Orq agent, a callback, or an Orq deployment).
- User simulator — plays a persona pursuing a scenario goal, turn by turn.
- Judge — scores whether the goal was met and whether any rules were broken.
Configure the simulator and judge with LLMCallConfig. The former AgentConfig, JudgeAgentConfig and UserSimulatorAgentConfig classes have been removed; pass judge scenario data through goal=, criteria= and ground_truth=, custom simulator instructions through system_prompt=, and authentication through ORQ_API_KEY or a preconfigured client in LLMCallConfig(client=...).
Requires the simulation extra and an ORQ_API_KEY:
Prefer pip? Use python -m pip install "evaluatorq[simulation]", which installs into the interpreter you just named rather than whichever pip happens to be first on your PATH.
Requires the simulation extra, the openai package, and an OPENAI_API_KEY:
Prefer pip? Use python -m pip install "evaluatorq[simulation]" openai, which installs into the interpreter you just named rather than whichever pip happens to be first on your PATH.
Requires the simulation extra, the anthropic extra, and an ANTHROPIC_API_KEY with no ORQ_API_KEY or OPENAI_API_KEY set, and every model role set to a Claude id. See Go Anthropic-direct:
sequenceDiagram
participant U as User simulator
participant A as Agent under test
participant J as Judge
U->>A: next user turn
A-->>U: agent reply
loop until max_turns or stop condition
U->>A: follow-up turn
A-->>U: response
end
U->>J: full transcript + scenario
Note over J: scores goal_achieved / criteria_met Going OpenAI-direct? Override the model roles
The default fast and smart models are openai/gpt-6-luna and openai/gpt-6-sol — provider-prefixed, because the default route is the Orq router, which resolves provider/model. With only OPENAI_API_KEY set, calls go straight to OpenAI, which does not know those ids and rejects them. Set EVALUATORQ_FAST_MODEL=gpt-6-luna and EVALUATORQ_SMART_MODEL=gpt-6-sol once, or pass the bare id on every model flag you use. The Models section of the Configuration page covers the roles.
Generate from a one-line description¶
The fastest start: generate_and_simulate() synthesizes the personas, scenarios, and opening messages from a short description of your agent — no hand-written Persona(...) / Scenario(...).
Point it at a hosted Orq agent with target="agent:<key>" (the agent key from AI Studio → Agents). The simulator and judge LLMs route through Orq by default.
Agents with a memory store attached reject calls that carry no memory scope (a 400 with memory_entity_id_required). A fresh entity id is minted per conversation automatically, so parallel conversations never share memory; pass memory_entity_id="..." (CLI: --memory-entity) to run every conversation against one specific, e.g. pre-seeded, entity instead.
# No CLI flag for experiment_description, orq_folder_path or
# raise_on_execution_failure: set them from Python or in --config
eq sim run \
--name support-agent-sim \
--target agent:my-support-agent \
--agent-description "Support agent for an e-commerce store" \
--num-personas 3 --num-scenarios 4 \
--max-turns 6 \
--evaluator goal_achieved --evaluator criteria_met \
--report support-simulation.json
import asyncio
from evaluatorq.simulation import generate_and_simulate
async def main():
results = await generate_and_simulate(
run_name="support-agent-sim",
target="agent:my-support-agent", # hosted Orq agent, routed via ORQ_API_KEY
agent_description=(
"Customer support agent for an e-commerce store; "
"handles refunds, orders, and product questions."
),
num_personas=3,
num_scenarios=4, # → 12 persona × scenario simulations
max_turns=6,
evaluator_names=["goal_achieved", "criteria_met"],
experiment_description="September support regression",
orq_folder_path="Support/September",
report_path="support-simulation.json",
save=True,
raise_on_execution_failure=True,
)
passed = sum(r.goal_achieved for r in results)
print(f"Pass rate: {passed}/{len(results)}")
if __name__ == "__main__":
asyncio.run(main())
Pass llm_config= to route the simulator and judge through OpenAI directly. Use target= for the agent under test.
import asyncio
from openai import AsyncOpenAI
from evaluatorq.contracts import LLMCallConfig, Message
from evaluatorq.simulation import generate_and_simulate
client = AsyncOpenAI()
SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful."
async def openai_agent(messages: list[Message]) -> str:
history = [{"role": "system", "content": SYSTEM}]
history += [{"role": m.role, "content": m.content or ""} for m in messages]
resp = await client.chat.completions.create(model="gpt-5.6-luna", messages=history)
return resp.choices[0].message.content or ""
async def main():
results = await generate_and_simulate(
run_name="support-agent-sim-openai",
target=openai_agent,
agent_description=(
"Customer support agent for an e-commerce store; "
"handles refunds, orders, and product questions."
),
num_personas=3,
num_scenarios=4,
llm_config=LLMCallConfig(model="gpt-5.6-luna"), # simulator + judge on OpenAI directly
max_turns=6,
evaluator_names=["goal_achieved", "criteria_met"],
upload_results=False,
)
passed = sum(r.goal_achieved for r in results)
print(f"Pass rate: {passed}/{len(results)}")
if __name__ == "__main__":
asyncio.run(main())
run_name is an optional label for this execution in run records, results, and traces. Existing calls with evaluation_name still work.
agent_description drives generation; num_personas × num_scenarios is the requested number of cases. The actual count can differ when generation returns a different number of personas or scenarios or cannot create an opening message for a pair. The simulation-side LLMs resolve their provider by precedence: an explicitly passed generation_client wins, then llm_config.client, then ORQ_API_KEY (the Orq AI Router), then OPENAI_API_KEY, then ANTHROPIC_API_KEY. See Configuration.
Pass generation_instructions="..." to steer the whole generated set. It is a free-text instruction applied to every persona AND scenario (e.g. "enterprise B2B buyers, frustrated, replying in German"), stacked on top of the built-in prompts. Unlike a seed, which names one archetype and yields one object, it shapes the entire batch, and it composes with seeds and edge_case_percentage. It is accepted by generate_and_simulate(), generate(), and the seed helpers below; to steer personas and scenarios differently, call generate_personas() / generate_scenarios() separately. On the CLI it is --generation-instructions.
simulate() and generate_and_simulate() accept the same result options:
| Python keyword | What it controls |
|---|---|
experiment_description | Description of the experiment uploaded to Orq; has no effect when upload_results=False. |
orq_folder_path | Folder for uploaded results in Orq. |
report_path | Local JSON report file, written when save=True. Without a path, save=True writes to the run store. |
raise_on_execution_failure | Raises SimulationDroppedError for a dropped, errored, or timed-out conversation. Defaults to True; goal scores do not trigger it. |
The earlier names evaluation_description, orq_results_path, report, and exit_on_failure still work. If you supply both names for one option, their values must match.
CI and local runs
A simulation that produced no conversation — dropped, or ended in error/timeout — raises by default; ordinary failed goals remain in the returned results. Set raise_on_execution_failure=False for exploratory runs. When ORQ_API_KEY is available, results upload to Orq by default; pass upload_results=False to suppress the Experiment upload. That is not an offline mode — see What gets uploaded.
raise_on_execution_failure gates on datapoints that never produced a conversation, not on scores, so it will not fail a build for an agent that simply answered badly. For a gate on the scores themselves — turning evaluator results into an exit code, with the env vars and workflow step to go with it — see In an evaluatorq Run › In CI.
Repeat each case to see how stable the result is¶
One conversation per case tells you what happened once. An agent, a simulated user and a judge are all models, so the same case can end differently the next time. repetitions runs every case several times, each as a separate fresh conversation, so you can see how often the goal is reached and not only whether it was reached once.
It works the same on simulate() and eq sim simulate, for cases you pass in yourself. The default is 1, or the replayed run's count when you replay. A value that is not a whole number of at least 1 raises ValueError before anything runs.
Three settings sound alike and do different things:
| Setting | What it multiplies |
|---|---|
num_personas, num_scenarios | The cases. More of them means more different situations. |
repetitions | The conversations per case. The persona, scenario, opening message and criteria stay the same; the conversation is run again from the start. |
Judge or jury repetitions (llm_jury, the red-team judge) | The scoring calls on one finished transcript. repetitions here never repeats the judge. |
What to expect from a repeated run:
- Generation runs once. Personas, scenarios and opening messages are created once per case, then every repetition reuses them.
- Each repetition is its own conversation. It gets its own history, its own thread id and its own copy of the target, made with the target's
new(), so anagent:target's default memory is not shared between repetitions. Three things are shared because you asked for them: an explicitmemory_entity_id, which is one memory scope for the whole run, any state your own target callable keeps outside the messages it is given, since the callable itself is not copied, and anything your ownAgentTarget.new()carries over to the copy. - Every conversation is kept. The run holds
cases x repetitionsresults, and each one is uploaded and saved on its own. Each result'smetadatahasbase_datapoint_id(the case),repetition_index(0, 1, 2, ...) and adatapoint_idunique to that conversation (<case id>#r<repetition_index>). Hooks receive that unique id. Group bybase_datapoint_idto join conversations to their case: it is set on every run, withrepetition_index0 and equal todatapoint_idwhen the run has one repetition. Because of that id scheme, a run withrepetitionsabove 1 rejects a case whose id already ends in#r<digits>. Runs saved by 1.97.0 store the position astrial_indexand the id as<case id>#t<index>; they still load, withtrial_indexread asrepetition_index, and the saveddatapoint_idis kept as it was. - Uploaded experiment rows tell repetitions apart. With
repetitionsabove 1, each row's inputs carrybase_datapoint_idandrepetition_indexnext to the case, so the repetitions of a case are not identical rows in the Orq experiment. - A retry is not a repetition. A target call that fails and is retried stays inside the same conversation and does not add a result.
datapoint_parallelismbounds all conversations together, not each case separately.- The run plan shows the total before it starts, for example
6 cases x 10 repetitions = 60 conversations. Theon_confirmandon_run_startpayload carriesnum_cases,num_conversationsandrepetitions; the run span recordsorq.simulation.cases_count,orq.simulation.conversations_countandorq.simulation.repetitions. - Report steps scale with the conversations. The judge, every evaluator and the recommendations pass run once per conversation, and the executive summary reads all of them, so 10 repetitions cost roughly 10 times as much as one in those steps too, not only in the conversations. The run plan says so when
repetitionsis above 1. - Replay restores the count. A saved run stores each case once, next to the
repetitionsit ran with.previous_run/--from-runreloads those cases and runs each of them that many times again. Passrepetitionsto override it, for examplerepetitions=1to run each case once.
In the dashboard, all repetitions of a case land in the same heatmap cell. The cell shows the mean goal-completion score and, when it holds more than one conversation, n; hovering n gives the completed and errored counts and how many conversations reached the goal, for example goal achieved in 7 of 10 (70%). The Markdown and HTML reports print that count next to the cell's rate, as 70% (7 of 10). The mean score and the goal-achieved rate are different numbers: three conversations scored 60% each have a mean of 60% and a goal-achieved rate of 0%.
Repeating a case measures how much one case varies. It does not remove judge bias, it does not make your cases representative of real traffic, and a difference between two runs is not shown to be significant just because each cell has several conversations.
Seed by archetype¶
The middle ground between "just give me five" and specifying every trait: name the archetype, and generate_persona() / generate_scenario() fill the rest. You get back real Persona / Scenario objects to inspect, tweak, and pass to simulate().
import asyncio
from evaluatorq.simulation import generate_persona, generate_scenario, simulate
async def main():
persona = await generate_persona(
"angry customer",
agent_description="e-commerce support agent",
)
scenario = await generate_scenario("disputes a refund denial")
results = await simulate(
run_name="seeded-simulation",
target="agent:my-support-agent",
personas=[persona],
scenarios=[scenario],
max_turns=6,
evaluator_names=["goal_achieved", "criteria_met"],
)
print(f"Goal achieved: {results[0].goal_achieved}")
if __name__ == "__main__":
asyncio.run(main())
Batch forms generate_personas([...]) / generate_scenarios([...]) take a list of seeds and return one object each.
Check that generated items fit the agent¶
A fit check asks a judge model whether each generated persona or scenario fits the agent and its seed, and regenerates the ones that do not. Reach for it when the generated set feeds a run you will trust as a gate; skip it for a quick exploratory run, because it adds model calls.
generate_checked_personas() and generate_checked_scenarios() take the same seeds as the batch forms above and return a FitReport: the items, one outcome per item, and what the check cost.
import asyncio
from evaluatorq.simulation import generate_checked_personas
async def main():
report = await generate_checked_personas(
["angry customer", "first-time user"],
agent_description="A customer support agent for a retail bank.",
)
for persona, outcome in zip(report.items, report.outcomes):
print(f"{outcome.status:9s} attempts={outcome.attempts} {persona.name}")
print(f"unresolved: {report.unresolved}, judge calls: {report.judge_calls}, tokens: {report.usage.total_tokens}")
if __name__ == "__main__":
asyncio.run(main())
The judge answers one question per item: does it fit? For a persona that means it is a plausible user of the agent, it embodies its seed, and its numeric traits agree with its background. For a scenario it means the goal is something a user would bring to the agent, it is built around its seed, and its criteria are about that goal, do not contradict each other and can be observed in a conversation. A short item that passes those checks fits, and a seed counts when its meaning is there, whatever the wording.
Difficult users and deliberate probes fit. A hostile or derailing persona is a valid user to test against. A scenario whose goal is out of scope on purpose, or a misuse or manipulation attempt, fits when the seed asks for it, the scenario is marked is_edge_case, or its criteria expect the agent to decline or resist. An unrelated goal with none of those signals is rejected.
A rejected item goes back to the generator with the judge's explanation, up to max_rounds times (default 2). An approved item is never regenerated. Pass max_rounds=0 to judge once and regenerate nothing.
Outcome status | Meaning | Counts as a pass |
|---|---|---|
approved | The judge said the item fits. | Yes |
rejected | The judge still said no after the last round, or the regeneration failed or returned nothing. explanation holds the reason. | No |
unchecked | There is no verdict: the judge call failed, the judge abstained, or its answer was not true or false. explanation says which. | No |
Read report.unresolved before you use the items. report.items always holds one item per seed, including the rejected and unchecked ones, so passing it straight to simulate() runs them without a word. report.all_approved is the one-line gate.
A judge model id that does not exist makes every item unchecked: each judge call fails on its own, so the report comes back complete and nothing in it is approved.
Pass agent_description. Without it the judge checks fit against "a general-purpose conversational assistant", which almost any item fits, and a warning is logged. outcome.attempts counts the items generated for a seed, so a regeneration that failed does not add one. report.usage includes a regeneration that failed after the provider billed it.
The judge runs on the sim.judge model role, so it is not the generator's model by default. Setting llm_config.model applies one model to every simulation-side call, the judge included. Pass judge_client= to send the judge calls through a different client than generation.
Cost is one judge call per pending item per round, plus one generator call per regenerated item. report.usage covers the judge and the regenerations, not the first generation. As a reference point, judging four generated personas with openai/gpt-6-sol through the Orq router took 4 calls, 2,790 tokens and $0.0078 in one run on 2026-11-17.
To judge personas or scenarios you already have, without regenerating anything, call check_personas(personas, agent_description=..., seeds=[...]) or check_scenarios(...). seeds is optional and takes one entry per item.
Full control: hand-build personas¶
When you want exact personas and pass/fail criteria, build them yourself and call simulate(). A persona is who is talking (patience, assertiveness, tone); a scenario is what they want plus the criteria the agent must (or must not) satisfy.
A persona requires its core traits — name, patience, assertiveness, politeness, technical_level, communication_style, and background. Only emotional_arc and cultural_context default (to None). A scenario needs just name and goal; everything else, including criteria, is optional.
Pass target="agent:<key>" (the agent key from AI Studio → Agents) to route to a hosted Orq agent.
import asyncio
from evaluatorq.simulation import simulate
from evaluatorq.simulation.contracts import (
CommunicationStyle, EmotionalArc, MustHappenCriterion, MustNotHappenCriterion, Persona,
Scenario, StartingEmotion,
)
async def main():
persona = Persona(
name="Impatient Customer",
patience=0.2, assertiveness=0.8, politeness=0.4, technical_level=0.3,
communication_style=CommunicationStyle.terse,
background="Received the wrong item and wants a refund urgently",
emotional_arc=EmotionalArc.escalating,
)
scenario = Scenario(
name="Wrong Item Refund",
goal="Get a full refund for the wrong item received",
context="Ordered headphones but received a phone case instead",
starting_emotion=StartingEmotion.frustrated,
criteria=[
MustHappenCriterion("Agent asks for order details"),
MustHappenCriterion("Agent acknowledges the mistake"),
MustNotHappenCriterion("Agent blames the customer"),
],
)
results = await simulate(
run_name="basic-simulation-example",
target="agent:my-support-agent", # hosted Orq agent, routed via ORQ_API_KEY
personas=[persona],
scenarios=[scenario],
max_turns=6,
evaluator_names=["goal_achieved", "criteria_met"],
)
result = results[0]
score = result.goal_completion_score or 0.0
print(f"Goal achieved: {result.goal_achieved} score={score:.2f}")
for msg in result.messages:
who = "User" if msg.role == "user" else "Agent"
print(f"{who}: {msg.content}")
if __name__ == "__main__":
asyncio.run(main())
Use target= with any async function that maps the conversation to your agent's reply. Pass llm_config= to run the simulator and judge on OpenAI directly. Set upload_results=False for a local-only run.
import asyncio
from openai import AsyncOpenAI
from evaluatorq.contracts import LLMCallConfig, Message
from evaluatorq.simulation import simulate
from evaluatorq.simulation.contracts import CommunicationStyle, MustHappenCriterion, Persona, Scenario
client = AsyncOpenAI()
SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful."
async def openai_agent(messages: list[Message]) -> str:
"""Your agent under test — a raw OpenAI model."""
history = [{"role": "system", "content": SYSTEM}]
history += [{"role": m.role, "content": m.content or ""} for m in messages]
resp = await client.chat.completions.create(model="gpt-5.6-luna", messages=history)
return resp.choices[0].message.content or ""
async def main():
persona = Persona(
name="Impatient Customer",
patience=0.2, assertiveness=0.8, politeness=0.4, technical_level=0.3,
communication_style=CommunicationStyle.terse,
background="Received the wrong item and wants a refund urgently",
)
scenario = Scenario(
name="Wrong Item Refund",
goal="Get a full refund for the wrong item received",
criteria=[
MustHappenCriterion("Agent asks for order details"),
],
)
results = await simulate(
run_name="openai-agent-simulation",
target=openai_agent, # your OpenAI agent
personas=[persona],
scenarios=[scenario],
llm_config=LLMCallConfig(model="gpt-5.6-luna"), # simulator + judge on OpenAI directly
max_turns=6,
evaluator_names=["goal_achieved", "criteria_met"],
upload_results=False, # local-only run, no Orq experiment
)
result = results[0]
score = result.goal_completion_score or 0.0
print(f"Goal achieved: {result.goal_achieved} score={score:.2f}")
if __name__ == "__main__":
asyncio.run(main())
One persona × one scenario yields one SimulationResult with goal_achieved, goal_completion_score, turn_count, rules_broken, and the full message transcript. Every field, and the saved SimulationRun file, is in the output reference.
How criteria are scored¶
MustHappenCriterion("...") and MustNotHappenCriterion("...") are shorthand for Criterion(description="...", type="must_happen") and Criterion(description="...", type="must_not_happen"); both forms are interchangeable and save as the same plain criterion.
The judge audits every Criterion on every turn and the runner folds those verdicts over the whole conversation, so a violation on turn 2 still shows up in a run that ends on turn 6:
must_happenpasses if it occurred in any turn. Never occurring is a failure — intent, plans, and paraphrases do not count.must_not_happenfails if it was violated in any turn. One violation is permanent; a clean later turn does not clear it.
The judge is only ever asked what occurred, never what passed — a pass/fail flag means the opposite thing for the two criterion types, and models invert it. Occurrence is mapped to pass/fail in code, so rules_broken is derived, not reported. Failures land in rules_broken (criterion ids), criteria_results (description → passed), and the criteria_met score.
Per-criterion detail is in metadata['criteria_meta'], where audited says whether the judge actually returned a verdict for that criterion:
for c in result.metadata['criteria_meta']:
if not c['passed'] and c.get('audited') is False:
print(f"{c['id']} was not audited by the judge")
A must_happen the judge confirmed never occurred and one it did not audit both show passed: False; use audited to distinguish them. It has three states, not two: True audited, False not audited, and None on runs saved before the field existed — which is why the example tests is False rather than not c['audited'].
Each entry also carries evidence — the quote from the turn where the criterion's occurrence first flipped, taken from the judge's criteria_verdicts audit. It is '' when the criterion never occurred (or occurred without a tracked quote) and None when no tracker was available, same as audited.
Both keys reach the reports. A criterion that was not audited renders as not audited (a neutral ?, never a green tick) and is counted separately from the "N/M criteria met" tally. A run with criteria_verified = False says so above the criteria list.
The criteria_met score applies the same rule: an unaudited criterion is not met, so the score and the tally beside it agree.
A custom judge= must report per-criterion verdicts
The built-in JudgeAgent audits each unsettled criterion every turn. A custom judge that does not populate Judgment.criteria_verdicts cannot provide a reliable per-criterion audit.
That run is marked SimulationResult.criteria_verified = False, the runner logs a warning naming the scenario, and criteria_met scores it 0.0 — unknown, not met. Check the field, not the transcript:
A run that ends in an error or a timeout never reaches the audit either. Those results also score criteria_met as 0.0 (not 1.0) and log a warning, so neither a crashed run nor an unaudited one can inflate the average. A target that dies mid-run keeps whatever the judge had already confirmed — a must_not_happen it saw violated stays failed — while a must_happen that simply had not happened yet is reported as unknown, never as failed. The run was cut short before that criterion had its chance: it is not the judge's verdict that nothing happened; nobody looked.
The callable passed to target is the only structural difference from the Orq path — personas, scenarios, criteria, and the result shape are identical. Swap the callback body for any HTTP/LLM agent.
The four built-in scorers¶
evaluator_names picks from four built-ins. Names must be unique because evaluator detail and numeric scores are keyed by name. Two read the judge's verdicts; two apply a policy you can change with scoring=:
| Scorer | 0–1 meaning | Tunable |
|---|---|---|
goal_achieved | 1.0 when the judge decided the persona's goal was reached, else 0.0. | no |
criteria_met | Fraction of the scenario's criteria the judge audited and found satisfied. | no |
turn_efficiency | How few turns it took to reach the goal. 0.0 if the goal was missed. | yes |
conversation_quality | Weighted composite of the other three. | yes |
The default is ["goal_achieved", "criteria_met"] — the two that are meaningful for every scenario. The other two are opt-in.
What turn_efficiency actually measures¶
It is a cost proxy, conditioned on success: given that the goal was reached, how many conversational turns did it take? The assumption behind "fewer is better" is that the extra turns are usually the agent re-asking for something it could have inferred, clarifying its own vague answer, or wandering — so a user who got what they came for in two turns had a better experience, and cost less to serve, than one who needed twelve. A run that did not reach the goal scores 0.0 outright: failing quickly is not efficiency.
Where that assumption breaks. A task that legitimately needs many turns — a long intake form, a multi-step troubleshooting tree, a negotiation — is penalised by this metric for doing its job properly. If your scenarios look like that, either move the cliffs out so the curve matches a realistic conversation length, or leave turn_efficiency out of evaluator_names and ignore the score. Do not read a low turn_efficiency as a quality problem without checking the transcript length you actually expect.
The default curve is a set of cliffs, then a linear decay to a floor:
| Turns | 1–2 | 3–4 | 5–6 | 7 | 8 | 9 | 10+ |
|---|---|---|---|---|---|---|---|
| Score | 1.0 | 0.9 | 0.7 | 0.6 | 0.5 | 0.4 | 0.3 |
Past the last cliff each turn costs turn_efficiency_decay_per_turn (0.1), starting from that cliff's score, until turn_efficiency_floor (0.3) — a very long conversation that did reach the goal keeps a non-zero score, because it was inefficient, not failed.
The conversation_quality composite¶
One number per conversation, weighted across the other three scorers:
| Weight field | Default | What it weighs |
|---|---|---|
goal_achieved_weight | 0.4 | Did the judge mark the persona's goal as reached? |
criteria_met_weight | 0.3 | What share of the scenario's criteria were audited and satisfied? |
turn_efficiency_weight | 0.3 | How few turns that took (the cost proxy above). |
The weights must sum to 1.0 — SimulationScoringConfig rejects any other sum at construction, so the composite stays on the same 0–1 scale as its parts and remains comparable across runs.
Worked example. A refund conversation runs 4 turns, the judge marks the goal achieved, and 1 of the scenario's 2 criteria is met. The conversation_quality scorer returns 0.82 as a ConversationQualityScore; its .breakdown carries the component scores and weights that produced that number.
goal_achieved=1.0criteria_met=1/2=0.5turn_efficiency=0.9(4 turns: past the<= 2cliff, inside the<= 4one)conversation_quality=1.0 × 0.4 + 0.5 × 0.3 + 0.9 × 0.3 = 0.4 + 0.15 + 0.27 = 0.82
Read score.breakdown when you call conversation_quality_scorer() directly; it contains components with goal_achieved=1.0, criteria_met=0.5, and turn_efficiency=0.9, plus weights with 0.4, 0.3, and 0.3. raw_output on the returned EvaluationResult exposes the same structure in JSON form, and a completed simulation exposes it as result.evaluator_details['conversation_quality'].
The criteria_met evaluator's raw_output carries either per-criterion CriteriaMeta records under criteria with an audited label, optionally alongside invalid, or the lossy criteria_results dictionary when no metadata is available. Both shapes carry the run-level criteria_verified and unverified_reason fields; unverified_reason is None for a verified record set and names the missing audit for an unverified one. A completed simulation also exposes this payload as result.evaluator_details['criteria_met']. Those structured fields are local to the returned EvaluationResult, the returned SimulationResult, saved SimulationRun JSON, and JSONL result exports; custom evaluator raw_output is retained in the same local result surfaces. evaluatorq strips raw_output before the Orq experiment upload, and it is not attached to the evaluator span. The span and upload still carry the score, explanation, and pass flag.
Changing the policy¶
cat > scoring.json <<'EOF'
{
"scoring": {
"turn_efficiency_cliffs": [[6, 1.0], [10, 0.9], [16, 0.7]],
"goal_achieved_weight": 0.4,
"criteria_met_weight": 0.5,
"turn_efficiency_weight": 0.1
}
}
EOF
eq sim simulate --input cases.jsonl --target agent:my-onboarding-agent \
--name onboarding-sim \
--evaluator goal_achieved --evaluator criteria_met \
--evaluator turn_efficiency --evaluator conversation_quality \
--config scoring.json
from evaluatorq.simulation import SimulationScoringConfig, simulate
results = await simulate(
run_name="onboarding-sim",
target="agent:my-onboarding-agent",
evaluator_names=["goal_achieved", "criteria_met", "turn_efficiency", "conversation_quality"],
# A guided onboarding flow needs ~6 turns before anyone should call it slow.
scoring=SimulationScoringConfig(
turn_efficiency_cliffs=((6, 1.0), (10, 0.9), (16, 0.7)),
# Criteria matter more than speed for this agent.
goal_achieved_weight=0.4,
criteria_met_weight=0.5,
turn_efficiency_weight=0.1,
),
)
scoring= is accepted by both simulate() and generate_and_simulate(), and scoring is a field of the --config file on the CLI. Omitting it uses the defaults above. The config is bounded and rejects unknown fields, so a typo fails at construction rather than silently scoring with the shipped policy. Two shapes it refuses on purpose, because both produce a report that is quietly wrong rather than obviously broken:
- cliffs that are not ordered — turn thresholds must strictly increase and scores must not increase with them, so a longer conversation can never score higher than a shorter one;
- weights that do not sum to
1.0.
Scoring results yourself¶
Every built-in is a SimulationScorer, which is just a type alias:
get_evaluator(name, config) returns one, and get_all_evaluators(config) returns all four keyed by name, so you can rescore saved results under a different policy without rerunning the conversations. evaluator_names only accepts the built-in names. For a check of your own, write a function with the same shape and run it over the returned results:
from evaluatorq.simulation import SimulationResult, SimulationScoringConfig, get_evaluator
def no_handoff(result: SimulationResult) -> float:
"""1.0 when the agent never handed the user off to a human."""
replies = [m.content for m in result.messages if m.role == "assistant" and isinstance(m.content, str)]
return 0.0 if any("transfer you" in r.lower() for r in replies) else 1.0
strict = get_evaluator("turn_efficiency", SimulationScoringConfig(turn_efficiency_cliffs=((1, 1.0), (2, 0.5))))
for r in results:
print(r.turn_count, strict(r), no_handoff(r))
Hooks receive failed evaluator outcomes¶
Pass a SimulationHooks implementation through hooks= when you need lifecycle events. The on_evaluator_complete contract changed: the old callback received (datapoint_id, name, score: float, result), while the new callback receives (datapoint_id, name, score: EvaluatorScore, result: SimulationResult). Update a custom hook to read the numeric verdict from score.score.value, the explanation from score.score.explanation, and evaluator failures from score.error.
The callback fires once for every evaluator score, including failed or non-numeric scores. metadata['evaluator_scores'] remains numeric-only; unusable outcomes are recorded in metadata['evaluator_errors'] and appear in reports as a Dropped count. A hook must therefore tolerate score.score.value being non-numeric and must not assume that every event has a usable float.
from evaluatorq import EvaluatorScore
from evaluatorq.simulation import DefaultHooks, SimulationResult
class AuditHooks(DefaultHooks):
async def on_evaluator_complete(
self,
datapoint_id: str,
name: str,
score: EvaluatorScore,
result: SimulationResult,
) -> None:
if score.error is not None:
print(f"{datapoint_id} {name} dropped: {score.error}")
return
print(f"{datapoint_id} {name}: {score.score.value!r}")
on_stage_end(stage, meta) reports stage failures through meta['error']. The value is the live exception object, not a preformatted string; test error is not None and use type(error).__name__ and str(error) when logging it. This applies to generate(), generate_and_simulate(), and simulate(). The standalone persona and scenario generation helpers do not accept hooks or emit stage events. DefaultHooks logs a WARNING, and RichHooks prints a failed-stage line when the key is set.
From existing traces and data¶
You do not have to invent every test case from scratch. If you already have recorded conversations, real production traces, or a batch of datapoints from an earlier run, you can feed that history back into simulation in two ways: replay the exact same cases, or mine them for the archetypes that drive fresh ones.
Replay stored datapoints¶
A SimulationDatapoint bundles one persona, one scenario, and the opening message. Every case simulation runs is one of these, and you can persist them for reuse. eq sim generate writes the cases it builds to a JSONL file with --datapoints PATH (one datapoint per line); eq sim run does the same alongside a live run with --datapoints PATH:
# Generate cases once and keep them
eq sim generate --agent-description "e-commerce support agent" \
--num-personas 3 --num-scenarios 4 \
--datapoints cases.jsonl
# Re-run the exact same cases against any target, as often as you like
eq sim simulate --input cases.jsonl --target agent:my-support-agent
import asyncio
from pathlib import Path
from evaluatorq.simulation import generate, simulate
from evaluatorq.simulation.utils import load_datapoints_from_jsonl
async def main():
# Generate cases once and keep them: one model_dump_json() line per datapoint,
# the same file `--datapoints` writes (ids included)
if not Path("cases.jsonl").exists():
generated = await generate(
agent_description="e-commerce support agent",
num_personas=3,
num_scenarios=4,
)
lines = [dp.model_dump_json() for dp in generated]
Path("cases.jsonl").write_text("\n".join(lines) + "\n")
# Re-load the exact same cases on every later run
datapoints = load_datapoints_from_jsonl("cases.jsonl")
results = await simulate(
run_name="replay-v2",
target="agent:my-support-agent-v2", # new version, same cases
datapoints=datapoints,
max_turns=6,
evaluator_names=["goal_achieved", "criteria_met"],
)
passed = sum(r.goal_achieved for r in results)
print(f"Pass rate: {passed}/{len(results)}")
if __name__ == "__main__":
asyncio.run(main())
Because the file pins the personas, scenarios, and first messages, the run is reproducible. That makes it the natural way to compare two agent versions, or the same agent under a new set of evaluators, on an identical bank of cases.
load_datapoints_from_jsonl() reads datapoint files, such as the ones eq sim generate --datapoints and eq sim run --datapoints write, or a file you save from generate(). To replay a saved run instead, see Replay a previous run.
An Orq dataset is a stored collection of rows. Here, one row is one simulation datapoint: a persona (who the simulated user is), a scenario (what they want), and an opening message. dataset_id is the dataset's identifier, not its display name. If you create one with eq sim upload-dataset --input cases.jsonl --name "Support cases", the command prints the ID to use below; see the upload command reference. For an existing dataset, use its ID. Each row's inputs must contain a datapoint object or a persona + scenario pair, matching the SimulationDatapoint shape above.
To replay those rows as they are, pass the ID to simulate():
run_name is an optional label you choose for this execution in its run record, results, and traces. It is separate from dataset_id, which selects the stored cases, and evaluator_names, which selects the scoring checks. target names the agent being tested. Existing calls with evaluation_name still work; if you supply both names, their values must match.
simulate() takes five mutually exclusive sources — datapoints, dataset_id, experiment_id, previous_run, and personas + scenarios. Pass exactly one per run.
To extend the coverage, call extend_from_dataset() with that same ID. It reads the stored personas and scenarios as examples, then asks the model for new cases of similar kinds. Exact repeats are included once in the generation prompt, while distinct personas or scenarios with the same name remain separate examples. It returns only new datapoints; it does not change the Orq dataset or include its original rows. The request is guidance to the model, so matching frequencies and avoiding duplicates are not guaranteed.
Extension sends every distinct seed object to the model. A large dataset can make that prompt expensive or too long for the model's context window; use a smaller dataset of representative cases when generating new ones.
This example uses ORQ_API_KEY from the setup above and the default generation model, the fast role (openai/gpt-6-luna). Replace the dataset ID and agent key with yours:
import asyncio
from evaluatorq.simulation import extend_from_dataset, simulate
async def main():
extra = await extend_from_dataset("your-dataset-id", num_personas=3, num_scenarios=5)
results = await simulate(
run_name="support-dataset-extended",
datapoints=extra,
target="agent:my-support-agent",
)
print(f"Ran {len(results)} new cases")
if __name__ == "__main__":
asyncio.run(main())
extend_from_dataset() generates cases; simulate() runs them against the target. To run both the original and new cases, combine await datapoints_from_dataset("your-dataset-id") with extra and pass the combined list as datapoints. extend_from_experiment() does the same kind of generation from a prior experiment run.
num_personas and num_scenarios request new user archetypes and situations, respectively. The generator attempts one case for each pair, but it can return a different number of valid personas or scenarios, and a pair is dropped if its opening message cannot be generated. For example, two valid personas and five scenarios yield at most ten cases even when you request three personas. Check len(extra) if your run needs a fixed number of cases. An explicit api_key authenticates both dataset fetching and generation through Orq; a client in llm_config takes precedence for generation.
Replay a previous run¶
A saved run already holds the personas, scenarios, and first messages it used, so you can re-run it against a new target or new evaluators without exporting anything. The CLI saves each run to .evaluatorq/sim-runs/ by default (--no-save skips it). From Python, pass save=True to simulate() to get the same file. List what is there with eq sim runs; its File column is the file name to pass below.
Replay the most recent run, or pick a specific one:
# Most recent saved run
eq sim simulate --from-run latest --target agent:support-v2
# A specific run: file name (the .json is optional), run id, or path
eq sim simulate --from-run refund-bot_20261015-093012 \
--target agent:support-v2
eq sim simulate --from-run ./archive/refund-bot_20261015-093012.json \
--target agent:support-v2
import asyncio
from evaluatorq.simulation import simulate
async def main():
# Most recent saved run
latest = await simulate(
previous_run="latest",
target="agent:support-v2",
evaluator_names=["goal_achieved", "criteria_met"],
)
# A specific run, by file name, run name, run id, or path
pinned = await simulate(
previous_run="refund-bot_20261015-093012",
target="agent:support-v2",
evaluator_names=["goal_achieved", "criteria_met"],
)
print(len(latest), len(pinned))
if __name__ == "__main__":
asyncio.run(main())
previous_run and --from-run take the same reference forms:
| Pass | Matches |
|---|---|
latest | The newest run in .evaluatorq/sim-runs/ by modification time |
A file name, such as refund-bot_20261015-093012 | That file, with or without .json. Auto-saved files are named <run-name>_<UTC time YYYYmmdd-HHMMSS>.json |
A run name, such as refund-bot | The newest run whose file name is <that name>_<timestamp>, as it appears in the File column (lowercased, other characters replaced with _). A shorter name also matches longer names that start with it plus _, so refund matches refund_bot_<timestamp> runs; pass the full name |
A run id prefix, such as 3f9a1c2e | The run with that id, which is the saved file's run_id field (eq sim runs does not print it, and report_id in eq sim runs --json is a different value; prefer the file name). A prefix works if it is at least 8 characters and matches one run |
| A path | That file, wherever it lives |
A timestamp on its own does not match; use the whole file name. If a name matches several runs, the newest wins. An id prefix that matches several runs raises ReplayError, so use a longer prefix. latest on an empty store raises ReplayError: No saved simulation runs found in <runs directory> — nothing to replay. A run saved before replay support has no stored cases and raises ReplayError too; re-run it once to get a replayable file.
Replay reuses the stored cases and the stored turn limit. It does not restore the evaluators: pass --evaluator or evaluator_names, or the defaults run. Pass --max-turns or max_turns to override the stored limit. The stored cases are the base cases, one per persona and scenario pair, even when the saved run used repetitions. Replay restores the stored repetitions the same way as the turn limit; pass --repetitions or repetitions to override it.
Ground new cases in real traces¶
Replay reruns what you already have. The other move is to generate new cases that are shaped by what really happened. Production traces show you the user archetypes and situations your agent actually meets.
The async datapoints_from_traces(...) helper accepts the same TraceInput used by core evaluation. It fetches and normalizes traces through the shared importer, then turns each usable conversation into a simulation datapoint; pass the returned rows to the existing simulate() runner. A query that selects no conversation with a user turn raises rather than returning an empty list, so a mistyped search= fails where you made it rather than as a run of zero personas.
Query mode pulls a bounded recent batch, no trace ID required:
from datetime import datetime, timedelta, timezone
from evaluatorq import TraceInput
from evaluatorq.simulation import datapoints_from_traces, simulate
datapoints = await datapoints_from_traces(
source=TraceInput(limit=50, start_time=datetime.now(timezone.utc) - timedelta(days=1)),
)
results = await simulate(target='agent:my-support-agent', datapoints=datapoints)
Pass one trace_id instead once you already have a specific conversation to ground a case on:
This call bills: unlike redteam.datapoints_from_traces, which is pure and free, this one makes one summarize call and one persona/scenario-inference call per trace, plus one first-message call on the default (generate_first_message=True), so cost and latency scale with how many conversations you fetch. Share the summarize step with extend_from_traces by calling summarize_conversations yourself once and passing the result to both as summaries= — the pattern is below, under "The same thing is available from Python".
The trace is source material for a new simulation, not a scored result. Simulation infers a persona and scenario from each conversation and writes a fresh opening message by default. The trace analysis and simulation calls use their own budgets; max_turns applies to the generated conversation and does not count trace analysis.
The direct route is eq sim from-traces, which pulls recent traces from the Orq traces API and writes one datapoint per conversation — persona and scenario inferred from a short summary of it, opening message written from that persona and scenario. Add --extend N to also generate N new datapoints matching the traffic distribution of the fetched traces, so you get cases around real traffic rather than only the recorded ones. The same thing is available from Python as datapoints_from_traces() and extend_from_traces():
import asyncio
from evaluatorq.simulation import (
datapoints_from_traces,
extend_from_traces,
fetch_trace_conversations,
simulate,
summarize_conversations,
)
async def main():
conversations = await fetch_trace_conversations(limit=50)
summaries = await summarize_conversations(conversations)
# One datapoint per recorded conversation, plus 20 new ones around that traffic
recorded = await datapoints_from_traces(conversations, summaries=summaries)
extra = await extend_from_traces(conversations, num_datapoints=20, summaries=summaries)
await simulate(target="agent:my-support-agent", datapoints=recorded + extra)
if __name__ == "__main__":
asyncio.run(main())
Full flag list: eq sim from-traces.
What happens between a trace and a datapoint¶
Fetching is shared; both modes are then map-then-reduce. Each conversation is summarized on its own (the map), and the summaries — never the raw transcripts — go into the call that produces the output (the reduce). That is what keeps a prompt's size a function of how many traces there are rather than how long any one of them ran: before it, a single long agentic session crowded out the twenty short conversations it should have been weighed against.
Both modes summarize unconditionally: every conversation gets exactly one summarize call, and nothing downstream reads the raw transcript again — direct mode's persona/scenario inference reads the summary, and so does extension mode's traffic-profile reduce. A run doing both calls summarize_conversations once and passes the result as summaries= to each, so no conversation is summarized twice.
flowchart TD
A["POST /v2/traces/v3oql<br/>paged listing"] --> B["GET /v2/traces/{id}/v3spans<br/>per trace, 5 at a time"]
B --> C["Reconstruct conversation<br/>root span first, then any span<br/>with messages; gen_ai attributes<br/>JSON-decoded"]
C --> D{"Has a usable<br/>user message?"}
D -- "no" --> E["Dropped, counted in a warning"]
D -- "yes" --> F["TraceConversation"]
F --> S["MAP: summarize_conversations<br/>one call per conversation, ~250 tokens,<br/>5 in flight, shared by both modes"]
S --> G["Direct mode<br/>datapoints_from_traces"]
S --> H["Extension mode<br/>extend_from_traces"]
G --> I["REDUCE: infer Persona + Scenario<br/>1 call per conversation"]
I --> J["Write the opening message<br/>from that persona and scenario<br/>--replay-first-message reuses<br/>the recorded one"]
J --> K["SimulationDatapoint<br/>id = trace-{trace_id}"]
H --> L["REDUCE: 1 call over up to max_reduce_summaries (default 50)<br/>repeat intents collapsed, not double-counted"]
L --> M["Traffic profile prose:<br/>intent mix and shares, tone and<br/>patience ranges, edge cases"]
M --> N["DatapointGenerator<br/>personas x scenarios<br/>grounded in that profile"]
N --> O["N new SimulationDatapoints<br/>synthetic, not replayed"] Every LLM-side limit lives on TraceAnalysisConfig, passed as config= to either function; the fetch-side ones are fixed:
| Limit | Default | Why |
|---|---|---|
| Rows per listing page | 200 | The API's own cap; pagination continues until --limit is met or a page adds nothing new |
| Span fetches in flight | 5 | Politeness to the traces API |
| LLM calls in flight | 5 | Same width the datapoint generator uses |
summary_target_tokens | 250 | Roughly how long a summary should be. Soft — it goes in the prompt, nothing cuts the result |
max_reduce_summaries | 50 | How many summaries the profile call carries; the rest are dropped with a warning naming the count |
summary_max_tokens | 10000 | Completion budget for a summarize call — reasoning headroom, not the length target |
max_tokens | 10000 | Completion budget for the inference and profile calls |
generate_first_message | True | Write the opening from the persona; False replays the recorded one |
redact_pii | True | Instruct the model to replace identifying values with placeholders as it writes |
summary_target_tokens is a target, not a cut, and deliberately so. Truncating a summary removes its end, which is exactly where the prompt puts what went wrong and what was unusual — the two things the next step most needs. A length the model can aim at (models reason in tokens, not characters) buys a soft bound that keeps whole sentences. The reduce prompt's expected size is that target times max_reduce_summaries.
The completion budgets, by contrast, are deliberately far above the answers they bound. Reasoning models spend most of a budget thinking before emitting anything, so a budget sized to the output gets consumed by reasoning tokens and truncates the answer to nothing — the prompt bounds the length, the budget bounds the failure. Truncation is never silent: generate_structured raises on a length-finished response on every path rather than handing back a cut-off object.
Pagination stops when --limit is met, when the API says there is no more, or when a page returns rows that all lack a trace_id — a page that adds nothing cannot be followed by one that does, so that is where the loop ends, and it says so in a warning. There is no fixed page ceiling, so a large --limit is honoured for as many pages as it genuinely takes.
A trace that fails its span fetch, returns a non-list payload, or yields no user message is dropped with a warning rather than failing the batch — likewise an inference or summarize call that raises or returns nothing parseable. Extension mode logs how many of the sampled conversations actually reached the profile, because that count is the denominator its shares are computed over. A run that produced fewer datapoints than traces has those warnings behind it.
Generation runs before a simulation exists, so its spend has no field on any result. Each phase reports its own total to the run log — Trace summarization, Trace persona/scenario inference, Trace traffic profiling, Persona/scenario generation and Simulation recommendations each log the tokens, the number of LLM calls, and the cost when the models are priced. The call count includes the fallback rungs a structured-output call burned on the way to an answer, and a rung whose usage the provider did not report is counted as one unpriced call rather than as zero, so the figure reads as a lower bound instead of a confident total. A call that raised is counted too — the rungs it burned before truncating or refusing were billed, and the exception carries their total for the phase to pick up.
SimulationRun.token_usage_total is the run-level figure: every result's token_usage, plus — for generate_and_simulate() — the GENERATE stage's persona/scenario generation cost, plus the executive summary's own completion cost when one was generated. It is recomputed after the executive summary runs so it never goes stale relative to that later-arriving cost. Recommendation generation is not folded in — that spend stays log-only (Simulation recommendations: N tokens over M LLM call(s), $X), the same as the trace-analysis phases above. None only when nothing in the run was ever billed.
What lands in the generated dataset¶
Trace-derived datapoints are built from real conversations, and a persona background or scenario context written straight from one carries whatever was in it — names, order numbers, emails — into a JSONL that then gets committed and shared. By default only the summarize prompt, the one step that reads a raw transcript, is instructed to redact as it writes, replacing identifying values with placeholders ([CUSTOMER_NAME], [ORDER_ID]) that keep the meaning; the persona/scenario inference and traffic-profile prompts see only summaries and are told to carry those placeholders through rather than invent concrete values.
--no-redact-pii (or TraceAnalysisConfig(redact_pii=False)) turns it off, for when the concrete values are the point — reproducing a specific incident, or a fixture where a changed order number breaks the comparison — and the dataset stays somewhere the raw traffic could already go. With it off the persona/scenario inference and traffic-profile prompts both drop their "keep the placeholders" line, since telling a model to preserve placeholders that were never introduced invites it to invent them, and invented placeholders read as redaction that did not happen.
Either way this is an instruction to a model, not a guarantee. Treat a generated dataset from production traffic as needing the same review any export of that traffic would.
Why the opening message is generated, not replayed¶
Replaying the real user's first message looks like the faithful choice and behaves worse. The simulated user is the persona; if turn one is production text the persona would not have written, the conversation opens in one voice and continues in another, and whatever the agent does with that mismatch is not evidence about either. Reusing recorded text also carries any PII in it into a generated dataset that then gets committed and shared.
--replay-first-message (or TraceAnalysisConfig(generate_first_message=False)) is the opt-out, for when you are reproducing one specific recorded case and want the exact opening back.
Hand-picked seeds¶
When you want curated archetypes rather than a straight pull from traffic, seed generation yourself. Pull the recurring patterns out of your traces (the impatient buyer disputing a charge, the confused first-time user, the edge case that broke last week), then hand them to generate_personas() / generate_scenarios() as short seed phrases:
eq sim run \
--name trace-grounded-sim \
--target agent:my-support-agent \
--agent-description "e-commerce support agent" \
--persona-seed "impatient repeat buyer" \
--persona-seed "confused first-time user" \
--persona-seed "polite but persistent negotiator" \
--scenario-seed "disputes a duplicate charge" \
--scenario-seed "cannot find order confirmation" \
--scenario-seed "asks for a discount after a late delivery" \
--max-turns 6 \
--evaluator goal_achieved --evaluator criteria_met
import asyncio
from evaluatorq.simulation import generate_personas, generate_scenarios, simulate
# Archetypes and situations distilled from real traces
persona_seeds = ["impatient repeat buyer", "confused first-time user", "polite but persistent negotiator"]
scenario_seeds = ["disputes a duplicate charge", "cannot find order confirmation", "asks for a discount after a late delivery"]
async def main():
personas = await generate_personas(persona_seeds, agent_description="e-commerce support agent")
scenarios = await generate_scenarios(scenario_seeds, agent_description="e-commerce support agent")
results = await simulate(
run_name="trace-grounded-sim",
target="agent:my-support-agent",
personas=personas, # 3 personas × 3 scenarios → 9 simulations
scenarios=scenarios,
max_turns=6,
evaluator_names=["goal_achieved", "criteria_met"],
)
passed = sum(r.goal_achieved for r in results)
print(f"Pass rate: {passed}/{len(results)}")
if __name__ == "__main__":
asyncio.run(main())
The seed is a steer, not a transcript: generation fills in the persona traits and scenario criteria and writes a natural opening message, so each run explores the space around the pattern rather than replaying one recorded conversation. Persist the generated cases (eq sim generate --datapoints, or eq sim run --datapoints) and they become a replayable bank for the section above.
Seeds are a deliberate choice, not the only route
Writing seed phrases by hand means you decide which archetypes matter, rather than inheriting whatever your recent traffic happened to contain. When you'd rather start from real traffic, use eq sim from-traces above — it infers the personas and scenarios for you.
Noisy first messages¶
Real users paste text with zero-width spaces, send half a sentence, or drop markdown into a chat box. DatapointGenerator.generate_from_description() can rewrite a share of the generated opening messages that way. perturbation_rate is the probability, per datapoint, that its first_message gets one randomly chosen perturbation; it defaults to 0.0:
from evaluatorq.simulation import DatapointGenerator, simulate
generator = DatapointGenerator()
try:
datapoints = await generator.generate_from_description(
agent_description="Customer support agent for an online bookstore",
perturbation_rate=0.3,
)
finally:
await generator.close()
results = await simulate(target="agent:my-support-agent", datapoints=datapoints)
Only the first message is perturbed; later user turns come from the simulator unchanged. Which perturbation was applied is not stored on the datapoint, so compare first_message against an unperturbed run when you need to know.
The perturbations are plain string functions with no LLM call, so you can apply them to your own datapoints as well:
from evaluatorq.simulation import PerturbationType, apply_perturbation, apply_random_perturbation
apply_perturbation("Where is my order?", PerturbationType.truncate)
text, kind = apply_random_perturbation("Where is my order?")
PerturbationType | What it does to the message |
|---|---|
unicode_noise | Inserts copy-paste artifacts: zero-width and non-breaking spaces, direction marks, curly quotes, a BOM. |
truncate | Cuts the message at 40 to 80 percent of its length, as if sent too early. Messages of 10 characters or fewer are left alone. |
markdown_injection | Adds markdown such as a heading, a fenced "SYSTEM" block, or a javascript: link. |
code_injection | Adds a code-shaped payload: a fake system-role JSON message, a script tag, SQL, a template expression. |
mixed_language | Mixes in a phrase from another language. |
apply_perturbations_batch(messages, perturbation_rate=0.3) does the same over a list and returns (message, type or None) pairs.
Reading a run in the dashboard¶
Runs are saved to .evaluatorq/sim-runs/<name>_<timestamp>.json — automatically by eq sim run (unless you pass --no-save), and by simulate() when called with save=True. eq dashboard browses those files locally, no external service:
uv add "evaluatorq[dashboard]"
eq dashboard # browse every saved run
eq dashboard .evaluatorq/sim-runs # scope to simulation runs
The Python examples above don't save by default
simulate() and generate_and_simulate() default to save=False, so a script copy-pasted from earlier on this page leaves the dashboard empty. Pass save=True, or drive the run from the CLI (eq sim run), which saves unless you pass --no-save.
Land on the cross-surface overview, pick the run from Agent Sim, then work down the report tabs from headline to transcript:
| Tab | What it answers |
|---|---|
| Overview | Summary, pass rate, average score, the four per-turn quality metrics |
| Breakdown | Persona × scenario heatmap — usually the fastest read |
| Recommendations | Suggested instruction edits, applicable to an Orq agent |
| Transcripts | Every conversation, expandable to criteria, rationale and messages |
| Turn quality | The four metrics trended by turn index |
| Config | Run metadata plus the persona dials |
| Compare | KPI and per-conversation deltas against a second run |
Two easily conflated numbers: pass rate is the share of conversations where the judge set goal_achieved; average score is the mean goal_completion_score, so a run can average 0.7 while passing half its conversations. The CONFIDENCE badge is a band on the pass rate, not a statistical confidence — it carries no sample-size meaning.
The tab-by-tab walkthrough, with screenshots, lives in the Dashboard reference: Reading a simulation run. See also filters, trace links and downloads.
External framework demos¶
Each recording runs one framework's example end to end — the user simulator drives the conversation, the agent under test responds, and the judge scores the transcript. Sources live in examples/agent_simulation/ (files 06–09).
LangGraph¶
OpenAI Agents SDK¶
Pydantic AI¶
CrewAI¶
Where to next¶
- In an evaluatorq Run —
wrap_simulation_agent(), scoring a transcript with your own evaluators, and shipping the batch as an Experiment. - Examples › Agent Simulation — tool simulation, hardening loops, LangGraph / CrewAI / OpenAI Agents targets.
- Red Teaming — adversarial, attack-driven testing.
- Tuning — target timeouts, per-simulation wall clock, reasoning effort, and provider options.