Simulation (eq sim)¶

Agent simulation subcommand group. Always registered; the simulation extra (evaluatorq[simulation]) supplies the optional dependencies some of its commands need at runtime. sim is shorthand for convenience — the feature is agent simulation.

Three main verbs: generate (datapoints only), simulate (run against pre-built datapoints), run (generate then simulate in one shot).

eq sim run¶

Generate personas and scenarios, then run simulations.

eq sim run --agent-description "..." --openai-model gpt-5.6-luna
eq sim run --target agent:<key>

Targets — provide exactly one:

eq sim simulate¶

Run simulations from a pre-built datapoints JSONL file.

eq sim simulate --input dp.jsonl --target agent:<key>

Targets — same three flags as eq sim run. Provide exactly one input source: --input (-i), --dataset-id, --experiment-id (optionally narrowed by --experiment-run-id), --from-run, or inline datapoints (or personas with scenarios) in --config.

There is no --target-reasoning-effort flag here — it is a eq sim run flag only. To pin the target's reasoning effort on a pre-built datapoint set, set "target_reasoning_effort" in --config, or call simulate() with target_reasoning_effort= from Python (see Tuning).

Driving the CLI from a config file¶

--config on eq sim simulate and eq sim run takes the keyword arguments of the Python simulate() and generate_and_simulate() functions as one JSON object, under their public names (run_name, experiment_description, raise_on_execution_failure). That reaches every data-shaped parameter, including the ones without a flag: inline personas and scenarios or datapoints, scoring, target_agent_timeout_ms, max_tool_result_chars, per_simulation_timeout_s, upload_results, the full llm_config, and on sim run edge_case_percentage. Pass a path, or - to read it from stdin. YAML is not accepted.

Inline datapoints, or personas with scenarios, count as the input source of eq sim simulate, so the file below needs no --input:

{
  "target": "agent:my-agent",
  "personas": [
    {
      "name": "Impatient customer",
      "patience": 0.2,
      "assertiveness": 0.8,
      "politeness": 0.4,
      "technical_level": 0.3,
      "communication_style": "terse",
      "background": "Wants a refund today"
    }
  ],
  "scenarios": [
    {
      "name": "Refund",
      "goal": "Get a full refund",
      "criteria": [{"description": "Agent asks for the order number", "type": "must_happen"}]
    }
  ],
  "max_turns": 6,
  "llm_config": {"model": "openai/gpt-5.6-luna", "temperature": 0.2},
  "target_reasoning_effort": "low",
  "per_simulation_timeout_s": 300
}

eq sim simulate --config sim.json --json > run.json

eq sim run takes the generation keywords the same way:

{
  "target": "agent:my-agent",
  "num_personas": 4,
  "edge_case_percentage": 0.25,
  "generation_instructions": "customers replying in German",
  "llm_config": {"model": "openai/gpt-5.6-luna"}
}

eq sim run --config gen.json --num-scenarios 3 --json

A flag passed on the command line beats the file, and the file beats the CLI default. The command checks where a value came from, not what it is, so --max-turns 10 wins over "max_turns": 6 even though 10 is the default. A null in the file means "not set" and is accepted only on fields that allow None, such as run_name, max_turns or datapoint_parallelism; the SDK default then applies. A value nobody sets (no flag, no file entry) is never sent: max_turns, datapoint_parallelism and run_name reach the SDK only when you pass them, so --from-run can still restore the replayed cap. "save": false in the file has the effect of --no-save. --llm-config and the file's "llm_config" merge field by field, and --sim-model wins for model.

The target and the input source are each one choice, so a command-line flag replaces the file's whole choice rather than adding a second one. --target, --vercel-url or --openai-model drops the file's "target". On eq sim simulate, --input, --dataset-id, --experiment-id or --from-run drops the file's "datapoints", "personas", "scenarios", "dataset_id", "experiment_id", "experiment_run_id" and "previous_run". --experiment-run-id alone narrows the file's "experiment_id" instead. Two flags for the same choice on the command line are still rejected.

Unknown keys fail the command before anything runs, at every depth, so "num_personas" in a sim simulate file or "patiense" inside a persona is rejected with the field path. "target" takes only the agent:<key> and deployment:<key> string forms; a callable or AgentTarget, user_simulator, judge, hooks and generation_client stay Python-only.

With --json, stdout carries the SimulationRun and nothing else; the run-store save, the report files and every progress line still happen, on stderr.

eq sim schema¶

Print a JSON schema: what --config accepts, or what --json prints.

eq sim schema [--input | --output] [--command simulate|run]

eq sim upload-dataset¶

Upload simulation datapoints to an Orq dataset, or append them to an existing dataset.

eq sim upload-dataset -i cases.jsonl -n "Support simulation set"
eq sim upload-dataset -i more.jsonl --dataset-id <id>

The first command prints the new dataset ID. Use that ID, not the display name, with eq sim simulate --dataset-id or the Python extend_from_dataset() function. Here, upload-dataset --dataset-id adds rows to the stored dataset; extend_from_dataset() generates new cases without changing the stored dataset. Persona and scenario objects are JSON-stringified because the Orq dataset API accepts scalar inputs values. The simulation reader restores them when the dataset is used with eq sim simulate --dataset-id.

eq sim generate¶

Generate simulation datapoints only — no simulation is run.

eq sim generate --datapoints dp.jsonl --agent-description "..."

eq sim from-traces¶

Build simulation datapoints from Orq production traces.

eq sim from-traces --output dp.jsonl --limit 50 --lookback-hours 24
eq sim from-traces --output dp.jsonl --extend 20 --agent-description "..."

Fetches recent traces from the Orq traces API (requires ORQ_API_KEY) and builds one datapoint per trace conversation: persona and scenario are inferred from the transcript, and the first message is the real user's opening message verbatim. Pass --extend N to additionally generate N new datapoints matching the traffic distribution of the fetched traces (extra LLM calls). Feed the output file to eq sim simulate --input to run it.

eq sim export¶

Export simulation results: OpenResponses payload JSON, or an HTML/Markdown report.

eq sim export --input results.jsonl --output payload.json
eq sim export --input sim-report.json --output report.html --format html --recommendations

Markdown/HTML exports include remediation suggestions if the input run JSON already carries them (runs generate them by default — see --no-recommendations), or if --recommendations is passed here to generate them at export time for a run that has none stored.

eq sim validate¶

Validate a simulation datapoints JSONL file.

eq sim validate --input dp.jsonl
eq sim validate dp.jsonl

eq sim validate-dataset (compatibility alias)¶

Deprecated alias for eq sim validate --input PATH. Retained for compatibility.

eq sim validate-dataset dp.jsonl

eq sim runs¶

List recent simulation runs.

eq sim runs [DIRECTORY] [--limit N]