Skip to content

Configuration

All configuration is via environment variables. No config file is required.

Environment variables

Variable Required? Default What it does
ORQ_API_KEY Required for Orq features — Authenticates against the Orq platform. Required to fetch datasets, upload results, and invoke deployments. Also auto-enables OpenTelemetry tracing (spans are sent to https://my.orq.ai/v2/otel).
ORQ_BASE_URL No https://my.orq.ai Overrides the Orq API base URL. Affects Orq SDK calls (dataset fetch, deployment invocation) and the derived OTLP tracing endpoint (<ORQ_BASE_URL>/v2/otel). Does not redirect OpenAI-compatible LLM calls — use OPENAI_BASE_URL for that.
ROUTER_BASE_URL Deprecated — Predecessor to ORQ_BASE_URL, no longer honoured. If set (and ORQ_BASE_URL is unset) it only logs a warning. Use ORQ_BASE_URL instead.
OPENAI_API_KEY Red-team / sim only, if not using Orq — API key for the OpenAI (or compatible) backend. Used by the red teaming pipeline and agent simulation when ORQ_API_KEY is absent. Not required for core evaluatorq() evaluation.
OPENAI_BASE_URL No OpenAI default Redirect OpenAI-compatible calls to a different host (vLLM, OpenRouter, Azure, local). Honoured by the red teaming and simulation LLM client.
ORQ_DISABLE_TRACING No unset Set to 1 or true to suppress all OpenTelemetry spans even when ORQ_API_KEY or OTEL_EXPORTER_OTLP_ENDPOINT is present.
ORQ_DEBUG No unset Set to any non-empty value to print tracing setup diagnostics to stdout (endpoint, auth headers, initialization errors).
EQ_DEBUG No unset Set to any non-empty value to show the full traceback on CLI errors instead of the one-line message. CLI-wide; distinct from ORQ_DEBUG, which only affects tracing diagnostics.
EVALUATORQ_DIR No .evaluatorq in the current directory Base directory for the run store, where both red teaming (runs/) and simulation (sim-runs/) persist reports. Must point at the store directory itself (e.g. /tmp/x/.evaluatorq), not its parent — only the working-directory fallback appends .evaluatorq. Empty is treated as unset.
EVALUATORQ_LOG_LEVEL No INFO Log level for the dashboard server. Accepts any level name (e.g. DEBUG).
ORQ_WORKSPACE / ORQ_WORKSPACE_SLUG No unset Workspace slug used to build dashboard deep-links into the Orq UI. ORQ_WORKSPACE wins when both are set. When neither is set, the deep-link buttons are hidden. See Dashboard.
ORQ_UI_BASE_URL No ORQ_BASE_URL, else https://my.orq.ai Base URL for dashboard deep-links into the Orq UI. Set this when the UI host differs from the API host.
OTEL_EXPORTER_OTLP_ENDPOINT No — Explicit OTLP HTTP endpoint. Takes precedence over the ORQ_BASE_URL-derived endpoint. See Tracing.
OTEL_EXPORTER_OTLP_HEADERS No — Comma-separated key=value pairs added to every OTLP export request. Format: key1=value1,key2=value2.
OTEL_SERVICE_NAME No evaluatorq Service name recorded on every span's service.name resource attribute.
OTEL_SERVICE_VERSION No 1.0.0 Service version recorded on every span's service.version resource attribute.
EVALUATORQ_CAPTURE_MESSAGE_CONTENT No true Set to false or 0 to strip LLM message content (prompts and responses) from spans. Token counts, model name, and latency are still recorded. Useful when exporting to third-party backends or to avoid capturing PII.
EVALUATORQ_SPAN_MAX_TEXT_CHARS No unset (no limit) Maximum characters per span text attribute. Set a positive integer (e.g. 8192) to truncate long strings. Unset or 0 / -1 means capture all.
EVALUATORQ_LLM_TIMEOUT_S No 60.0 Per-LLM-call timeout in seconds. Simulation only — has no effect on red teaming or core evaluation. Increase for slow self-hosted endpoints.
EVALUATORQ_LLM_MAX_TOKENS No 8192 Maximum completion tokens per LLM call. Simulation only — has no effect on red teaming or core evaluation. Increase for reasoning models that exhaust the default budget before emitting a tool call.
EVALUATORQ_REASONING_EFFORT No medium Reasoning effort hint passed to reasoning-capable models. Simulation only — has no effect on red teaming or core evaluation. Set to "", none, or off to omit the parameter entirely.

.env file

The library itself does not call load_dotenv(). The examples ship with python-dotenv calls in their scripts. To load a .env file in your own code, call load_dotenv() before importing evaluatorq:

from dotenv import load_dotenv

load_dotenv()  # must run before evaluatorq reads env vars

from evaluatorq import evaluatorq, DataPoint

A minimal .env for Orq platform use:

ORQ_API_KEY=your_orq_api_key_here

With OpenAI as the LLM backend (red teaming / simulation, no Orq):

OPENAI_API_KEY=sk-...

Self-hosted LLM endpoint:

OPENAI_API_KEY=dummy
OPENAI_BASE_URL=http://localhost:8000/v1

To send traces to a custom OTLP collector instead of Orq:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

Where to next