Is your agent behaving like you expect?¶
KickstartAI TechDays · 15 October 2026. Build one refund agent in orq.ai, simulate customer conversations, improve its instructions, and prove the result with a regression eval.
Start with onboarding Cheat sheet
How the pieces fit¶
Your chat UI and simulation run on your laptop and call the AI Gateway. orq.ai runs the managed refund agent's model loop and its three code tools, then returns the answer and tool results. Every call leaves a trace in the Studio. Your coding agent can also use the orq.ai MCP server to work with your workspace.
Requirements¶
You need a laptop with Python 3.11–3.13, git, make, and uv. Install the orq.ai CLI for inspection and quick API reproduction.
# Clone once. `make setup` runs `uv sync --all-groups`.
git clone https://github.com/orq-ai/kickstarterai-techdays-workshop-2026
cd kickstarterai-techdays-workshop-2026
make setup
make setup installs the pinned workshop runtime from pyproject.toml:
| Package | Why |
|---|---|
orq-ai-sdk |
Create and version orq.ai resources; invoke managed agents from Python |
openai |
OpenAI-compatible client used for router and evaluatorq calls |
evaluatorq |
Simulations, dataset jobs, scorers, and eq dashboard |
You do not need to install these packages individually with pip; uv uses the repository lockfile for the tested versions. Your orq.ai key, project, workspace, and seat prefix arrive by email on the day after you scan the QR code on the first slide.
Official resources¶
- orq.ai documentation: Agents, AI Gateway, tools, evaluators, experiments, and observability
- orq.ai Python SDK: the
orq_ai_sdkpackage used by this workshop - orq.ai CLI reference: install and command reference
- evaluatorq documentation: simulations, jobs, evaluators, and dashboards
- evaluatorq on GitHub: source, examples, and issue tracker
Learning path¶
| Module | Focus | What you do | Page |
|---|---|---|---|
| Onboarding | Gateway access and traces | Make one model request and inspect its trace | Onboarding |
| Coding agents (optional) | Assistant setup | Connect orqi, Claude Code, or another MCP client when you need one | Coding agents |
| Build a Managed Agent | Managed agent fundamentals | Run hello, seed the refund agent, finish the Python call, inspect its trace | Build a Managed Agent |
| Simulation | Conversation evidence | Define cases and inspect judged transcripts | Simulation |
| Improve and Evaluate | Versioned improvement | Publish, replay, simulate, and gate a new version | Improve and Evaluate |
First-hour checkpoint: one gateway request with a trace, a working hello agent, and a refund agent whose Python client prints tool results and a trace ID. You do not need the coding-agent installation, a second model, the chat UI, or all three API interfaces to reach that checkpoint. Install dependencies and prepare your key before the session if possible.
Go further¶
For broader platform topics (AI Gateway, guardrails, tracing, knowledge bases, MCP, trace insights, and red teaming), see the official orq.ai documentation.