# evaluatorq > Run LLM evaluations, red-team agents, and simulate multi-turn conversations in Python. This is the curated docs index. For the complete docs as one Markdown file, use [llms-full.txt](llms-full.txt). ## Getting started - [Home](https://orq-ai.github.io/evaluatorq/index.md): Overview, why evaluatorq, and installation. - [FAQ](https://orq-ai.github.io/evaluatorq/faq/index.md): Common questions about red teaming agents and how evaluatorq answers them. - [Get Started](https://orq-ai.github.io/evaluatorq/guides/getting-started/index.md): Run your first evaluation end to end. ## Guides - [Agent Simulation](https://orq-ai.github.io/evaluatorq/guides/agent-simulation/index.md): Multi-turn, persona-driven conversation testing with an LLM judge. - [Overview](https://orq-ai.github.io/evaluatorq/guides/red-teaming/index.md): Adaptive OWASP-mapped adversarial security testing for agents. - [Custom Evaluators & Frameworks](https://orq-ai.github.io/evaluatorq/custom-evaluators-and-frameworks/index.md): Define custom evaluators and frameworks. - [LLM as a Jury](https://orq-ai.github.io/evaluatorq/llm-as-a-jury/index.md): Multi-judge panel (jury) evaluation. - [Pairwise Judging](https://orq-ai.github.io/evaluatorq/pairwise-judging/index.md): Pairwise comparison judging. - [Evaluator Template Variables](https://orq-ai.github.io/evaluatorq/evaluator-template-variables/index.md): Reference for pointwise and pairwise jury template variables. - [Dashboard](https://orq-ai.github.io/evaluatorq/dashboard/index.md): Browse red team and simulation reports in a local web dashboard. ## Setup - [Configuration](https://orq-ai.github.io/evaluatorq/configuration/index.md): Configuration and environment variables. - [Tracing](https://orq-ai.github.io/evaluatorq/tracing/index.md): OpenTelemetry tracing. - [Orq Deployment](https://orq-ai.github.io/evaluatorq/orq-deployment/index.md): Running against Orq deployments. ## API reference - [Introduction](https://orq-ai.github.io/evaluatorq/api-reference/index.md): Introduction to the Python API surface. - [evaluatorq](https://orq-ai.github.io/evaluatorq/reference/evaluatorq/index.md): The evaluatorq package API surface. - [integrations](https://orq-ai.github.io/evaluatorq/reference/evaluatorq/integrations/index.md) - [openresponses](https://orq-ai.github.io/evaluatorq/reference/evaluatorq/openresponses/index.md) - [redteam](https://orq-ai.github.io/evaluatorq/reference/evaluatorq/redteam/index.md) - [simulation](https://orq-ai.github.io/evaluatorq/reference/evaluatorq/simulation/index.md) - [tracing](https://orq-ai.github.io/evaluatorq/reference/evaluatorq/tracing/index.md) - [Overview](https://orq-ai.github.io/evaluatorq/cli-reference/overview/index.md): CLI commands (evaluatorq / eq) overview. - [Red Teaming](https://orq-ai.github.io/evaluatorq/cli-reference/redteam/index.md): CLI reference for the eq redteam command. - [Simulation](https://orq-ai.github.io/evaluatorq/cli-reference/simulation/index.md): CLI reference for the eq sim command. ## Examples - [Basic Simulation](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/01_basic_simulation/index.md) - [ORQ Deployment Simulation](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/02_orq_deployment_simulation/index.md) - [Tool Simulation](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/03_tool_simulation/index.md) - [Hardening Loop](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/04_hardening_loop/index.md) - [Wrap And Experiment](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/05_wrap_and_experiment/index.md) - [LangGraph Simulation](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/06_langgraph_simulation/index.md) - [OpenAI Agents Simulation](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/07_openai_agents_simulation/index.md) - [PydanticAI Simulation](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/08_pydantic_ai_simulation/index.md) - [CrewAI Simulation](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/09_crewai_simulation/index.md) - [Build Pdf](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/webinar_demo/build_pdf/index.md) - [External Agent](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/webinar_demo/external_agent/index.md) - [Provision](https://orq-ai.github.io/evaluatorq/examples/agent_simulation/webinar_demo/agent_build/provision/index.md) - [Eval Reuse](https://orq-ai.github.io/evaluatorq/examples/lib/basics/eval_reuse/index.md) - [LLM Eval With Results](https://orq-ai.github.io/evaluatorq/examples/lib/basics/llm_eval_with_results/index.md) - [Pass Fail Simple](https://orq-ai.github.io/evaluatorq/examples/lib/basics/pass_fail_simple/index.md) - [Simple Local Eval](https://orq-ai.github.io/evaluatorq/examples/lib/basics/simple_local_eval/index.md) - [Example Cosine Similarity](https://orq-ai.github.io/evaluatorq/examples/lib/cli/example_cosine_similarity/index.md) - [Example LLM](https://orq-ai.github.io/evaluatorq/examples/lib/cli/example_llm/index.md) - [Example Using CLI](https://orq-ai.github.io/evaluatorq/examples/lib/cli/example_using_cli/index.md) - [Example Using CLI Two](https://orq-ai.github.io/evaluatorq/examples/lib/cli/example_using_cli_two/index.md) - [Country Dataset Eval](https://orq-ai.github.io/evaluatorq/examples/lib/datasets/country_unit_test/index.md) - [Dataset Example](https://orq-ai.github.io/evaluatorq/examples/lib/datasets/dataset_example/index.md) - [LangChain Integration Example](https://orq-ai.github.io/evaluatorq/examples/lib/integrations/langchain/langchain_integration_example/index.md) - [LangGraph Integration Example](https://orq-ai.github.io/evaluatorq/examples/lib/integrations/langchain/langgraph_integration_example/index.md) - [LangGraph Research Eval](https://orq-ai.github.io/evaluatorq/examples/lib/integrations/langchain/langgraph_research_eval/index.md) - [Path Organization](https://orq-ai.github.io/evaluatorq/examples/lib/structured/path_organization/index.md) - [Structured Rubric Eval](https://orq-ai.github.io/evaluatorq/examples/lib/structured/structured_rubric_eval/index.md) - [Structured Safety Eval](https://orq-ai.github.io/evaluatorq/examples/lib/structured/structured_safety_eval/index.md) - [Structured Sentiment Eval](https://orq-ai.github.io/evaluatorq/examples/lib/structured/structured_sentiment_eval/index.md) - [Basic Dynamic](https://orq-ai.github.io/evaluatorq/examples/redteam/01_basic_dynamic/index.md) - [Static Dataset](https://orq-ai.github.io/evaluatorq/examples/redteam/02_static_dataset/index.md) - [Hybrid Mode](https://orq-ai.github.io/evaluatorq/examples/redteam/03_hybrid_mode/index.md) - [Filter Categories](https://orq-ai.github.io/evaluatorq/examples/redteam/04_filter_categories/index.md) - [Custom LLM Client](https://orq-ai.github.io/evaluatorq/examples/redteam/05_custom_llm_client/index.md) - [Multi Target](https://orq-ai.github.io/evaluatorq/examples/redteam/06_multi_target/index.md) - [Report Inspection](https://orq-ai.github.io/evaluatorq/examples/redteam/07_report_inspection/index.md) - [Quick Smoke Test](https://orq-ai.github.io/evaluatorq/examples/redteam/08_quick_smoke_test/index.md) - [Custom Hooks](https://orq-ai.github.io/evaluatorq/examples/redteam/09_custom_hooks/index.md) - [OpenAI Backend](https://orq-ai.github.io/evaluatorq/examples/redteam/10_openai_backend/index.md) - [Redteam Config](https://orq-ai.github.io/evaluatorq/examples/redteam/11_redteam_config/index.md) - [Vulnerability Filter](https://orq-ai.github.io/evaluatorq/examples/redteam/12_vulnerability_filter/index.md) - [Attacker Instructions](https://orq-ai.github.io/evaluatorq/examples/redteam/13_attacker_instructions/index.md) - [Recommendations And Artifacts](https://orq-ai.github.io/evaluatorq/examples/redteam/14_recommendations_and_artifacts/index.md) - [Tool Chaining](https://orq-ai.github.io/evaluatorq/examples/redteam/15_tool_chaining/index.md) - [LLM As A Jury](https://orq-ai.github.io/evaluatorq/examples/redteam/16_llm_as_a_jury/index.md) - [LangGraph Target](https://orq-ai.github.io/evaluatorq/examples/redteam/17_langgraph_target/index.md) - [OpenAI Agents Target](https://orq-ai.github.io/evaluatorq/examples/redteam/18_openai_agents_target/index.md) - [PydanticAI Target](https://orq-ai.github.io/evaluatorq/examples/redteam/19_pydantic_ai_target/index.md) - [CrewAI Target](https://orq-ai.github.io/evaluatorq/examples/redteam/20_crewai_target/index.md) - [Compare](https://orq-ai.github.io/evaluatorq/examples/redteam/crypto_stealing_demo/compare/index.md) - [Run](https://orq-ai.github.io/evaluatorq/examples/redteam/crypto_stealing_demo/run/index.md) - [Run Redteam](https://orq-ai.github.io/evaluatorq/examples/redteam/refund_agent_demo/agent_build/run_redteam/index.md) ## About - [Contributing](https://orq-ai.github.io/evaluatorq/contributing/index.md): How to contribute. - [Changelog](https://orq-ai.github.io/evaluatorq/changelog/index.md): Release notes. - [Roadmap](https://orq-ai.github.io/evaluatorq/roadmap/index.md): Planned work.