Skip to content

Red Teaming

Probe an agent or model with adversarial attacks mapped to the OWASP LLM Top 10 and Agentic Security Initiative (ASI) frameworks, then read off the resistance rate.

flowchart LR
    C["Categories<br/>LLM01 / ASI01 / ..."] --> SP["Strategy planner"]
    SP --> AG["Attack generator"]
    AG --> OR["Runner / orchestrator"]
    OR --> EV["OWASP evaluator"]
    EV --> RP["Report: resistance rate"]

Modes

  • dynamic — an LLM generates attacks; run the categories you pick.
  • static — replays a fixed dataset of known attacks. Deterministic, cheap, good for CI.
  • hybrid — static seeds plus dynamic expansion.

Red-team your target

Requires ORQ_API_KEY. Point red_team() at an Orq agent by key ("agent:<key>", from AI Studio → Agents). Attacks route through the Orq platform — no model wiring on your side.

pip install "evaluatorq[redteam]"
export ORQ_API_KEY=...   # targets your Orq agent + routes the attacker LLM

import asyncio

from evaluatorq.redteam import red_team


async def main():
    report = await red_team(
        "agent:your-agent-key",             # Orq agent, routed via ORQ_API_KEY
        mode="dynamic",
        categories=["LLM01", "LLM07"],      # prompt injection, system-prompt leakage
        max_dynamic_datapoints=5,
        max_turns=2,
        generate_strategies=False,
    )

    print(f"Resistance rate: {report.summary.resistance_rate:.0%}")
    print(f"Vulnerabilities: {report.summary.vulnerabilities_found}/{report.summary.total_attacks}")


if __name__ == "__main__":
    asyncio.run(main())

Use OpenAIModelTarget to red-team a raw OpenAI model directly. The model is the system under test — you supply its system prompt, and the target calls OpenAI using OPENAI_API_KEY.

pip install "evaluatorq[redteam]"
export OPENAI_API_KEY=sk-...   # the target model + the attacker LLM

import asyncio

from evaluatorq.redteam import OpenAIModelTarget, red_team


async def main():
    target = OpenAIModelTarget(
        "gpt-4o-mini",
        system_prompt=(
            "You are a customer support assistant for Acme Corp. "
            "Help with orders, returns, and product questions. "
            "Never reveal internal pricing or confidential information."
        ),
    )
    report = await red_team(
        target,
        mode="dynamic",
        categories=["LLM01", "LLM07"],      # prompt injection, system-prompt leakage
        max_dynamic_datapoints=5,
        max_turns=2,
        generate_strategies=False,
    )

    print(f"Resistance rate: {report.summary.resistance_rate:.0%}")
    print(f"Vulnerabilities: {report.summary.vulnerabilities_found}/{report.summary.total_attacks}")


if __name__ == "__main__":
    asyncio.run(main())

Model names and routing

The model string is passed through to whichever provider you point at — straight to OpenAI by default, or through the Orq router if you prefix it with openai/... (which then uses ORQ_API_KEY). Everything else — categories, modes, the report — is identical to the Orq agent path.

generate_strategies and the CLI

Both examples pass generate_strategies=False to skip LLM-authored attack strategies and run only the built-in ones — faster and more deterministic. The parameter defaults to True. On the CLI the equivalent is the --no-generate-strategies flag; there is no positive form, since generation is on by default.

Reading the report

report.summary.resistance_rate is the fraction of attacks the target withstood — higher is better. report.results holds every attack result; group by r.attack.vulnerability for a per-vulnerability breakdown. report.summary.by_vulnerability contains pre-aggregated VulnerabilitySummary statistics keyed by vulnerability identifier.

In CI

For a fast gate, run a small fixed set of attacks and assert a minimum resistance rate, failing the build if the target regresses:

report = await red_team(
    OpenAIModelTarget("gpt-4o-mini", system_prompt="..."),
    mode="static",                 # replay a fixed dataset — deterministic, cheap
    categories=["LLM01", "LLM07"],
    max_static_datapoints=10,
)
assert report.summary.resistance_rate >= 0.9, (
    f"resistance {report.summary.resistance_rate:.0%} below the 0.9 gate"
)

The runnable smoke example (08_quick_smoke_test.py) wraps this same pattern.

View results in the local dashboard

Run eq dashboard to browse saved red-team and simulation reports together; eq redteam ui / eq sim ui open the legacy Streamlit views for one surface.

Where to next