swo-bench docs

Overview

swo-bench replays catalog scenarios or your own captured sessions against an inference server. You get time to first token, inter-token latency, tokens per second, failure rate, and wall-clock runtime.

Use it to compare hardware, tune a serving stack, or prove a deploy before production traffic hits it. Works with vLLM, SGLang, Anthropic Messages, and any server that speaks the OpenAI API.

Start with a smoke test like trivial-qa. Run doctor before a long replay. Move to heavier catalog scenarios or your own recordings when the endpoint looks healthy.

representative-recording copies a smaller subset of a local capture. It needs no license and no network. simulate dynosim is experimental: it turns a replay proxylog into an uncalibrated DynoSim prediction. Use replay or capacity when you need a live measurement.

Install

Requires Python 3.10 or newer. Install from PyPI:

Standard

pip install swo-bench

DynoSim tokenizer extra

pip install "swo-bench[dynosim]"

HTTP/2 extra

pip install "swo-bench[http2]"

swo-bench[deepseek] is a compat alias and installs nothing extra. swo-bench[dynosim] adds Hugging Face transformers for convert-proxylog --tokenizer and simulate dynosim --tokenizer. swo-bench[http2] adds h2. simulate dynosim also needs a local Docker daemon. The first run builds that image.

License

You need a SwarmOne license key. Keys look like swo_live_... or swo_test_.... swo-bench resolves the key in this order:

The --license-key flag
The SWO_LICENSE_KEY environment variable
The ~/.swarmone/license.key file

export SWO_LICENSE_KEY=swo_live_yourkey

BENCH
Run catalog scenarios, doctor, capacity, speed tests, and simulate dynosim on a proxylog you already have.
BENCH_PRO
Everything in BENCH, plus record your own traffic, replay custom recordings, and serve them with fake-llm.

In CI, set SWO_NO_PROMPT=1 so a missing key exits 2 instead of waiting for a prompt.

Quick start

Same sequence as swo-bench --help:

swo-bench list-scenarios

swo-bench scenario-info <name>

swo-bench doctor --endpoint http://localhost:8000/v1 --model <model>

swo-bench replay --scenario trivial-qa --endpoint http://localhost:8000/v1 --model <model>

swo-bench record --output traffic.jsonl --upstream <upstream>

swo-bench replay --recording traffic.jsonl --endpoint <endpoint>

Replace the URL with your server and --model with an id advertised by GET /v1/models. A unique substring is enough when it matches one model, for example -m glm. The record line writes traffic.jsonl. Pass a .msgpack path when you want the compact capture.

Examples

Smoke test a local server

trivial-qa is a small catalog workload. Use it to prove the CLI, license, and endpoint before you spend time on a heavy scenario.

swo-bench replay \
  --scenario trivial-qa \
  --endpoint http://localhost:8000/v1 \
  --model <model> \
  --json-output ./smoke.json

Diagnose the endpoint first

doctor probes DNS, TCP, TLS, auth, GET /v1/models, streaming SSE, token usage, and served-model identity. --full adds three uncached readiness probes.

swo-bench doctor \
  --endpoint http://localhost:8000/v1 \
  --model <model> \
  --api-key "$API_KEY" \
  --full \
  --format json

Catalog replay with copies and concurrency

-r copies each selected task. -c caps in-flight requests. This does not change how many conversations the plan schedules. It only limits load on the server.

swo-bench replay \
  --scenario js-coding-opus \
  --endpoint http://localhost:8000/v1 \
  --model <model> \
  -c 4 -r 10 \
  --verbose

Realistic fleet timing

--auto turns on recorded tool delays, recorded arrivals, even stagger when there are copies, and raises concurrency to the planned peak. Flags you set yourself are kept.

swo-bench replay \
  --scenario js-coding-opus \
  --endpoint http://localhost:8000/v1 \
  --model <model> \
  --auto \
  -r 8 \
  --seed 42

Record live traffic, then replay it

Start the proxy, point the agent at it, press Ctrl+C when done. Prefer .msgpack. Inspect the capture offline, then replay it against any endpoint. Needs BENCH_PRO.

swo-bench record \
  --output traffic.msgpack \
  --upstream http://localhost:8000 \
  --port 19000

# Point the agent at http://localhost:19000
# Then:
swo-bench recording-info --recording traffic.msgpack
swo-bench replay \
  --recording traffic.msgpack \
  --endpoint http://localhost:8000/v1 \
  --model <model>

Record Codex against DeepSeek

record translates Responses and Anthropic traffic for you. Use --no-translate when client and upstream already share the same API.

export SWO_LICENSE_KEY=swo_live_yourkey
export DEEPSEEK_API_KEY=your_deepseek_key
swo-bench record \
  --output traffic.msgpack \
  --upstream https://api.deepseek.com \
  --port 19010

# Point Codex at http://localhost:19010
# Health: curl http://127.0.0.1:19010/health

Fake-LLM replay into a DynoSim trace

Replay against recorded responses (no live model). Write a proxylog, then convert it for NVIDIA DynoSim or the SGLang simulator.

swo-bench replay \
  --recording traffic.msgpack \
  --fake-llm \
  --fake-llm-transcript run.fake.proxylog.jsonl

swo-bench convert-proxylog run.fake.proxylog.jsonl --format mooncake
swo-bench convert-proxylog run.fake.proxylog.jsonl --format agentic_mooncake
swo-bench convert-proxylog run.fake.proxylog.jsonl --format autobench \
  --tokenizer <hf-tokenizer> \
  --recording traffic.msgpack

Find the largest copy count that still hits SLOs

capacity searches repetition count. It requires TTFT p99, ITL p99, failure-rate, and a seed. Do not pass -r or -c. Reproduce a step with replay -r N --seed <step seed>.

swo-bench capacity \
  --scenario trivial-qa \
  --endpoint http://localhost:8000/v1 \
  --model <model> \
  --max-ttft-p99-ms 800 \
  --max-itl-p99-ms 40 \
  --max-failure-rate 0.01 \
  --seed 7 \
  --min-repetitions 1 \
  --max-repetitions 64 \
  --json-output ./capacity.json

Convert a Codex rollout

Each Codex rollout file is one thread. Convert each file separately. Spawned subagents get their own rollouts.

find ~/.codex/sessions -name 'rollout-*.jsonl' | head

swo-bench convert-from-codex \
  --session ~/.codex/sessions/2026/06/16/rollout-<id>.jsonl \
  --output my-session.jsonl

swo-bench replay \
  --recording my-session.jsonl \
  --endpoint http://localhost:8000/v1 \
  --model <model>

Custom multi-task scenario

Bundle local recordings into one manifest. Paths are relative to the JSON file. Use --task to run one task.

{
  "name": "my-scenario",
  "description": "Mixed coding traffic",
  "tasks": [
    { "id": "t1", "name": "Task 1", "recording": "t1.msgpack" },
    { "id": "t2", "name": "Task 2", "recording": "t2.msgpack" }
  ]
}

swo-bench replay \
  --scenario-file scenario.json \
  --endpoint http://localhost:8000/v1 \
  --model <model>

Copy a smaller recording

representative-recording stays on this machine. No license, no SwarmOne connection, no GPU. It writes a plaintext .msgpack subset and never overwrites the source.

swo-bench representative-recording \
  --recording traffic.msgpack \
  --output representative.msgpack \
  --target-token-percent 25 \
  --seed 7 \
  --report representative.json

Experimental uncalibrated simulation

Produce a proxylog with replay --fake-llm, then run simulate dynosim. It converts that proxylog to Mooncake and runs aisimulate predict in Docker. The numbers are uncalibrated. Use replay or capacity for a live measurement. Replaying a custom recording needs BENCH_PRO. simulate dynosim needs a valid license, Docker, and the dynosim extra.

swo-bench replay \
  --recording traffic.msgpack \
  --fake-llm \
  --fake-llm-transcript run.fake.proxylog.jsonl

swo-bench simulate dynosim \
  --proxylog run.fake.proxylog.jsonl \
  --tokenizer <hf-tokenizer> \
  --config predict.yaml \
  --output ./sim

Compare models under one DynoSim config

One Mooncake conversion, then one prediction per model. Hardware and parallelism stay in the predict YAML. --model can be repeated, or pass --models-file. The run writes comparison.json, comparison.md, and a terminal ranking.

swo-bench simulate dynosim \
  --proxylog run.fake.proxylog.jsonl \
  --tokenizer <hf-tokenizer> \
  --config model-matrix.yaml \
  --models-file models.yaml \
  --rank-by throughput_per_gpu \
  --output ./sim-compare

Re-upload saved results

If live backend upload failed, keep the --json-output file and upload it later. --private strips local conversation ids.

swo-bench upload-results ./smoke.json
swo-bench upload-results ./private-run.json --private

list-scenarios

List built-in catalog scenarios with task counts and token sizing. Pick a name for replay --scenario.

Example

swo-bench list-scenarios
swo-bench list-scenarios --format json

Arguments

Common

--license-key

string

SwarmOne license key. Checked in this order: --license-key, SWO_LICENSE_KEY, then ~/.swarmone/license.key.

Advanced

--format

choice, default table

table (default) or json with full scenario data. Choices: table, json.

Table output ellipsizes long Name and Model cells instead of wrapping them.
JSON includes full workload_stats with token distributions.
Use scenario-info <name> for per-task breakdowns and copy-paste replay commands.

scenario-info

Inspect one catalog scenario: tasks, request counts, token distributions, and ready-to-paste replay commands.

Example

swo-bench scenario-info trivial-qa
swo-bench scenario-info js-coding-opus --format json

Arguments

name

argument, required

Scenario id from list-scenarios, for example trivial-qa or js-coding-opus.

Common

--license-key

string

SwarmOne license key. Checked in this order: --license-key, SWO_LICENSE_KEY, then ~/.swarmone/license.key.

Advanced

--format

choice, default text

text for the terminal; json for the full API response. Choices: text, json.

NAME is the id from list-scenarios.
A missing task id is reported as a missing task (exit 4) with the available ids, not as an unknown scenario.

doctor

Probe an OpenAI-compatible endpoint before a long replay. Checks reachability, auth, model listing, streaming, usage, and served-model identity.

Example

swo-bench doctor -e http://localhost:8000/v1 -m <model>
swo-bench doctor -e http://localhost:8000/v1 -m <model> --full -k "$API_KEY"

Arguments

Required

--model, -m

string, required

Model id the endpoint should serve. Matched against GET /v1/models.

Endpoint

--endpoint, -e

string, required

Inference API base URL, for example http://localhost:8000/v1.

Common

--api-key, -k

string

API key for the inference endpoint. Sent as a Bearer token by default.

--api-key-header

string, default Authorization

Header for the API key. Default Authorization (Bearer). Use X-API-Key when the endpoint requires it.

--max-tokens

int, default 8

Max tokens for probe generations. Keep small to bound cost.

--timeout

float, default 15

Per-request timeout in seconds.

--license-key

string

SwarmOne license key. Checked in this order: --license-key, SWO_LICENSE_KEY, then ~/.swarmone/license.key.

Advanced

--format

choice, default text

text for the terminal; json for a machine-readable report. Choices: text, json.

--full

flag, default off

Also run three uncached probes and warn when TTFT/decode is still drifting.

Tool-call checks report not_applicable when the server does not support tools.
--full adds three uncached readiness-stability probes and warns when TTFT or decode is still drifting.
Keep --max-tokens small. This is a probe, not a benchmark.

replay

Replay a catalog scenario or a local recording against your inference endpoint, or against recorded responses with --fake-llm.

Example

swo-bench replay --scenario trivial-qa --endpoint http://localhost:8000/v1 --model <model> -c 4 -r 10

Arguments

Workload source (pick one)

--scenario, -s

string

Name of a built-in catalog scenario. Run 'swo-bench list-scenarios' to see choices.

--task, -t

string

Replay only the task with this id from a multi-task --scenario or --scenario-file. Omit to replay every task.

--recording

string

Path to a local capture created by 'swo-bench record' (.jsonl, .msgpack, .jsonl.zst, or .gz). Replays the recorded conversation against your endpoint. Mutually exclusive with --scenario and --scenario-file.

--scenario-file

string

Path to a local scenario.json manifest for replaying your own single-task or multi-task scenario. Use this instead of --scenario/-s when the scenario is not in the SwarmOne catalog. Minimal format: '{"tasks": [{"id": "t1", "name": "Task 1", "recording": "t1.jsonl"}]}'. Each task requires 'id' and 'recording'; 'name' is optional. Recording paths are relative to the manifest file and may point to JSONL or msgpack recordings. Use --task TASK_ID to replay only one task. Mutually exclusive with --scenario and --recording.

Endpoint

--endpoint, -e

string

Inference API base URL. Supports OpenAI-compatible endpoints, for example http://localhost:8000/v1, and the Anthropic Messages API at https://api.anthropic.com. A full request path such as /v1/chat/completions is accepted and reduced to the /v1 base. Required unless --fake-llm is used.

Common

--fake-llm

flag, default off

Start a local server that returns responses from the selected recording instead of contacting a real inference endpoint. Requires --recording or --scenario-file; use --task when a scenario file contains multiple tasks.

--fake-llm-transcript

path

Write fake-LLM request/response traffic to this capture path. Required with --fake-llm unless the source recording is encrypted.

--fake-llm-timing

choice, default unpaced

Response timing for --fake-llm: unpaced returns recorded responses immediately; recorded reproduces each captured response delay. Choices: unpaced, recorded.

--model, -m

string

Model id to replay against. Required for normal endpoints; --fake-llm infers it when the recording contains exactly one model. The id must be listed by GET /v1/models, so provider aliases that silently resolve to another model are rejected. A unique substring is accepted (for example, '-m glm'). Use --allow-unlisted-model to override the model-list check.

--concurrency, -c

int, default 1

Maximum requests in flight at once (1-256). This limits endpoint load; it does not change how many conversations are scheduled. With --tool-time, set it at or above the preflight planned peak so ready requests are not delayed.

--auto

flag, default off

Realistic fleet defaults: enable --tool-time, --arrival recorded, and --stagger even when there are copies; raise --concurrency to the planned peak. Flags you set explicitly are kept. Tool-time needs recorded response timings, or pass --tool-time-ms N. Replays recorded tool-execution gaps. Does not change --pace.

--max-tokens

int, default 4096

Max output tokens per request.

--max-requests

int

Maximum source turns kept across the entire plan. Turns are kept in conversation order. --repetitions or token-budget copy counts still multiply them. Omit to keep all turns.

--api-key, -k

string

API key for the inference endpoint. Sent as a Bearer token by default.

--api-key-header

string, default Authorization

Header for the API key. Default Authorization (Bearer). Use X-API-Key when the endpoint requires it.

--license-key

string

SwarmOne license key. Checked in this order: --license-key, SWO_LICENSE_KEY, then ~/.swarmone/license.key.

--json-output, -o

path

Write benchmark results as JSON to this file.

--timeout

int, default 300

Total wall-clock budget in seconds for one request, including retries and backoff. Connection failures are not retried.

--inference-retries

int, default 2

Retries for safe failures before any response content arrives: HTTP 408, 429, and 5xx responses; read/write timeouts; and protocol errors. Connection failures are not retried. Total attempts = retries + 1. Set to 0 to disable retries.

--inference-retry-backoff

float, default 0.5

Exponential backoff base in seconds (with jitter), capped at 8s. Short Retry-After values take precedence.

--max-bench-time

int

Wall-clock cap for the run (seconds); drains in-flight work when hit.

--repetitions, -r

int, default 1

Number of copies scheduled for each source conversation. For example, 40 source conversations with --repetitions 4 create 160 replay slots. Multi-task scenarios and scenario files repeat every selected task this many times. With --tool-time, set --concurrency at or above the preflight planned peak. Cannot be combined with --input-tokens or --output-tokens.

--input-tokens

int

Target the sum of recorded prompt_tokens across the replay fleet. v4 only. One or both of --input-tokens/--output-tokens may be set. Cannot be combined with --repetitions other than the default 1.

--output-tokens

int

Target the sum of recorded completion_tokens across the replay fleet. v4 only. One or both of --input-tokens/--output-tokens may be set. Cannot be combined with --repetitions other than the default 1.

--debug

flag, default off

Log HTTP details when requests fail.

Advanced

--passphrase-file

path

File with the decryption passphrase for an encrypted capture. Overrides SWO_BENCH_RECORDING_PASSPHRASE.

--force-fake-llm-transcript

flag, default off

Overwrite an existing --fake-llm-transcript file.

--allow-unlisted-model

flag, default off

Allow a model not listed by GET /v1/models. The served model is still recorded; skew is still warned.

--cache-mode

choice, default realistic

Prefix-cache behavior: realistic (typical reuse), allcold (always cold), allwarm (always warm; needs --history-mode recorded). Choices: realistic, allcold, allwarm.

--history-mode

choice, default faithful

faithful combines recorded structure with live assistant text; recorded replays captured messages; live carries live responses forward. Choices: live, recorded, faithful.

--max-tokens-mode

choice, default flat

flat: one --max-tokens for all turns. recorded-completion: each turn's recorded completion_tokens; missing/zero usage falls back to --max-tokens. Choices: flat, recorded-completion.

--shared-head-chars

int

Set a known shared-prefix length in characters (0-1,000,000); omit to use backend calibration. Use only when the workload's shared prefix is known.

--pace

string, default instant

instant: no wait between turns. 1x/2x/10x: wait after each send using recorded gaps divided by speed. Captured gaps are measured from request send, so slower responses consume the gap. Not tool idle (use --tool-time).

--strip-prefix

choice, default auto

auto: strip unstable prefix fields for KV-cache reuse. none: leave prefixes as recorded. Choices: auto, none.

--thinking-replay

choice, default preserve

preserve: keep captured reasoning fields in history. strip: omit them. Choices: preserve, strip.

--time-scatter

float, default 0.0

Add backend-generated random variation to the request schedule. Higher values spread requests farther apart; 0 disables it. Unlike --stagger and --arrival, this variation is not from the recording.

--stagger

choice, default none

When -r > 1, delay each copy's first request so copies do not all start together. even splits the capture's first-request time span across copies; random shuffles within that span; none is lockstep. Timing only: request bodies stay the same. Shift can be tiny if the capture already started together. Choices: none, even, random.

--arrival

choice, default immediate

When each conversation becomes eligible to start. immediate makes all conversations eligible at once; recorded reproduces their captured first-request offsets. Actual sends still obey --concurrency. With --repetitions greater than 1, --stagger adds a per-copy delay. Choices: immediate, recorded.

--tool-time, --no-tool-time

flag, default off

Requires a latency-bearing, multi-turn recording. Replay captured tool-execution delays when available. The trivial-qa scenario does not contain these delays.

--tool-time-pct

float, default 100

Scale captured tool delays as a percentage. Use 0 to report delay statistics without waiting. Implies --tool-time.

--tool-time-max-ms

int

Cap each replayed tool delay in milliseconds. Implies --tool-time.

--tool-time-ms

int

Fixed tool wait per turn in milliseconds. Parallel tools in one turn share that delay. Implies --tool-time.

--think-time-pct

float, default 0

Scale recorded human think-time gaps by this percentage. A value of 0 skips them. A nonzero value implies --tool-time.

--fail-on-tool-time-clamp-rate

flag, default off

Exit if too many tool gaps were clamped to zero (likely mis-grouped conversations).

--policy

choice, default interleaved_random

Turn order across conversations: interleaved_random (default), round_robin, sequential, or random. Choices: round_robin, sequential, random, interleaved_random.

--seed

int

Seed for replay scheduling randomness. This does not change LLM sampling. Use --extra-body '{"seed":42}' to set an inference sampling seed.

--model-context-length

int

Model context-window size in tokens. Requests that still exceed it after enabled compaction are skipped.

--resolve-model-context, --no-resolve-model-context

flag, default on

Look up the model's context window via GET /v1/models (default: on).

--clamp-max-tokens-to-context, --no-clamp-max-tokens-to-context

flag, default on

Trim max_tokens so each request fits the context window.

--autocompact, --no-autocompact

flag, default off

Compact history when the context window fills. Requires a known context window.

--autocompact-pct

int

Context-window percentage that activates --autocompact (default: 95).

--autocompact-window

int

Override the context-window size used by --autocompact.

--recorded-compaction, --no-recorded-compaction

flag, default on

Honor compaction events captured in the recording. Separate from --autocompact.

--anthropic-beta

string

Comma-separated Anthropic beta flags sent as the anthropic-beta header (e.g. prompt-caching-2024-07-31).

--extra-body

string

JSON merged into each request body (vendor fields, sampling seed, etc.).

--enable-thinking

flag, default off

Shortcut for --extra-body '{"chat_template_kwargs":{"enable_thinking":true}}'.

--max-consecutive-failures

int

Stop the run after N consecutive failures in one slot. Default: no limit.

--verbose, -V

flag, default off

Show live per-task progress with phase, request count, and decode tok/s.

--verbose-text, --VV

flag, default off

Plain-text line per request (no progress bars). For logs and CI.

--json

flag, default off

JSON results on stdout; status on stderr. For piping.

Pick exactly one workload source: --scenario, --recording, or --scenario-file.
--endpoint is required unless --fake-llm is set.
--repetitions copies every selected task. 3 tasks at -r 10 is 30 slots.
--input-tokens and --output-tokens cannot be combined with --repetitions other than the default 1.
--cache-mode allwarm requires --history-mode recorded.
If the URL ends in /chat/completions, /messages, or /responses, swo-bench strips it to the API base so preflight hits /v1/models.
--json-output includes job completion time (JCT): one job is one copy of one catalog task, or one harness agent tree at one copy index.
--json and --json-output also include an occupancy series and a demand_envelope of successful sends. The saved config stamps history_mode, cache_mode, and engine.
Diff two saved files with compare-json-output. The usual pair is the same endpoint with --history-mode faithful and --history-mode recorded.
If a non-Anthropic endpoint rejects a Codex developer role, replay merges the leading developer text into one system message, turns any later developer message into a user message, and keeps that rewrite for the rest of the run.

capacity

Find the largest repetition count that still meets TTFT p99, ITL p99, and request-level failure-rate SLOs. Geometric ramp, then binary search over copies.

Example

swo-bench capacity --scenario trivial-qa --endpoint http://localhost:8000/v1 --model <model> --max-ttft-p99-ms 800 --max-itl-p99-ms 40 --max-failure-rate 0.01 --seed 7

Arguments

Workload source (pick one)

--scenario, -s

string

Name of a built-in catalog scenario. Run 'swo-bench list-scenarios' to see choices.

--task, -t

string

Replay only the task with this id from a multi-task --scenario or --scenario-file. Omit to replay every task.

--recording

string

--scenario-file

string

Required

--max-ttft-p99-ms

float, required

Required TTFT p99 SLO in milliseconds. Omit to exit 4.

--max-itl-p99-ms

float, required

Required ITL p99 SLO in milliseconds. Omit to exit 4.

--max-failure-rate

float, required

Required request-level failure-rate SLO (0..1). Omit to exit 4.

--seed

int, required

Required command seed for scheduling. Omit to exit 4.

Capacity

--min-repetitions

int, default 1

Smallest repetition count to test.

--max-repetitions

int, default 64

Largest repetition count to test (max 4096).

--confirmations

int, default 2

Extra runs at nominated pass/fail points. 0 trusts discovery.

--min-measured-requests

int, default 100

Early-stop floor only; a finished step still scores below this.

Endpoint

--endpoint, -e

string

Common

--json-output, -o

path

Write the capacity report JSON to this file.

--json

flag, default off

Print the capacity report JSON once at process end.

--fake-llm

flag, default off

--fake-llm-transcript

path

Write fake-LLM request/response traffic to this capture path. Required with --fake-llm unless the source recording is encrypted.

--fake-llm-timing

choice, default unpaced

Response timing for --fake-llm: unpaced returns recorded responses immediately; recorded reproduces each captured response delay. Choices: unpaced, recorded.

--model, -m

string

--max-tokens

int, default 4096

Max output tokens per request.

--max-requests

int

Maximum source turns kept across the entire plan. Turns are kept in conversation order. --repetitions or token-budget copy counts still multiply them. Omit to keep all turns.

--api-key, -k

string

API key for the inference endpoint. Sent as a Bearer token by default.

--api-key-header

string, default Authorization

Header for the API key. Default Authorization (Bearer). Use X-API-Key when the endpoint requires it.

--license-key

string

SwarmOne license key. Checked in this order: --license-key, SWO_LICENSE_KEY, then ~/.swarmone/license.key.

--timeout

int, default 300

Total wall-clock budget in seconds for one request, including retries and backoff. Connection failures are not retried.

--inference-retries

int, default 2

--inference-retry-backoff

float, default 0.5

Exponential backoff base in seconds (with jitter), capped at 8s. Short Retry-After values take precedence.

--debug

flag, default off

Log HTTP details when requests fail.

Advanced

--passphrase-file

path

File with the decryption passphrase for an encrypted capture. Overrides SWO_BENCH_RECORDING_PASSPHRASE.

--force-fake-llm-transcript

flag, default off

Overwrite an existing --fake-llm-transcript file.

--allow-unlisted-model

flag, default off

Allow a model not listed by GET /v1/models. The served model is still recorded; skew is still warned.

--cache-mode

choice, default realistic

Prefix-cache behavior: realistic (typical reuse), allcold (always cold), allwarm (always warm; needs --history-mode recorded). Choices: realistic, allcold, allwarm.

--history-mode

choice, default faithful

faithful combines recorded structure with live assistant text; recorded replays captured messages; live carries live responses forward. Choices: live, recorded, faithful.

--max-tokens-mode

choice, default flat

flat: one --max-tokens for all turns. recorded-completion: each turn's recorded completion_tokens; missing/zero usage falls back to --max-tokens. Choices: flat, recorded-completion.

--shared-head-chars

int

Set a known shared-prefix length in characters (0-1,000,000); omit to use backend calibration. Use only when the workload's shared prefix is known.

--pace

string, default instant

--strip-prefix

choice, default auto

auto: strip unstable prefix fields for KV-cache reuse. none: leave prefixes as recorded. Choices: auto, none.

--thinking-replay

choice, default preserve

preserve: keep captured reasoning fields in history. strip: omit them. Choices: preserve, strip.

--time-scatter

float, default 0.0

Add backend-generated random variation to the request schedule. Higher values spread requests farther apart; 0 disables it. Unlike --stagger and --arrival, this variation is not from the recording.

--stagger

choice, default none

--arrival

choice, default immediate

--tool-time, --no-tool-time

flag, default off

Requires a latency-bearing, multi-turn recording. Replay captured tool-execution delays when available. The trivial-qa scenario does not contain these delays.

--tool-time-pct

float, default 100

Scale captured tool delays as a percentage. Use 0 to report delay statistics without waiting. Implies --tool-time.

--tool-time-max-ms

int

Cap each replayed tool delay in milliseconds. Implies --tool-time.

--tool-time-ms

int

Fixed tool wait per turn in milliseconds. Parallel tools in one turn share that delay. Implies --tool-time.

--think-time-pct

float, default 0

Scale recorded human think-time gaps by this percentage. A value of 0 skips them. A nonzero value implies --tool-time.

--fail-on-tool-time-clamp-rate

flag, default off

Exit if too many tool gaps were clamped to zero (likely mis-grouped conversations).

--policy

choice, default interleaved_random

Turn order across conversations: interleaved_random (default), round_robin, sequential, or random. Choices: round_robin, sequential, random, interleaved_random.

--model-context-length

int

Model context-window size in tokens. Requests that still exceed it after enabled compaction are skipped.

--resolve-model-context, --no-resolve-model-context

flag, default on

Look up the model's context window via GET /v1/models (default: on).

--clamp-max-tokens-to-context, --no-clamp-max-tokens-to-context

flag, default on

Trim max_tokens so each request fits the context window.

--autocompact, --no-autocompact

flag, default off

Compact history when the context window fills. Requires a known context window.

--autocompact-pct

int

Context-window percentage that activates --autocompact (default: 95).

--autocompact-window

int

Override the context-window size used by --autocompact.

--recorded-compaction, --no-recorded-compaction

flag, default on

Honor compaction events captured in the recording. Separate from --autocompact.

--anthropic-beta

string

Comma-separated Anthropic beta flags sent as the anthropic-beta header (e.g. prompt-caching-2024-07-31).

--extra-body

string

JSON merged into each request body (vendor fields, sampling seed, etc.).

--enable-thinking

flag, default off

Shortcut for --extra-body '{"chat_template_kwargs":{"enable_thinking":true}}'.

--max-consecutive-failures

int

Stop the run after N consecutive failures in one slot. Default: no limit.

--verbose, -V

flag, default off

Show live per-task progress with phase, request count, and decode tok/s.

--verbose-text, --VV

flag, default off

Plain-text line per request (no progress bars). For logs and CI.

Rejected on this command

These flags are the search axis or always-on behavior. Passing them exits 4:

-r / --repetitions
-c / --concurrency
--input-tokens
--output-tokens
--max-bench-time

Required: --max-ttft-p99-ms, --max-itl-p99-ms, --max-failure-rate, --seed, and exactly one workload source.
--auto is always on. Concurrency is sized to the planned peak after each plan.
Reproduce a nominated step with: swo-bench replay -r N --seed <step seed>.
Transport, TLS, or auth failures abort the search. They are not counted as overload.
Ctrl+C keeps whatever the report already knows when you pass -o / --json.
Hosted result upload is off for capacity.

record

Record inference traffic through a local proxy for later replay. Requires a BENCH_PRO license.

Example

swo-bench record --output traffic.msgpack --upstream http://localhost:8000

Arguments

Required

--output, -o

path, required

Recording file path. Use .msgpack (compact default). .jsonl is legacy. Replay with: swo-bench replay --recording FILE ...

--upstream, -u

string, required

Upstream inference URL to forward to (e.g. https://api.example.com/v1).

Common

--model, -m

string

Override the model id sent upstream (translating mode only). Omit to keep the client's model. DeepSeek defaults to deepseek-chat.

--license-key, -l

string

BENCH_PRO license key. Checked in this order: --license-key, SWO_LICENSE_KEY, then ~/.swarmone/license.key.

--debug

flag, default off

Log HTTP details when requests fail.

Advanced

--host

string, default 127.0.0.1

Bind address (default 127.0.0.1). Use 0.0.0.0 for LAN clients.

--port, -p

int, default 19000

Local proxy port. Point the client at http://localhost:PORT.

--upstream-timeout

float, default 300.0

Upstream request timeout in seconds (translating mode).

--upstream-retries

int, default 2

Retry failed upstream requests (translating mode).

--upstream-retry-backoff

float, default 0.25

Wait time between retries, in seconds (translating mode).

--upstream-connect-retries

int, default 8

Retries when the upstream connection fails to open (translating mode).

--reasoning-audit-output

string

Path for the reasoning-config audit log (translating mode; default: <output>.reasoning.jsonl).

--no-translate

flag, default off

Forward requests unchanged. No Anthropic/Responses to Chat Completions translation. Use when client and upstream already share the same API.

--also-jsonl

flag, default off

With --output NAME.msgpack, also write legacy NAME.jsonl.

--session-header

string

Header name for session id (repeatable). Checked before built-in names (x-session-id, x-claude-code-session-id, etc.). Use for custom harnesses.

--agent-header

string

Header name for per-agent id (repeatable). Checked before built-in Claude/Codex headers. Required for multi-agent replay when the harness sends its own ids.

--parent-agent-header

string

Header name for parent agent id (repeatable). Checked before built-in Claude/Codex parent headers. Use for subagent trees in custom harnesses.

--no-proxy-log, --proxy-log

flag, default no-proxy-log

Passthrough mode only. --proxy-log writes raw HTTP events to NAME.proxylog.jsonl beside the recording. Default is --no-proxy-log (off).

--raw-archive

path

Optional path for a raw HTTP event log (full request/response bodies). Used by fake-LLM replay; normal recordings store normalized turns only.

--inject-session-header

flag, default off

If the client sends no session header, generate x-session-id, forward it upstream, and echo it on the response. Off by default.

Point the client at http://localhost:PORT. The proxy forwards to --upstream.
Verify with: curl http://127.0.0.1:PORT/health
record probes the upstream before starting and exits 3 if it is unreachable.
Prefer NAME.msgpack. NAME.jsonl is legacy. --also-jsonl dual-writes both, each with its own disk cap.
Use --no-translate for byte passthrough when Anthropic/Responses translation is not needed.

recording-info

Summarize a local recording offline. No license and no SwarmOne connection. Shows entry counts, turns, models, time span, and conversation grouping.

Example

swo-bench recording-info --recording traffic.msgpack
swo-bench recording-info --recording traffic.jsonl.zst --format json

Arguments

Workload source (pick one)

-o, --recording

path, required

Path to a local recording (.jsonl, .msgpack, compressed, or encrypted).

Advanced

--format

choice, default text

text summary or json. Choices: text, json.

--passphrase

string

Passphrase for an encrypted recording. Overrides SWO_BENCH_RECORDING_PASSPHRASE. Mutually exclusive with --passphrase-file.

--passphrase-file

path

Passphrase file for an encrypted recording. Else SWO_BENCH_RECORDING_PASSPHRASE, else an interactive prompt. Mutually exclusive with --passphrase.

Recordings with agent or thread ids report separate conversations.
Older captures without those ids are grouped by session only.
Claude Code's untagged main session (shared session_id, no agent_id) is treated as the tree root.
Encrypted captures take --passphrase or --passphrase-file. Pass only one. Otherwise SWO_BENCH_RECORDING_PASSPHRASE, then an interactive prompt.

representative-recording

Copy a representative subsequence of a local recording. Fully local: no license, no SwarmOne connection, and no GPU.

Example

swo-bench representative-recording --recording traffic.msgpack --output representative.msgpack --target-token-percent 25 --seed 7

Arguments

Required

--recording

path, required

Local recording (.msgpack, .jsonl, compressed, or encrypted).

--output

path, required

Destination .msgpack path. Always a rebuilt plaintext chunklog. Must differ from --recording.

--target-token-percent

float, required

Percent of eligible source tokens to keep. Greater than 0 and at most 100. 100 copies every eligible session.

--seed

int, required

Selection seed. The same source and seed produce the same output.

Advanced

--force

flag, default off

Replace --output and --report if they already exist. Never replaces --recording.

--report

path

Optional JSON report path. Omit this to write no report file.

--passphrase-file

path

Passphrase file for an encrypted --recording. Else SWO_BENCH_RECORDING_PASSPHRASE, else an interactive prompt.

--output must be a new .msgpack path. This command never overwrites --recording.
100 keeps every eligible session. A lower percent drops whole sessions, not individual turns.
Subagents with no captured root are omitted, including at 100%.
A single-session file cannot be thinned below that tree. Two large sessions can land well above the requested percent.
An encrypted source is decrypted to a temp file and deleted. The subset is written as plaintext msgpack.

fake-llm

Run a local OpenAI-compatible server on 127.0.0.1 that answers with recorded responses. Requires a BENCH_PRO license. Press Ctrl+C to stop.

Example

swo-bench fake-llm --recording traffic.msgpack --transcript run.fake.proxylog.jsonl --port 19100

Arguments

Workload source (pick one)

--recording

path, required

Recording to serve (.msgpack or exchange/event JSONL).

Required

--transcript

path, required

Write all served traffic to this event JSONL path (file must not already exist).

Common

--model

string

Only serve exchanges for this model id (required when the recording mixes models).

--license-key

string

SwarmOne license key. Checked in this order: --license-key, SWO_LICENSE_KEY, then ~/.swarmone/license.key.

Advanced

--port

int, default 19100

Listen port on 127.0.0.1.

--loop

flag, default off

When a client runs out of recorded turns, replay from the first turn again.

--timing

choice, default unpaced

unpaced: respond immediately. recorded: wait to match each exchange's captured total delay. Choices: unpaced, recorded.

Needs BENCH_PRO. Catalog-only licenses cannot serve a custom recording.
Point clients at http://127.0.0.1:PORT/v1.
--transcript must not already exist.
For in-process fake replay without a server, prefer: swo-bench replay --fake-llm --recording ...
Inspect conversation counts first with recording-info.

convert-proxylog

Convert a fake-LLM proxylog JSONL into a DynoSim Mooncake trace or an SGLang Autobench trace.

Example

swo-bench convert-proxylog run.fake.proxylog.jsonl --format mooncake
swo-bench convert-proxylog run.fake.proxylog.jsonl --format autobench --tokenizer <hf-tokenizer> --recording traffic.msgpack

Arguments

proxylog

argument, required

Path to the fake-LLM proxylog JSONL written by replay --fake-llm --fake-llm-transcript.

Advanced

-o, --output

path

Output trace JSONL. Default is <stem>.mooncake.jsonl, <stem>.agentic_mooncake.jsonl, or <stem>.autobench.jsonl.

--format

choice, default mooncake

Trace format. mooncake: DynoSim, one serial session per harness tree. agentic_mooncake: DynoSim, one session per agent conversation with wait_for and branches. autobench: SGLang simulator, token-id prompts and millisecond timestamps. Choices: mooncake, agentic_mooncake, autobench.

--block-size

int

Tokens per hash_ids block for mooncake and agentic_mooncake (default: 512).

--min-turns

int, default 1

Drop sessions with fewer than N paired turns.

--tokenizer

string

HuggingFace tokenizer. Required for autobench. Optional for mooncake and agentic_mooncake hash_ids. Needs pip install 'swo-bench[dynosim]'.

--recording

path

Required with --format autobench. Capture (.msgpack or JSONL) joined by matched_recorded_index.

--timestamps-from

choice, default recording

Autobench clock. recording uses the original capture span in milliseconds. proxylog uses the compressed fake-LLM clock. Choices: recording, proxylog.

mooncake: one serial session per harness tree (root plus every subagent).
agentic_mooncake: one session per agent conversation, with wait_for and branches so concurrent subagents stay concurrent.
autobench: SGLang simulator rows with prompt token ids, prompt_len, output_len, and a millisecond timestamp. Requires --tokenizer and --recording. Open-loop, with no wait_for.
--timestamps-from recording (the default) uses the original capture span. proxylog uses the compressed fake-LLM clock.
Missing identity fails the whole convert rather than writing a partial DAG, except Claude Code's untagged main session when the same tree has tagged agents.
--tokenizer needs: pip install 'swo-bench[dynosim]'.

simulate dynosim

Experimental uncalibrated DynoSim simulation. Converts a replay proxylog to Mooncake and runs aisimulate predict. Use replay or capacity for a live measurement.

Example

swo-bench simulate dynosim --proxylog run.fake.proxylog.jsonl --tokenizer <hf-tokenizer> --config predict.yaml --output ./sim

Arguments

Required

--proxylog

path, required

Proxylog JSONL from swo-bench replay, usually --fake-llm --fake-llm-transcript.

--tokenizer

string, required

HuggingFace tokenizer id or path for Mooncake hashing. Needs pip install "swo-bench[dynosim]".

--config

path, required

aisimulate predict YAML. swo-bench rewrites traffic.source onto the generated Mooncake trace and leaves the rest unchanged.

--output

path, required

Directory for simulation artifacts. Created if missing. Must be writable.

Common

--license-key

string

SwarmOne license key. Also SWO_LICENSE_KEY or ~/.swarmone/license.key.

Advanced

--timeout

int, default 1800

Engine run timeout in seconds.

--set

string

Repeatable aisimulate override applied after --config, as PATH=YAML. Later assignments win, including over the file.

--model

string

Repeatable HuggingFace model id. Compare these models under the same --config. Each replaces engine.model.

--models-file

path

YAML with a models list of ids or {id, label, overrides} mappings. Per-model overrides are PATH: value pairs.

--jobs

int

Model comparison: simulations to run in parallel. Default is the CPU count, capped at the model count.

--rank-by

choice, default throughput_per_gpu

Model comparison metric that picks the recommended model. Choices: throughput_per_gpu, throughput, ttft_p99, itl_p99, e2e_p99.

--strict

flag, default off

Model comparison: exit non-zero if any model could not be simulated.

swo-bench simulate prints engine choices. DynoSim flags live on swo-bench simulate dynosim.
Pass a proxylog from swo-bench replay, usually --fake-llm --fake-llm-transcript.
A SwarmOne license is required: --license-key, SWO_LICENSE_KEY, or ~/.swarmone/license.key.
Needs Docker. The first run, and any pin change, builds swo-bench-dynosim:1.5.0. That image runs aisimulate predict --stack dynamo with ai-dynamo 1.5.0. AISimulate comes from GitHub tag v0.12.0.
--tokenizer needs: pip install 'swo-bench[dynosim]'.
--config is an aisimulate predict YAML. swo-bench points traffic.source at the generated Mooncake trace. Mooncake block size is traffic.source.block_size (default 512). Repeat --set PATH=YAML to override the file, for example --set router.policy=kv_router.
A single run writes trace.mooncake.jsonl, predict.yaml, manifest.json (including proxylog_sha256), prediction.json, report.json, and simulator.log.
Repeat --model, or pass --models-file, to compare models under one --config: one conversion, one prediction each, then comparison.json, comparison.md, and a terminal table. Per-model reports land in candidates/. A model that cannot be simulated is listed with a reason. The run exits non-zero if every model fails, and also if --strict is set and any model fails.
--jobs is how many of those predictions run at once. Default is the CPU count, capped at the model count. --rank-by defaults to throughput_per_gpu.
HF_TOKEN and HUGGING_FACE_HUB_TOKEN are forwarded into the container.

convert-from-codex

Turn a Codex CLI rollout log into a swo-bench recording you can replay. No live proxy needed when you already have the file.

Example

swo-bench convert-from-codex --session ~/.codex/sessions/2026/06/16/rollout-<id>.jsonl --output my-session.jsonl

Arguments

--session, -i

path, required

Codex rollout file to convert (one thread per file). See ~/.codex/sessions/.

--output, -o

path, required

Path for the replay recording JSONL to write.

Codex sessions live under ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl.
Each rollout file is one thread. Convert each file separately.
Find a session with: find ~/.codex/sessions -name 'rollout-*.jsonl' | head

describe

Print or embed a markdown description on a recording so later you know what the capture is.

Example

swo-bench describe --recording traffic.msgpack --text "# Session notes\nRecorded from prod."
swo-bench describe --recording traffic.msgpack

Arguments

Workload source (pick one)

-o, --recording

path, required

Recording file (.jsonl or .msgpack).

Advanced

--text

string

Markdown to embed (replaces existing). Omit to print.

Omit --text to print the current description.
Passing --text replaces any existing description.

upload-results

Re-upload a saved --json-output file when live backend upload failed during replay.

Example

swo-bench upload-results ./results.json
swo-bench upload-results ./private-run.json --private

Arguments

results-file

argument, required

JSON file from swo-bench replay --json-output. Must include session_id, timings, and wall_clock_ms.

Common

--license-key

string

SwarmOne license key. Checked in this order: --license-key, SWO_LICENSE_KEY, then ~/.swarmone/license.key.

Advanced

--private

flag, default off

Omit local conversation IDs from uploaded timings. Use for private/encrypted replays. Keeps opaque hosted IDs when present.

The file must include session_id, timings, and wall_clock_ms.
HTTP 409 means the hosted report is already stored. Do not retry.

compare-json-output

Diff two replay --json-output files turn by turn. Use it on the same endpoint with --history-mode faithful and --history-mode recorded.

Example

swo-bench compare-json-output faithful.json recorded.json
swo-bench compare-json-output a.json b.json --allow-live

Arguments

results-a

argument, required

First replay --json-output file. Printed deltas are B minus A.

results-b

argument, required

Second replay --json-output file, compared against results-a.

Advanced

--allow-live

flag, default off

Allow a file whose history_mode is live. Live history diverges in structure from faithful and recorded.

Turns join on (source_task_id, copy_ordinal, conversation_bind, turn_idx). Deltas are B minus A.
Each turn prints prompt tokens, cache hit and miss tokens, TTFT, latency, and status. When both files have jobs[], job completion time is included.
Both files must stamp history_mode, cache_mode, source, endpoint, model, and concurrency. Re-run older results with a current swo-bench if config is missing.
Exit 4 when cache_mode differs, allwarm is paired with faithful, history_mode is live without --allow-live, or endpoint, model, concurrency, or recording source differ. A different engine also fails when both files stamp one. Mismatched turn or job keys exit 4.
Local files only. No license.

version

Print the installed swo-bench version and check PyPI for updates.

Example

swo-bench version
swo-bench --version

Arguments

This command takes no arguments.

Shows upgrade instructions when a newer version is on PyPI.
The root CLI also accepts --version.

Workflows

Run a catalog scenario

SwarmOne ships agent workloads you can run immediately. List them, inspect one, then replay it.

swo-bench list-scenarios
swo-bench scenario-info js-coding-opus
swo-bench replay --scenario js-coding-opus --endpoint http://localhost:8000/v1 --model <model> -c 4 -r 10

Record your own session

Capture how the agent talks to a model, then replay that traffic on any server. Needs BENCH_PRO. Prefer .msgpack. .jsonl is legacy.

swo-bench record --output traffic.msgpack --upstream http://localhost:8000
swo-bench recording-info --recording traffic.msgpack
swo-bench replay --recording traffic.msgpack --endpoint http://localhost:8000/v1 --model <model>

Serve recorded responses locally

For in-process fake replay, prefer replay --fake-llm. fake-llm starts a long-lived server on 127.0.0.1 for other clients. Both need BENCH_PRO when the source is a custom recording.

swo-bench fake-llm \
  --recording traffic.msgpack \
  --transcript serve.fake.proxylog.jsonl \
  --port 19100

Thin a capture, then simulate it

representative-recording is local and needs no license. Replaying that subset with --fake-llm needs BENCH_PRO and writes a proxylog. simulate dynosim then runs an uncalibrated DynoSim prediction. It needs a valid license, Docker, and swo-bench[dynosim]. Use replay or capacity for a live measurement.

swo-bench representative-recording \
  --recording traffic.msgpack \
  --output representative.msgpack \
  --target-token-percent 25 \
  --seed 7

swo-bench replay \
  --recording representative.msgpack \
  --fake-llm \
  --fake-llm-transcript representative.proxylog.jsonl

swo-bench simulate dynosim \
  --proxylog representative.proxylog.jsonl \
  --tokenizer <hf-tokenizer> \
  --config predict.yaml \
  --output ./sim

Configuration

Every CLI flag can also be set as an environment variable with the SWO_ prefix. --endpoint becomes SWO_ENDPOINT. --model becomes SWO_MODEL.

SWO_LICENSE_KEY

License key, if you did not pass --license-key or a license file.

SWO_ENDPOINT

Inference API base URL. Same as -e / --endpoint.

SWO_NO_PROMPT

Set to 1 in CI so a missing license exits 2 instead of prompting.

SWO_BENCH_RECORDING_PASSPHRASE

Passphrase for an encrypted capture. --passphrase-file overrides this.

Hidden --backend-url exists on several commands for a private SwarmOne API host. Contact support if you need to override service URLs or timeouts.

Exit codes

Success

General error

Authentication or license error

Network error

Configuration error, including unknown flags and missing required options

Scenario not found

Support

Questions, license upgrades, or something not working?

Email: [email protected]

Talk to the team: Contact

PyPI: pypi.org/project/swo-bench