livekit-agent-simulator

livekit-agent-simulator

Dial any LiveKit voice agent with an AI simulated caller — WebRTC room, inbound SIP, or outbound SIP — and keep a full forensic log. Standalone MCP server + CLI (lks, alias: lk-sim). Black-box testing: no imports from the agent under test, no edits to its code or .env.

curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

Install via coding agent (copy-paste)

Paste into Claude Code, Cursor, Codex, AmpCode, Windsurf, or any coding agent from the repo you want to test:

Install and configure livekit-agent-simulator (CLI: lks) for this project by following the instructions here:
https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.md

Target project root is this workspace. Use absolute --root paths. Install the portable CLI if missing, run lks init, help fill .agent-sim/config.yaml from my local env or ask me for LiveKit + Gemini + agent_name, ensure .agent-sim is gitignored, run preflight, and stop before execute if the voice agent worker is not running. Do not edit agent application source outside .agent-sim/.

Same idea, one line:

Install and configure livekit-agent-simulator by following: https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.md

TL;DR

The Problem

Voice agents fail in ways unit tests never see:

The Solution

livekit-agent-simulator drives a Gemini Live persona from scenario JSONL over one of three transport modes (Caller.mode), observes transcripts / tools / flow / room events, and writes a timestamped report you can play back.

Why Use lks?

Quick Example

# Install once
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

# In the repo you want to test (agent worker must already be running)
lks init --root /path/to/target
# edit /path/to/target/.agent-sim/config.yaml  (LiveKit + Gemini keys, agent_name)

lks preflight --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks report <run-id> --root /path/to/target
lks web --root /path/to/target          # Ctrl+C to stop

Design Philosophy

1. The agent under test is a black box. We never import or patch target application code. Dispatch metadata is opaque JSON.

2. Generic core, target-owned config. Language, timezone, topics, and business strings belong in the target’s .agent-sim/ — not hardcoded in the package.

3. One ops layer for CLI and MCP. execute_* validates then runs. No “run vs execute” forks.

4. Forensics over vibes. Every run produces structured events you can compare, log, and play back.

5. CI-friendly gates. Hard fails on status / assert / script; optional strict judge for softer LLM scoring.

How It Works

1. Read <target>/.agent-sim/config.yaml
2. Pick SimLeg from scenario Caller.mode (webrtc_sim | inbound_sip | outbound_human_pickup | outbound_sim_callee | agent_dials)
3. Connect leg → LiveKit room(s) / SIP hairpin as needed; Gemini stays WebRTC in the sim room
4. Bridge audio; observe transcripts, tools, timing, interruptions
5. Write reports/<run-id>/ + runs.sqlite
6. Optional LLM judge vs PassCriteria

                    Caller.mode (scenario)
         ┌───────────────┬────────────────┬──────────────────┬────────────────────┐
         │  webrtc_sim   │  inbound_sip   │   outbound_human_pickup   │ outbound_sim_callee│
         │  room audio   │  sim dials DID │ human answers → │  Gemini SIP callee │
         │               │                │ Gemini colocated│  (2-room hairpin)  │
         └───────┬───────┴────────┬───────┴────────┬─────────┴─────────┬──────────┘
                 │                │                │                   │
                 └────────────────┼────────────────┼───────────────────┘
                                  ▼
                    ┌──────────────────────────┐
                    │  Gemini Live persona     │
                    │  + LiveKit agent (black  │
                    │    box under test)       │
                    └────────────┬─────────────┘
                                 │ observe
                                 ▼
                    reports/<run-id>/ · runs.sqlite · judge

Mode details and config: docs/telephony.md. Templates: inbound-caller-sim, outbound-human-pickup, outbound-callee-sim.

How lks Compares

When to use lks:

• Regression suites for LiveKit voice agents
• Agent-driven CI / coding-agent workflows (MCP)
• Debugging turn-taking, tools, and silence without reading agent source

When it might not be ideal:

• Pure text chatbots with no LiveKit room
• Offline environments without LiveKit + Gemini API access

Installation

Quick install (recommended)

Download only — no uv/pip/build on your machine. CI ships a portable pack (embedded Python + deps + report player).

# macOS / Linux
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

# Windows PowerShell
irm "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.ps1" -OutFile "$env:TEMP\lk-sim-install.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\lk-sim-install.ps1" -Verify

Also available from a release asset:

curl -fsSL "https://github.com/quangdang46/livekit-agent-simulator/releases/download/v0.1.0/install.sh" \
  | bash -s -- --verify

By default the installer registers the MCP server livekit-agent-simulator (lks mcp) into detected tools: Claude Code, Cursor, Cline, Windsurf, VS Code Copilot, Gemini CLI, Amazon Q, OpenCode, Codex, Warp.

Agent-oriented install playbook (long form): docs/guide/installation.md Raw URL for paste into agents: https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.md

From source (maintainers / contributors)

git clone https://github.com/quangdang46/livekit-agent-simulator.git
cd livekit-agent-simulator
uv sync --extra dev
uv run lks --help

Requires Python 3.10–3.13.

Web UI (maintainers)

Users never build this — CI packs web/dist into the wheel as web_static. Source: web/.

pnpm --dir web install
pnpm --dir web build                    # → web/dist/ (attached by Hatch on uv build)
pnpm --dir web dev                      # HMR; proxy /api + /runs → lks web :8765

See web/README.md.

Quick Start

# Agent worker must be running and registered with LiveKit
lks guide
lks init --root /path/to/target
# fill .agent-sim/config.yaml

lks preflight --root /path/to/target
lks scenario-init smoke-hello --root /path/to/target   # if needed
lks validate smoke-hello --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks runs --root /path/to/target
lks report <run-id> --root /path/to/target
lks web --root /path/to/target

Minimal scenario (smoke-hello)

{"apiVersion":"agent-sim/v1","kind":"Scenario","metadata":{"id":"smoke-hello","locale":"en-US","tags":["smoke"]}}
{"kind":"Persona","spec":{"name":"Alex","brief":"First-time caller; confirm you reached the right place, then end politely.","goals":["Hear the agent","Say you will call back"],"style":"polite, brief"}}
{"kind":"Execute","spec":{"max_turns":2,"timeout_s":90,"first_speaker":"user"}}
{"kind":"PassCriteria","spec":{"criteria":["The agent responded to the caller","The agent responded in the caller's language"]}}

Optional multi-judge PassCriteria: judges[] + mode (all | majority | any). Assert highlights (tool_order, constraint_respected, recovery/latency): lks guide.

Full-line // comments in scaffolded JSONL are guides — runtime ignores them.

Configuration

Target-only data lives under <target>/.agent-sim/ (gitignored). Created by init.

See template: templates/config.yaml. Consumer-specific wiring: docs/portability.md.

Commands

CLI and MCP share the same public ops (ops.py). Prefer execute (validate then run).

lks execute smoke-hello --root /path/to/target
lks execute-all --tag smoke --root /path/to/target
lks log <run-id> --root /path/to/target
lks compare <run-a> <run-b> --root /path/to/target
lks compare <baseline> <candidate> --baseline --root /path/to/target
lks web --port 8765 --root /path/to/target

Every MCP tool needs project_root except guide.

MCP config examples

Installer writes this when tools are detected. Manual Cursor:

{
  "mcpServers": {
    "livekit-agent-simulator": {
      "command": "lks",
      "args": ["mcp"],
      "env": {}
    }
  }
}

Dev checkout (package not installed globally):

{
  "mcpServers": {
    "livekit-agent-simulator": {
      "command": "uv",
      "args": ["run", "--directory", "/abs/path/livekit-agent-simulator", "lks", "mcp"]
    }
  }
}

Equivalent one-shot entry: lks-mcp (alias: lk-sim-mcp; same process as lks mcp / lk-sim mcp).

Architecture

src/livekit_agent_simulator/
├── cli.py / mcp_server.py     # thin surfaces
├── ops.py                     # shared public ops
├── run_orchestrator.py        # room lifecycle + run
├── scenario.py                # JSONL parse / validate
├── config.py                  # .agent-sim/config.yaml
├── preflight.py
├── asserts.py / suite.py      # CI gates
├── gemini/                    # Live caller + judge
├── livekit/                   # room, dispatch, observe
├── audio/ · script/ · plugins/
└── web/                       # report player server

CI / Release

# Local check
uv sync --extra dev
pnpm --dir web build
uv run pytest -q

# Release (pre-1.0 may force-retag a single 0.1.0)
git tag v0.1.0
git push origin v0.1.0

Troubleshooting

preflight fails connectivity

lks preflight --root /path/to/target
# Confirm livekit.url / api_key / api_secret and that the project is reachable.
# Skip API check while editing config:
lks preflight --no-connectivity --root /path/to/target

Agent never joins the room

• Worker process must be running and registered with the same livekit.agent_name.
• Increase livekit.agent_join_timeout_ms if cold start is slow.
• Check dispatch metadata is valid JSON string if your worker requires it.

Gemini / simulator auth errors

Set simulator.google_api_key in .agent-sim/config.yaml. The sim caller uses Gemini Live (gemini-3.1-flash-live-preview by default).

No audio in report player

With observe.record_audio enabled (default true): reports/<run-id>/conversation.wav

lks web --root /path/to/target

MCP tools not listed

lks mcp   # must be what the host launches
# or reinstall without --no-mcp
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

Scenario validation errors

lks validate my-case --root /path/to/target
lks scenario-init my-case --root /path/to/target   # fresh scaffold with // guides

Limitations

What lks Doesn't Do (Yet)

• Not an agent framework — it tests agents; it does not implement business tools
• Not offline-first — needs LiveKit + Gemini (or configured backends)
• Not a load generator — one simulated caller per run (batch via execute-all)

Known Limitations

FAQ

Does it modify my agent repo?

Only scaffolds .agent-sim/ (gitignored). It does not edit agent source.

CLI vs MCP — which should I use?

Same ops. Use CLI in terminals/CI; MCP inside coding agents. Prefer execute_* over ad-hoc run paths.

How do I pass project-specific dispatch fields?

livekit.dispatch_metadata or scenario Dispatch.spec.metadata as an opaque JSON string. Core does not parse consumer keys. See docs/portability.md.

Can I assert on tool calls?

Yes — Assert.spec.tools, tool_order (required tool.start subsequence), observe.tool_event_patterns, Script/assert plugins, and/or PassCriteria + judge. See docs/plugins.md and lks guide.

Where are reports stored?

<target>/.agent-sim/reports/<run-id>/ plus runs.sqlite under .agent-sim/.

Is the report player separate?

No — lks web serves the prebuilt player from the install pack. Maintainers build from web/.

Docs

About Contributions

Please don't take this the wrong way, but I do not accept outside contributions for any of my projects. I simply don't have the mental bandwidth to review anything, and it's my name on the thing, so I'm responsible for any problems it causes; thus, the risk-reward is highly asymmetric from my perspective. I'd also have to worry about other "stakeholders," which seems unwise for tools I mostly make for myself for free. Feel free to submit issues, and even PRs if you want to illustrate a proposed fix, but know I won't merge them directly. Instead, I'll have Claude or Codex review submissions via gh and independently decide whether and how to address them. Bug reports in particular are welcome. Sorry if this offends, but I want to avoid wasted time and hurt feelings. I understand this isn't in sync with the prevailing open-source ethos that seeks community contributions, but it's the only way I can move at this velocity and keep my sanity.

License

MIT