benchwarmers

Put me in, coach.Every model on your bench gets its minutes.

Benchwarmers is a self-hosted panel gateway for LLMs. Your app asks for oneof your collections, and the whole panel answers: the models you pickedfor that kind of work, plus explorer seats that rotate in the models youhave used least. Every answer comes back side by side, labelled by model andseat. You check them against the real work and record a verdict, and thoseverdicts decide who sits on the next panel.

What it is best at:

• Asking several models at once. One request, in the OpenAI, Anthropic,Gemini or Ollama format, a panel of answers. Where they disagree is whereto look.
• Agents too. Claude Code and other tool-calling clients use a collectionthrough route mode: one member per turn, kept for the conversation, withfailover, and every call on the record.
• Reviewing. Code review, claims about code, designs: different modelscatch different things, and two models agreeing is not proof.
• Playing to each model's strengths. A collection per kind of work(reviewers, designers, enumerators), each with the models that do it best.
• Giving every model playing time. Explorers cycle through the rest ofthe bench, so new and free models get tried beside the ones you trust,without you choosing them by hand.
• Benchmarking on your own projects. A review suite seeded from your ownrepositories, plus your verdicts on real work, instead of public leaderboards.
• Knowing who to bench. Who is confidently wrong, who has stoppedanswering (retired automatically), and who is resting after a rate limit.
• Keeping free and cheap models useful. Health probes, rate limits takenfrom what providers say and enforced, and one endpoint for every app, overany OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.

Contents

• Try it
• How it works
• Asking a panel
• From the command line
• From Claude Code and other agents
• The suite
• Docker and hosting
• What leaves your machine
• Limits
• Security
• Development
• Licence

Day-to-day use (the dashboard's pages, connecting apps, onboarding in bulk) isin USAGE.md. Running it on a server or a hosting platform is inHOSTING.md. A coding agent setting it up for someone should startwith AGENTS.md.

Try it

v0.3, early. It runs with Python 3.11 or later on Windows, macOS andLinux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read thecode where these pages are thin.

Quickest, with Python 3.11+ installed, from PyPI:

pipx install benchwarmers        # or: uv tool install benchwarmers
benchwarmers                     # or try it first, without installing: uvx benchwarmers

or from npm, with Node 18+ as well:

npx @wendellresearch-ai/benchwarmers

Either way you get the same two commands, benchwarmers and its short namebench. It keeps your data in your user profile (BENCHWARMERS_HOME tochoose another folder), sets it up on the first run, and opens the dashboardon http://localhost:8122, where steps 3 to 5 below continue. The built-inmodel client makes every call, so nothing else needs installing. Nodependencies either way: the app is the Python standard library.

From source:

1. Get the code and a Python 3.11+ environment:

   git clone https://github.com/WendellResearch-AI/benchwarmers
   cd benchwarmers
   python -m venv .venv             # nothing to pip install: the standard library is enough
   

   The built-in model client (metrics/client.py) makes every call. Thellm CLI is an alternative for its plugins:install it in the venv and choose it on Settings (or BENCH_CLIENT=llm,which wins).

2. Make a data folder. Everything that is yours (endpoints, modelregistrations, collections, verdicts, probe history, task answers) livesthere, never in the code:

   python scripts/init.py           # or: python scripts/init.py <folder>
   

   It starts the folder from examples/ (in your OS's per-user data folderunless you name one) and points the checkout at it with a.benchwarmers-home file; BENCHWARMERS_HOME overrides that. With--endpoint NAME BASE_URL --key it also adds a first endpoint and asks forits key, which goes to the key file in llm's user folder (keys.json,shared with the llm CLI if you have it, not needed otherwise), never intothe data folder. BENCHWARMERS_KEY_<NAME> in the environment works too.

3. Start the dashboard. With no endpoint yet it opens on adding one: signin with a provider on its own page (OpenRouter, NanoGPT, Hugging Face,Pollinations, your ChatGPT plan, GitHub Models, or a local Ollama), or pastean API key for any OpenAI-compatible API, a hub such as Open WebUI orLiteLLM included. Anthropic and Gemini keys can also be called throughthose providers' own APIs (the API choice in the same dialog). Only providers that publish a sign-in for other apps areoffered; adding one is an entry in metrics/connect.py (its docstring sayshow).

   python serve.py                  # http://localhost:8122
   

4. Onboard a few models from it ("Onboard a model"): tick one or several;each is registered, probed and run through the suite, one at a time, withthe progress beside the button. Quick probes now and leaves the suiteto the scheduled passes; Quick + ad hoc also keeps the bench from evercalling the model on its own, for a metered or trial key.

5. Make a collection on the Collections page (the models a panel alwaysincludes, and what it is for). The dashboard answers the gateway's routes onits own port (http://127.0.0.1:8122/v1); python gateway.py runs thegateway alone on http://127.0.0.1:8123/v1.

How it works

• Collections and seats. A collection names the models a panel alwaysincludes and the shape of work it is for (review, design, enumeration...).Each call adds explorer seats chosen by how long a model has gone unused, sothe bench keeps learning about models a router would stop asking.
• Verdicts. Each seat's answer carries a call id. Once you have checked ananswer against the real work, record whether it was useful, and which of itsclaims were real or false. Verdicts, not vibes, decide the next panel; amodel that is often confidently wrong shows as that.
• The catalog. Context window, output limit, tools, vision and price comefrom models.dev for the provider that actually serves a model, thenOpenRouter's public catalog and FreeLLMAPI, and anything you set by hand.
• Health. Every morning each model gets a one-word prompt; once a week, ashort and a long one for speed. A model that stops answering, or that itsendpoint no longer serves, is retired automatically after one confirmingcall, with the evidence kept, and can be restored in one click. Thebuilt-in scheduler runs all of this once Settings switches it on; a freshinstall spends no calls until then.
• Rate limits and usage. Limits come from providers' rate-limit headersand error messages, and from what you set per connection or model(requests a minute, day or month; tokens a minute or day). They are kept perconnection, since models behind one key share them, and enforced: a callwaits when the wait fits, or is refused before it goes out. Every call landsin a monthly usage ledger with its tokens, and the Quotas page shows what isleft and when it resets.
• Discover. models.dev and FreeLLMAPI, merged into one list of modelsworth adding, each with the routes this install has to it.

Asking a panel

Any OpenAI-compatible client works: ask for the model bench/<collection>.

curl http://127.0.0.1:8122/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "bench/<collection>", "messages": [{"role": "user", "content": "Where can this function return a stale value? ..."}]}'

The reply carries every seat's answer in the message, and the same seats asdata in a bench field: model, seat, status, seconds and a call id. For a chatwindow that shows one answer, ask for bench/<collection>:merged: a mergermodel writes one answer saying where the seats agreed and disagreed, and everyseat is still in bench. Record a verdict per call once you have checked it:

curl http://127.0.0.1:8122/bench/verdict -H "Content-Type: application/json" \
  -d '{"call": "<call id>", "useful": true, "note": "found the real bug", "real": ["..."], "false": ["..."]}'

or from the dashboard. See GATEWAY.md for the full contract:images, prompt sizes, who may call, and the reply's shape.

Other clients. The same panel answers in other APIs' shapes, so a clientthat speaks one of them works with the server's address as its base URL:

curl http://127.0.0.1:8122/v1/messages -H "Content-Type: application/json" \
  -d '{"model": "bench/<collection>", "max_tokens": 1024, "messages": [{"role": "user", "content": "..."}]}'

Each streams when asked: the reply starts at once, and each seat's answerarrives as that seat finishes. A panel answers with text, so a requestcarrying tools is refused with a 400 that points to route mode(bench/<collection>:route, one member with its tools; seebelow). GATEWAY.md, "Other API formats",has a curl for each and what is refused or ignored.

From the command line

bench (or benchwarmers, the same program) does what the dashboard does,without it: ask and judge, and run the bench itself. It is a client of arunning server, so the same commands work against one on another machine.

bench status                                   what is running, and where
bench collections                              what you can ask
bench ask core "Where can this return a stale value?" -f cache.py
git diff | bench ask core "Review this diff"   a pipe is the prompt
bench verdict <call id> --useful --real "the race in refresh()" --false "the SQL injection"

bench models --find mercury                    the roster, with each model's status
bench candidates --new                         what your endpoints serve that is not on the bench
bench onboard <served name> --wait             onboard it (registered, probed, scored)
bench model retire <alias> --reason "..."      and back: bench model restore <alias>
bench collection save reviewers --members a,b,c --description "adversarial review"
bench collection save reviewers --add d --remove a
bench endpoint add openrouter --base https://openrouter.ai/api/v1   (asks for the key, hidden)
bench key create my-app --collections core --rpm 30
bench settings scheduler run --off perf

--help on any command lists its options with examples, and --json printswhat the server answered, for scripts. --url points one command at anotherserver. ask prints every seat's answer on stdout and the table of call ids tojudge them by on stderr, so bench ask core "..." > review.md keeps only theanswers. Retiring a model, deleting a collection or an endpoint and revoking akey ask first; --yes answers for a script. A key is never an argument (otherson the machine can read those): endpoint add prompts for it, or reads itfrom stdin with --key-stdin.

Another program called bench (the Frappe framework's) may come first on yourPATH; type benchwarmers instead, which takes the same commands.

Exit codes: 0 done, 1 refused or failed (the reason on stderr), 2 bad usage, 3no server at BENCHWARMERS_URL.

From Claude Code and other agents

A running Benchwarmers is an MCP server at /mcp on its own port(http://127.0.0.1:8122/mcp; /v1/mcp works too), over streamable HTTP,with three tools: list_collections, ask (a collection, a prompt,optionally merged, with and system) and verdict. Nothing to start perclient: every agent session shares the one server, behind the same Host,Origin and gateway-key checks as /v1. The tool descriptions carry therules: ask a collection, not a favourite model; verify before acting; nameevery seat; send a verdict per seat you judged.

Claude Code plugin. The plugin connects to that /mcp and adds a/benchwarmers:ask <collection> <what to ask> command and a benchwarmersskill saying when a panel helps (adversarial review, exhaustive enumerationover real files) and how to judge it:

/plugin marketplace add WendellResearch-AI/benchwarmers
/plugin install benchwarmers@benchwarmers

It finds the server through BENCHWARMERS_URL (defaulthttp://127.0.0.1:8122; set it if you changed BENCH_PORT) and sendsBENCHWARMERS_KEY where keys are required. It also adds a Stop hook: atthe end of a turn it asks the server (the pending_verdicts tool, so nothingextra runs on your machine) which seats of that session's panels have noverdict yet, and reminds Claude of them once, as hook feedback. The serverfinds the session's panels in its transcript, so the reminder needs a serveron the same machine; with one elsewhere the hook stays silent. A verdict sentany way counts, the dashboard's included. With the plugin enabled and noBenchwarmers running, the hook shows a hook-error notice at the end of eachturn, as /mcp shows the server as failed; start the server, or disable theplugin for that session.

Cursor and Antigravity plugins. The same skill, with the stdio serverbelow (so they need Node and Python 3.11 or later): Cursor's ison cursor.directory, andAntigravity's installs from the repository:

agy plugin install https://github.com/WendellResearch-AI/benchwarmers

Any other MCP client that speaks HTTP:

{"mcpServers": {"benchwarmers": {"type": "http", "url": "http://127.0.0.1:8122/mcp"}}}

with "headers": {"Authorization": "Bearer <gateway key>"} where keys arerequired. A client that only speaks stdio: run benchwarmers mcp onceinstalled, uvx benchwarmers mcp ornpx -y @wendellresearch-ai/benchwarmers mcp without installing, or python mcp_server.py from acheckout, a small process per client that calls the same server. For example:

{"mcpServers": {"benchwarmers": {"command": "npx", "args": ["-y", "@wendellresearch-ai/benchwarmers", "mcp"],
                                 "env": {"BENCHWARMERS_URL": "http://127.0.0.1:8122"}}}}

On native Windows, a client that cannot start npx itself needs"command": "cmd", "args": ["/c", "npx", "-y", "@wendellresearch-ai/benchwarmers", "mcp"].

Behind an MCP aggregator (1mcp, MetaMCP, mcp-proxy, Docker's MCP Gatewayand the like): add http://127.0.0.1:8122/mcp as an HTTP (streamable HTTP)upstream, with the Authorization header where keys are required; anaggregator that only runs stdio servers takes benchwarmers mcp as above.Give it a request timeout of a few minutes or more, as a panel takes thatlong; 1mcp's default of 60 seconds ends most of them. With 1mcp, for example:

1mcp mcp add benchwarmers --type http --url http://127.0.0.1:8122/mcp --timeout 900000

The aggregator prefixes the tool names (1mcp's arebenchwarmers_1mcp_ask and so on). If you also use the Claude Code plugin,disable the plugin's own benchwarmers server in /mcp so the tools are notlisted twice; the skill and /benchwarmers:ask keep working, though thecommand then asks before each tool call, since its pre-approved tools carrythe plugin's names. The plugin's verdict hook calls the plugin's own server,so it does not work through an aggregator: with that server disabled it onlyshows a hook-error notice at the end of each turn.

An agent running on a collection. The MCP tools ask a panel for asecond opinion; an agent can also be driven by a collection, in routemode (bench/<collection>:route): one member, picked by its record, getseach turn with its tools, stays on for the whole conversation, and isreplaced by the next member if it fails before answering. Claude Codepoints at it with ANTHROPIC_BASE_URL=http://127.0.0.1:8122 andANTHROPIC_MODEL=bench/core:route; Codex, opencode, Cline and otherOpenAI-, Responses- or Ollama-speaking agents with the same base URL.GATEWAY.md, "Route mode", has the set-up and the limits.

The suite

Models are scored on review work: find the faulty line, or say whether aclaim about some code is true. The repository ships a starter pack:

• Real bugs (suite/tasks/rebench-*): recent fixes from public projects,drawn from the SWE-rebench-V2 dataset, each paired with a control.
• Claims (suite/tasks/claim-*): true and false statements about thisproject's own code, with the code attached.

Add your own from your repositories (config/repos.json in the data foldermaps a name to a folder): python suite/seed.py --repo <name> plantsrealistic bugs in your real functions, each with a clean control. Privatecode makes the best tasks, because no model has trained on it. Seesuite/README.md, suite/REBENCH.md andBENCHMARKS.md.

Docker and hosting

One container runs the dashboard, the gateway (on the same port, at /v1)and the scheduler, with the data folder as a volume and no llm CLI (thebuilt-in client makes every call). Put the secrets in a .env file besidecompose.yaml:

BENCHWARMERS_ADMIN_TOKEN=<16 or more characters>
BENCHWARMERS_KEY_OPENROUTER=sk-or-...

then docker compose up -d and open http://127.0.0.1:8122. The compose filepublishes on this machine only; see its comments for a proxy.

Hosted, always on: HOSTING.md compares Fly.io, Railway,Render and a VPS with Caddy (cost, sleep, disks, HTTPS), with a template foreach under deploy/.

Render is one click: a paid instance with a 1 GB disk and a generated admintoken. Fly.io is four commands from a checkout (cp deploy/fly.toml .,fly launch --copy-config --no-deploy, fly secrets set BENCHWARMERS_ADMIN_TOKEN=..., fly deploy). Railway is a few steps in its CLIor browser, in HOSTING.md.

What leaves your machine

Benchwarmers has no account, no telemetry and no server of its own; nothingreaches Wendell Research. What it sends, and where:

• Your prompts, only to the endpoints you add: a provider, an aggregator,a hub such as Open WebUI, or a local runtime. That includes the probes, thesuite and onboarding, which spend calls on your keys. Add only endpoints youare content to send that work to.
• Catalog reads, no key and nothing of yours: models.dev and FreeLLMAPIonce a day, and OpenRouter's public model list.
• Sign-ins, only when you choose one: the provider's own sign-in page.
• Charts: the dashboard's browser page loads Observable Plot from jsDelivrat a pinned version.
• Seeding real-bug tasks, only when you run suite/rebench.py: theSWE-rebench-V2 rows from Hugging Face and the files from GitHub.

Keys stay in the environment or llm's key file (keys.json in its userfolder), never in the data folder. PRIVACY.md and SECURITY.md have thedetail.

Limits

• Tools go through route mode only. A panel answers with text; a requestwith tools is refused there, not trimmed. Route mode passes turns only tomembers on chat-completions endpoints, keeps its sticky conversations inmemory (a restart starts each on a fresh pick), and has been run end to endwith Claude Code; the Codex, opencode and Cline set-ups are written fromtheir docs. Shadow seats (other members answering an agent's turn forevaluation) are planned, not built.
• Some parts are checked against fakes only. The native Anthropic andGemini endpoints, and the LiteLLM and CLI Proxy API hub adapters, are builtfrom those APIs' documentation and source and tested against fake servers,not yet against the real services.
• A merged reply arrives in one piece once the merger has finished; theseats before it stream as they finish.
• Pacing and the ledger cover the built-in client. Calls made through thellm CLI, when you choose it, are neither paced nor counted.
• Calls cost. Probes, onboarding and the suite run on your keys and freetiers. Onboarding runs one model at a time for that reason (Quick spreadsthe suite over the scheduled passes); a suite pass is paced, not burst.
• The scheduler lives in the server. Its jobs run while the dashboard (orthe container, or the service) is up; a missed run is made up once, withinseven days, when it comes back.
• One install, one person at heart. Logins and gateway keys exist for aLAN or a server, but there are no per-user views or roles.
• The onboarding queue lives in the dashboard's memory. Restarting itdrops the rest of a batch; each started model's log stays in runs/.
• Free tiers change without notice. Discover's prices come from publiccatalogs, and some providers' free models work only in their own app(switched per provider on Settings).

Security

Both servers listen on 127.0.0.1 by default and accept only same-origin writesfor their own host name, so a web page in your browser cannot drive them. Tolisten anywhere else, set BENCHWARMERS_ADMIN_TOKEN first; without it theyrefuse to start. With it, the dashboard asks for a login and the gateway takesnamed, revocable gateway keys. Settings, under Server and access, shows howthis stands (whether a secret is set, never its value).

SECURITY.md has the whole model and how to report avulnerability.

Development

python scripts/check.py             # every self-check and test (--quick skips the slow one)

The core is the Python standard library only. CI runs the checks on Linux andWindows for Python 3.11 to 3.13, and on macOS.

Benchwarmers isn't taking outside contributions yet; that opens once it ismore established. A security problem goes through SECURITY.md.

Licence

MIT, Copyright (c) 2026 Wendell Research (LICENSE). The startertasks and bundled fonts carry their own licences, listed in NOTICE.Use is also covered by TERMS.md.