AISIX AI Gateway

AISIX AI Gateway is a Rust-native gateway that puts a single, OpenAI-compatible API infront of every LLM provider — OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI,DeepSeek, and any OpenAI-compatible endpoint. It gives platform teams one place to route,govern, secure, and observe LLM traffic, with first-class SSE streaming and low gatewayoverhead.

It runs as a single static binary — low cold-start, lock-free config reads, and hotconfiguration reloads with no restarts: declare resources in one resources.yaml andreload on SIGHUP, or point the gateway at etcd for a multi-replica cluster. Run theopen-source gateway in your infrastructure, or connect it toAISIX Cloudfor centralized management with team governance, budgets, audit, and a dashboard.

AISIX AI Gateway (this repo) is the open-source product. It runs without a controlplane using declarative configuration or etcd. When connected toAISIX Cloud,the same gateway serves as the data plane. AISIX Cloud adds a commercial control plane,either hosted by API7 (Hybrid Cloud) or hosted by you in your infrastructure(On-Premises). In both options, the gateway runs in your environment and callsproviders directly; live AI traffic does not pass through the control plane or API7.The proxy API is identical throughout.Talk to us about AISIX Cloud →

⚡ Quickstart

One container. No control plane, no database, no configuration store — the gateway readsevery dynamic resource from one declarative resources.yaml.

# config.yaml
resources_file: /etc/aisix/resources.yaml
proxy:
  addr: "0.0.0.0:3000"
admin:
  enabled: false          # a declarative gateway needs no admin listener
observability:
  metrics:
    prometheus:
      enabled: true
      addr: "0.0.0.0:9090"

# resources.yaml
_format_version: "1"

provider_keys:
  - display_name: openai-main
    provider: openai
    api_key: ${OPENAI_API_KEY}        # interpolated from the environment

models:
  - display_name: my-model
    provider: openai
    model_name: gpt-4o-mini
    provider_key: openai-main

api_keys:
  - display_name: local-dev
    key_env: CALLER_API_KEY           # hashed at load; the plaintext is never stored
    allowed_models: ["my-model"]

export OPENAI_API_KEY="YOUR_PROVIDER_KEY"
export CALLER_API_KEY="YOUR_CALLER_KEY"

docker run -d --name aisix \
  --platform linux/amd64 \
  -v "$(pwd)/config.yaml:/etc/aisix/config.yaml:ro" \
  -v "$(pwd)/resources.yaml:/etc/aisix/resources.yaml:ro" \
  -e OPENAI_API_KEY -e CALLER_API_KEY \
  -p 3000:3000 -p 127.0.0.1:9090:9090 \
  ghcr.io/api7/aisix:latest        # proxy → :3000, metrics + status → :9090
#                                  ^ the metrics/status listener is unauthenticated;
#                                    keep it on loopback or a private network

Then call the gateway exactly like OpenAI:

curl http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $CALLER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"my-model","messages":[{"role":"user","content":"hello"}]}'

Edit resources.yaml and send SIGHUP (docker kill -s HUP aisix) to apply changes withno restart — an invalid file is rejected whole and the last good configuration keepsserving. Check a file before booting with aisix validate --resources resources.yaml.

Full walkthrough: theGateway Quickstart ·every field: the resources file reference.For a multi-replica cluster, point the gateway at etcd instead — resources_file andetcd are mutually exclusive.

✨ Why AISIX

• One API, every model. Speak the OpenAI or Anthropic wire format in; the gatewaytranslates to whichever provider each model points at. Point an OpenAI or Claude SDK atone base_url and switch models without changing code.
• A real gateway, in Rust. Single static binary, low cold-start, lock-free config readson the hot path, native streaming.
• Open source, free forever. Apache-2.0 licensed and built to run in yourinfrastructure. Choose AISIX Cloud when you want centralized management through acontrol plane and dashboard.
• Production controls built in. Routing & failover, rate limits, guardrails, caching,and observability ship in the box. (Budgets and spend caps are an AISIX Cloud feature —the gateway enforces the control plane's decisions.)

🧩 Features — available today

Covered by 183 end-to-end scenario files (496 cases) that run against real gateway processes.

• OpenAI-compatible proxy (:3000) — chat/completions, completions, responses,embeddings, rerank, images/generations, audio/{speech,transcriptions,translations},videos (submit → poll → fetch), files, batches, fine_tuning/jobs, realtime,GET /v1/models, plus a root-level /passthrough/:provider/* escape hatch. Native SSE streaming,tool/function calling, JSON mode, vision/multimodal input, and reasoning-content support.
• Anthropic Messages API — POST /v1/messages as a first-class route, working againstany configured upstream: requests and responses (including streaming) are translatedboth ways when a model points at a non-Anthropic provider.
• Routing & failover — virtual/routing models with six strategies: round_robin(smooth weighted round-robin), consistent_hash (session affinity keyed by header /cookie / API key / client IP), failover, plus metric-based least_cost,least_latency, and least_busy. Per-target priority tiers (active/backup pools),retry budgets, cooldowns, tag-conditional targets, and per-attempt timeouts.
• Ensemble models — fan one request out to a panel of models concurrently, then have ajudge model synthesize a single answer, with a minimum-successful-responses threshold.
• Semantic routing — one virtual model that dispatches by the meaning of eachrequest: it embeds the prompt, scores it against per-route example utterances, and routesto the best match (or a default). See thesemantic routing docs.
• Rate limiting & concurrency — RPS/RPM/RPH/RPD + TPM/TPD + concurrency caps,AND-combined across caller keys, models, and policy scopes (api_key / model / team /member / team_member). Counters are per-process by default, or shared across replicaswith the Redis backend.
• Guardrails — content-policy enforcement on input and output, in-process or through aprovider: keyword/regex, built-in PII detection and redaction, Presidio, Lakera, OpenAIModeration, AWS Bedrock Guardrails, Azure AI Content Safety (Prompt Shield + textmoderation), and two Alibaba Cloud services. A block returns 422 content_filter;monitor mode records what would have happened without blocking.
• Caching — exact-match response cache with per-policy TTL and model/key scope matchers;memory and Redis backends; cost-saved telemetry on every hit. Separately, automaticprompt caching can be enabled per direct Anthropic model to inject cache breakpoints, socallers get provider-side prompt discounts without changing their requests.
• MCP gateway — front registered upstream MCP servers at /mcp with gateway-heldcredentials, per-server tool namespaces, and per-caller access. It serves everyStreamable HTTP revision from 2025-03-26 through stateless 2026-07-28 withoutdownstream sessions. Upstreams use initialize by default or server/discover withprotocol_version: "2026-07-28". CI runs the official MCP suite's applicable tools-onlyprotocol scenarios. Also exposes a REST API as MCP tools from its OpenAPI description.
• A2A agent gateway — front A2A (Agent-to-Agent) agents at /a2a/:agent, serving eachagent's card with URLs rewritten to the gateway, over JSON-RPC 2.0.
• Inbound authentication — caller API keys (SHA-256 hashed, model allowlists, expiry,rotation), or OIDC/JWT bearer tokens validated against registered providers (Entra ID,Okta, Google Workspace, or any OIDC issuer) with JWKS caching.
• Observability — Prometheus /metrics, structured per-request access logs, usageevents, OTLP/GenAI span export (Langfuse, Honeycomb, Grafana Cloud, or any OTLP receiver),plus dedicated Datadog and Aliyun SLS log exporters and object-storage (S3/GCS/Azure Blob)telemetry.
• Declarative configuration — one resources.yaml carries all ten resource collections(provider keys, models, caller keys, guardrails, MCP servers, A2A agents, cache policies,observability exporters, rate-limit policies, OIDC providers), validated against the sameJSON Schemas the gateway uses at runtime. aisix validate checks a file offline; SIGHUPreloads it atomically.
• Operational endpoints — /livez and /readyz on the proxy listener; /status/config,/status/ready, /status/models, and Prometheus /metrics on a dedicated metricslistener (:9090). The admin listener (:3001) additionally serves a read-onlyresource surface, OpenAPI 3 with a Scalar UI, and a playground. Resources are manageddeclaratively — through the resources_file (reloaded on SIGHUP) or direct etcdwrites — not through the admin listener; its former write endpoints were removed.

🔌 Supported providers

AISIX dispatches through five native adapter families — distinct wire-protocol bridges,not one generic relabel. Whatever the upstream protocol, the client-facing API staysOpenAI-shaped.

Plus specialized handling for vendor quirks (e.g. DeepSeek reasoning content) and dedicatedrerank / embeddings vendors (Cohere, Jina). Details inadapter protocol families.

☁️ Open source vs AISIX Cloud

Same gateway binary, same proxy API — in every form the gateway runs in your environment.AISIX Cloud adds a commercial control plane, either hosted by API7(Hybrid Cloud) or hosted in your infrastructure (On-Premises).

The AISIX Cloud dashboard — overview metrics, multi-provider models, guardrails, budgets (with hard-stop spend caps), and observability exporters, across all your gateways. ▶ Try the live dashboard demo — aisix-demo.api7.ai

→ Want the AISIX Cloud control plane, governance, budgets, and dashboard?Talk to API7 aboutHybrid Cloud or On-Premises, or book a demo.

🏗️ Architecture

A single Cargo workspace; the aisix-server crate builds one binary named aisix thatwires the crates together.

crates/
├── aisix-core           Config, snapshot, resource model, resources.yaml source, errors
├── aisix-etcd           Config provider + watch supervisor
├── aisix-gateway        Hub & bridge, SSE parser, provider trait
├── aisix-proxy          /v1/*, /mcp, /a2a handlers, routing, middleware
├── aisix-admin          Read-only resource surface + playground + OpenAPI
├── aisix-provider-*     openai · anthropic · azure-openai · bedrock · vertex
├── aisix-mcp            MCP gateway — server registry, tool ACL, transports
├── aisix-a2a            A2A agent gateway — agent cards, JSON-RPC bridge
├── aisix-ratelimit      fixed-window + token accounting + concurrency (local | redis)
├── aisix-cache          memory + redis backends
├── aisix-redis          shared Redis connection for cache + rate limits
├── aisix-guardrails     pre/post content-policy hooks
├── aisix-obs            tracing, metrics, access log, exporters
└── aisix-server         the `aisix` binary — bootstrap + CLI

🗺️ Roadmap

Highlights on the roadmap; tracked live inissues:

• Semantic (embedding-similarity) response caching
• More observability sinks — Langsmith, Helicone, Slack alerts
• Prompt templates managed as gateway resources
• Llama-Guard as a guardrail provider

Shipped since this list was last written: the MCP gateway, the A2A agent gateway,OIDC/JWT inbound auth, Redis-backed distributed rate limiting, and the Lakera, Presidio,PII, and OpenAI Moderation guardrails — see Features above.

🛠️ Development

Prerequisites: the Rust toolchain pinned in rust-toolchain.toml. Docker is only neededfor the tests that exercise etcd, Redis, or provider emulators.

cargo check --workspace
cargo fmt --check
cargo clippy --workspace -- -D warnings
cargo test --workspace

# Coverage (matches the CI gate)
cargo llvm-cov --workspace --lcov --output-path lcov.info

# Run locally against a resources.yaml (no etcd needed). Copy the Quickstart's two files
# and change resources_file to the local path, e.g. resources_file: ./resources.yaml
cargo run -p aisix-server --bin aisix -- --config config.local.yaml

# Check a resources file without starting a listener
cargo run -p aisix-server --bin aisix -- validate --resources resources.yaml

💬 Community

• Discord — discord.gg/dUmRZ7Rvf
• Issues & discussions — github.com/api7/aisix/issues
• Contributing — CONTRIBUTING.md
• Website — api7.ai/ai-gateway

If AISIX is useful to you, a ⭐ helps other engineers find it.

📄 License

Apache 2.0.