LLM Inference MCP Server

LLM Inference MCP Server

Multi-model LLM routing with cost optimization, structured output, and batch inference.

Features

• Smart Routing: Automatically selects the best model based on task type (quick/reasoning/creative/code/extraction/chat/translation/summary)
• Multi-Provider: Supports OpenAI, DeepSeek, Anthropic Claude, and local vLLM
• Cost Optimization: Compare costs across models, track spending, find cheapest options
• Structured Output: Force JSON schema output from any model (native or prompt-engineered)
• Batch Inference: Process up to 50 prompts in parallel
• Token Counting: Estimate tokens for text/messages before making API calls
• Model Comparison: Run same prompt on multiple models side-by-side

Tools

Configuration

Set environment variables to enable providers:

# OpenAI
export OPENAI_API_KEY="sk-..."
export OPENAI_BASE_URL="https://api.openai.com/v1"  # optional

# DeepSeek
export DEEPSEEK_API_KEY="sk-..."
export DEEPSEEK_BASE_URL="https://api.deepseek.com/v1"  # optional

# Anthropic Claude
export ANTHROPIC_API_KEY="sk-ant-..."
export ANTHROPIC_BASE_URL="https://api.anthropic.com"  # optional

# vLLM (local GPU inference)
export VLLM_BASE_URL="http://localhost:8000/v1"  # default

# Custom OpenAI-compatible providers (up to 5)
export CUSTOM_LLM_1_API_KEY="..."
export CUSTOM_LLM_1_BASE_URL="https://..."

Installation

Using with Claude Desktop / Cursor / Windsurf

Add to your MCP settings:

{
  "mcpServers": {
    "llm-inference": {
      "command": "python",
      "args": ["-m", "llm_inference_mcp.server"],
      "env": {
        "OPENAI_API_KEY": "sk-...",
        "DEEPSEEK_API_KEY": "sk-...",
        "VLLM_BASE_URL": "http://localhost:8000/v1"
      }
    }
  }
}

Using with uvx

uvx llm-inference-mcp

Using with pip

pip install llm-inference-mcp
llm-inference-mcp

Smart Routing

The router automatically selects the best model based on task type:

Cost Optimization Examples

# Find cheapest model for a task
chat_completion(messages=[...], prefer_cheapest=True)

# Compare costs before running
estimate_inference_cost(input_text="Analyze this report...", output_tokens=1000)

# Use local vLLM for zero-cost inference
chat_completion(messages=[...], provider="vllm")

Supported Models

License

MIT