Spark History Server MCP (TypeScript)

Spark History Server MCP (TypeScript)

Give an LLM read access to your Spark History Server so it can do the tediouspart of Spark work: finding why a job failed, and finding where a slow job spendsits time.

It is a TypeScript port ofkubeflow/mcp-apache-spark-history-server,verified response-for-response against the Python original — seePARITY.md. On top of the port it ships two agent skills thatturn the raw tools into an expert workflow for root-cause analysis andperformance tuning.

                    ┌──────────────────┐
  data engineer ──▶ │  LLM client      │   Claude Code / Claude Desktop / any MCP client
                    │  + skills        │   ← skills/ supply the method
                    └────────┬─────────┘
                             │ MCP (stdio or streamable-http)
                    ┌────────▼─────────┐
                    │  this server     │   17 tools, 2 prompts
                    └────────┬─────────┘
                             │ HTTP  GET /api/v1/...
                    ┌────────▼─────────┐
                    │ Spark History    │   your existing one, or the bundled demo
                    │ Server           │
                    └────────┬─────────┘
                             │ reads
                    ┌────────▼─────────┐
                    │ event logs       │   s3://…, hdfs://…, file://…
                    └──────────────────┘

The server only ever issues GET requests to the History Server's REST API. Itcannot modify anything.

Contents

1. Quick start
2. Pointing it at your Spark History Server
3. Connecting your LLM client
4. Installing the skills
5. The tools
6. How it works
7. Deployment
8. Troubleshooting
9. Development

1. Quick start

Option A — Docker (nothing to install but Docker)

Starts a Spark History Server loaded with sample event logs and this MCP:

git clone https://github.com/ukonduru91/spark-history-mcp.git
cd spark-history-mcp
docker compose up --build

The bundled logs include a healthy pipeline and a deliberately failed job, so thetools have something real to show before you point them at your own cluster.

To run only the History Server:

./start_local_spark_history.sh          # macOS / Linux / Git Bash
.\start_local_spark_history.ps1         # Windows PowerShell

Option B — from source

Requires Node.js 20+ (22 recommended).

git clone https://github.com/ukonduru91/spark-history-mcp.git
cd spark-history-mcp
npm install
npm run build
npm start

Verify it works

node scripts/mcp-cli.mjs list-tools
node scripts/mcp-cli.mjs call list_applications '{"limit": 5}'

If applications come back, you are connected.

2. Pointing it at your Spark History Server

This is the one thing you must configure. Three ways, highest precedencefirst — environment variables win over the .env file, which wins over YAML.

a. Environment variables (best for containers and CI)

Nesting uses a double underscore. LOCAL below is just a name you choose forthe server:

export SHS_SERVERS__LOCAL__URL=http://spark-history.internal:18080
export SHS_SERVERS__LOCAL__DEFAULT=true

b. A YAML config file

The server looks for one in this order:

1. the path given to --config, or $SHS_MCP_CONFIG
2. ./config.yaml in the working directory
3. ~/.config/spark-mcp/config.yaml

servers:
  prod:
    url: "https://spark-history.company.com:18080"
    default: true          # used when a tool call omits `server`
    verify_ssl: true
    ssl_ca_cert: "/etc/ssl/custom-ca/ca-bundle.pem"   # private CA
    timeout: 30            # seconds
    auth:
      username: admin
      password: ${SPARK_PASSWORD}   # see the note below
      # token: <bearer token>       # or a bearer token instead

  staging:
    url: "https://spark-history-staging.company.com:18080"

On secrets: values in YAML are literal — ${SPARK_PASSWORD} is notexpanded. Keep credentials in environment variables(SHS_SERVERS__PROD__AUTH__PASSWORD), which override the file. This matchesthe upstream project's behaviour.

c. A .env file

Same variable names as (a), read from .env in the working directory.

Multiple servers

Configure as many as you like. Tools take an optional server argument; when itis omitted the server discovers which configured History Server has thatapplication and uses it (cached for 5 minutes). An engineer can therefore askabout an application id without knowing which cluster ran it.

Every setting

Single-underscore variables (SHS_MCP_PORT) still work but log a deprecationwarning, exactly as upstream.

Reaching a History Server you cannot route to

An SSH tunnel plus use_proxy: true covers the common locked-down-cluster case:

ssh -D 8157 -N user@bastion    # SOCKS5 proxy on :8157

3. Connecting your LLM client

stdio (Claude Code, Claude Desktop, most clients)

{
  "mcpServers": {
    "spark-history": {
      "command": "node",
      "args": ["/absolute/path/to/spark-history-mcp/dist/index.js"],
      "env": {
        "SHS_MCP__TRANSPORT": "stdio",
        "SHS_SERVERS__PROD__URL": "https://spark-history.company.com:18080",
        "SHS_SERVERS__PROD__DEFAULT": "true"
      }
    }
  }
}

Claude Code users can do the same in one line:

claude mcp add spark-history \
  --env SHS_MCP__TRANSPORT=stdio \
  --env SHS_SERVERS__PROD__URL=https://spark-history.company.com:18080 \
  --env SHS_SERVERS__PROD__DEFAULT=true \
  -- node /absolute/path/to/spark-history-mcp/dist/index.js

streamable-http (one shared server for a team)

Run it once, point everyone at it:

SHS_MCP__TRANSPORT=streamable-http SHS_MCP__ADDRESS=0.0.0.0 npm start

Clients connect to http://<host>:18888/mcp. The server is read-only, but it isalso unauthenticated — put it behind your normal internal ingress, and enable DNSrebinding protection if it is reachable from a browser:

mcp:
  transport_security:
    enable_dns_rebinding_protection: true
    allowed_hosts: ["spark-mcp.internal:*"]
    allowed_origins: ["https://spark-mcp.internal"]

4. Installing the skills

The tools give the model access to the data. The skills give it the method — theorder to gather evidence in, the thresholds that separate a finding from noise,and the rule that it must not name a cause it has not seen in the data.

# per project
mkdir -p .claude/skills
cp -r skills/spark-rca skills/spark-optimization .claude/skills/

# or for every project
mkdir -p ~/.claude/skills
cp -r skills/spark-rca skills/spark-optimization ~/.claude/skills/

They trigger on their own from a normal question — nobody has to remember acommand:

"the 2am load failed again, app_1724… — can you look?"

See skills/README.md for what is inside each one and how toextend them with your team's own knowledge.

5. The tools

All 17 live in src/tools/tools.ts; their JSON schemas arein src/schemas/generated.ts. Runnode scripts/mcp-cli.mjs list-tools to see them with their arguments.

Finding things

Going deep

Diagnosing

Comparing two runs

Prompts

investigate_failure(app_id, server?) andcompare_applications(app_a, app_b, server?, context?) — interactive walkthroughsfrom the upstream project, for when the engineer wants to drive instead ofhanding the analysis over.

6. How it works

A tool call becomes one or more GETs against /api/v1/..., and the JSON comesback shaped exactly as the Python original shaped it.

src/
  index.ts                 CLI entry, transport selection (stdio | streamable-http)
  config/config.ts         YAML + .env + SHS_* resolution and precedence
  core/
    app.ts                 MCP request handlers; maps results to content blocks
    validation.ts          pydantic-compatible argument validation and messages
    json.ts                Python-compatible JSON rendering
    pyfloat.ts             int/float fidelity across the JSON round-trip
    pyrepr.ts              Python repr() for validation messages
    errors.ts              error text shaping
  api/
    httpClient.ts          HTTP transport, ApiException taxonomy, auth, TLS, SOCKS
    sparkClient.ts         Spark REST facade: pagination, attempts, status filters
  models/
    generated.ts           model shapes, generated from the upstream OpenAPI models
    deserialize.ts         from_dict / model_dump equivalents
    mcpTypes.ts            curated LLM-facing output models
  tools/tools.ts           the 17 tools
  prompts/prompts.ts       the 2 prompts
  schemas/generated.ts     tool + prompt catalogue (names, descriptions, schemas)

Three details worth knowing if you plan to modify it:

• models/generated.ts and schemas/generated.ts are generated, bytools/gen_models.py and tools/gen_schemas.py, from the upstream Pythonproject. Regenerate rather than hand-edit — that is what keeps the catalogue andthe response shapes identical to the original.
• The low-level Server API is used, not McpServer, because the resultshape has to match FastMCP's: one text block per list element, andstructuredContent only for the tools whose Python signature declared aconcrete return type.
• Application discovery lets tools omit server. ApplicationDiscoveryprobes each configured server for the application id and caches the answer for5 minutes.

7. Deployment

Docker

docker build -t spark-history-mcp .
docker run -p 18888:18888 \
  -e SHS_SERVERS__PROD__URL=https://spark-history.company.com:18080 \
  -e SHS_SERVERS__PROD__DEFAULT=true \
  -e SHS_MCP__ADDRESS=0.0.0.0 \
  spark-history-mcp

Kubernetes

Run it as a normal Deployment with the URL in the env and credentials from aSecret:

env:
  - name: SHS_MCP__TRANSPORT
    value: streamable-http
  - name: SHS_MCP__ADDRESS
    value: "0.0.0.0"
  - name: SHS_SERVERS__PROD__URL
    value: http://spark-history-server.spark.svc.cluster.local:18080
  - name: SHS_SERVERS__PROD__DEFAULT
    value: "true"
  - name: SHS_SERVERS__PROD__AUTH__TOKEN
    valueFrom:
      secretKeyRef: { name: spark-history-auth, key: token }

The process is stateless apart from the 5-minute discovery cache, so it scaleshorizontally without coordination.

8. Troubleshooting

Set SHS_MCP__DEBUG=true for verbose logs.

9. Development

npm install
npm run build        # compile to dist/
npm run dev          # run from source, no build step
npm test             # unit tests
npm run typecheck    # tsc --noEmit

Cross-implementation parity testing lives in parity/ — it runs thesame MCP calls against this server and the Python original and diffs everyresponse. PARITY.md records the results and the exact differencesthat remain.

Not ported from upstream

License

Apache-2.0, as with the upstream project.