Caseware Procurement Knowledge Base

Caseware Procurement Knowledge Base

Local RAG pipeline and MCP server for the Caseware Senior AI Data take-home exercise.

Resumen visual (HTML): docs/take-home-summary.html Repositorio: https://github.com/leonardorhojas/caseware-procurement-kb

Quick start (how to run)

Run these steps in order from the project root:

cd /Users/rafael/Projects/CASEWARE

# 1. Create and activate the virtual environment (once)
python3 -m venv .venv
source .venv/bin/activate

# 2. Install dependencies (once)
pip install -e .

# 3. Configure API key (once)  optional if using local embeddings
cp .env.example .env
# Edit .env and set OPENAI_API_KEY=sk-...

# 4. Build the knowledge base index (run after setup, or when data changes)
python scripts/ingest.py

# 5. Use the MCP server from Cursor (recommended)
#    See "Connect from Cursor" below  reload MCP after config changes.
#    Then ask questions in Cursor chat, e.g.:
#    "Which documents support order 10687?"

# Optional: run MCP server manually in a terminal (stdio  for debugging only)
# python mcp_server/server.py

Expected ingest output: JSON summary with documents_indexed, chunks_indexed, and artifact_dir: ".../artifacts". If that folder exists with kb.sqlite and vectors.npy, the index is ready.

Typical workflow:

The MCP server does not expose a web UI. It communicates over stdio with Cursor (or another MCP client). Running python mcp_server/server.py in a terminal will appear to hang  that is normal; connect it through Cursor instead.

Prerequisites

• Python 3.11+
• OpenAI API key (optional; local embeddings are used automatically if OpenAI quota is unavailable)
• Tesseract OCR for invoice JPGs
  • macOS: brew install tesseract

Setup (detailed)

Same steps as Quick start; use this if you need more context.

cd /Users/rafael/Projects/CASEWARE

python3 -m venv .venv
source .venv/bin/activate

pip install -e .

cp .env.example .env
# Edit .env and set OPENAI_API_KEY=sk-...

On macOS, activate the venv in each new terminal:

source /Users/rafael/Projects/CASEWARE/.venv/bin/activate

Environment variables

If OpenAI returns a quota error during ingest, the pipeline automatically falls back to local embeddings and records the provider in artifacts/manifest.json.

Run the data pipeline

Must run before using MCP. Re-run if you add documents or install Tesseract for JPG OCR.

source .venv/bin/activate
python scripts/ingest.py

Alternative entry point:

caseware-ingest

This will:

1. Scan data/ for PDFs and images
2. Extract text (PyMuPDF for PDFs, Tesseract for JPGs)
3. Chunk text and infer metadata (doc_type, order_id, inventory period)
4. Build embeddings and persist:
   • artifacts/kb.sqlite
   • artifacts/vectors.npy
   • artifacts/manifest.json

Run the MCP server

The MCP server reads the pre-built index from artifacts/  it does not ingest documents at startup.

Option A  Cursor (recommended)

1. Ensure ingest has completed (artifacts/kb.sqlite exists).
2. Config is in .cursor/mcp.json (and optionally ~/.cursor/mcp.json).
3. Restart Cursor or reload MCP in Settings ? MCP.
4. Enable caseware-kb and ask questions in chat.

Option B  Manual stdio (debugging)

source .venv/bin/activate
python mcp_server/server.py

This blocks the terminal waiting for an MCP client on stdin/stdout. Use only to debug; normal usage is through Cursor.

Alternative entry point:

caseware-mcp

Connect from Cursor

Project config: .cursor/mcp.json

Also added to your global Cursor MCP settings (~/.cursor/mcp.json).

Restart Cursor (or reload MCP servers in Settings > MCP), then enable caseware-kb.

{
  "mcpServers": {
    "caseware-kb": {
      "command": "/Users/rafael/Projects/CASEWARE/.venv/bin/python",
      "args": ["/Users/rafael/Projects/CASEWARE/mcp_server/server.py"],
      "cwd": "/Users/rafael/Projects/CASEWARE",
      "env": {
        "CASEWARE_EMBEDDING_PROVIDER": "local"
      }
    }
  }
}

Architecture overview

flowchart TB
    subgraph ingest [Offline ingest]
        Files[data PDFs and JPGs]
        Extract[extract.py]
        Chunk[chunk.py]
        Embed[embed.py]
        Store[store.py SQLite plus numpy]
        Files --> Extract --> Chunk --> Embed --> Store
    end
    subgraph runtime [MCP runtime]
        Load[KnowledgeBase.load]
        Tools[MCP tools]
        Store --> Load --> Tools
    end
    Agent[Cursor or Claude Desktop] --> Tools

Data flow

1. Files in data/ are classified by folder (invoices, purchase_orders, etc.).
2. order_id is parsed from filenames (invoice_10687.pdf) or document text (Order ID: 10687).
3. Text is chunked (~900 chars, 100 overlap) with page references preserved.
4. Embeddings are stored in vectors.npy; metadata and chunk text live in SQLite.
5. MCP tools query the pre-built index (no extraction at request time).

Data modeling

Retrieval strategy

The system uses layered retrieval  not a single search engine. Implementation: src/caseware_kb/retrieve.py.

What we do not use: BM25, SQLite FTS5, or score fusion (RRF). The current keyword fallback is a simple substring match, not ranked lexical retrieval.

Hybrid flow today: try vector search first; if empty, fall back to keyword LIKE. See hybrid_search() in retrieve.py.

Structured extraction vs document retrieval

• Structured: order_id, doc_type, and period are parsed at ingest time and queried via dedicated MCP tools. Best for audit-style questions with clear keys (order 10687, missing POs).
• Document retrieval: chunk text + embeddings for open-ended questions (contract clauses, vendor/item mentions). Returns cited snippets, not pre-extracted fields.

Future improvement: BM25 lexical search

Why it would help: Vector search finds semantically similar text but can miss exact tokens (order numbers, product SKUs, legal clause references like "Clause 7", vendor names). Our current LIKE fallback has no relevance ranking and requires all terms to match.

How BM25 works: BM25 (Best Matching 25) is a probabilistic ranking function for keyword search. For each query term it scores documents higher when:

• the term appears often in the chunk (term frequency), with diminishing returns;
• the term is rare across the whole corpus (inverse document frequency  discriminative terms rank higher);
• shorter chunks are not unfairly boosted (length normalization).

A typical upgrade path for this project:

1. Index chunk text in SQLite FTS5 (or a small BM25 library) at ingest time.
2. Run vector search and BM25 in parallel on the same query.
3. Merge ranks (e.g. reciprocal rank fusion) so exact ID/clause matches and semantic matches both surface.

That would improve queries like "Find evidence for product Spegesild" or "Clause 14 damages" without replacing the structured tools for order-level joins.

MCP tools

Citation strategy

Search results return:

• chunk_id, doc_id, doc_type
• file_path, page, order_id
• score, snippet

Use get_source_excerpt(chunk_id=...) for longer grounded context.

MCP demo evidence

Validated with caseware-kb MCP server connected in Cursor (June 2026).

1. Order document bundle  get_order_documents

Query: "Which documents support order 10687?"

Tool: get_order_documents(order_id="10687")

No purchase order, inventory report, or contract is linked to this order. Customer: HUNGO (Hungry Owl All-Night Grocers), order date 2017-09-30.

2. Cross-document gaps  find_document_gaps

Query: "Which invoices are missing a matching purchase order?"

Tool: find_document_gaps()

Invoices without PO (3):

Context from the knowledge base:

• 8 invoices, 8 purchase orders, 16 shipping orders indexed
• 5 order IDs with full invoice + PO + shipping match: 10248, 10382, 10442, 10603, 10711

3. Contract terms  search_knowledge_base

Query: "Summarize contract terms relevant to supply of goods"

Tool: search_knowledge_base(query="contract terms supply of goods", doc_type="contract")

Source: TotalEnergies Master Contract for Supply of Goods and/or Services (dated 20231231)

Key terms retrieved from indexed contract chunks (primarily Clause 7, with supporting clauses):

Practical takeaway: Customer-favorable terms on delivery, conformity, warranty, and remedies  with firm pricing and clear rights to reject, terminate, or substitute supply.

4. Inventory reports  list_inventory_reports

Query: "What inventory reports are available?"

Tool: list_inventory_reports()

7 stock reports (1 page each), covering July 2016 through January 2018:

Coverage is roughly quarterly with gaps (e.g. no 2017-12 report). None are linked to specific order IDs.

5. Shipment vs invoice match  get_order_documents + document content

Query: "Does shipment 10436 match the related invoice?"

Tools: get_order_documents(order_id="10436") plus line-item comparison from indexed text

Result: Yes  shipment 10436 matches invoice 10436.

Line items (identical on both):

Grand total: 2,210.8 on both documents.

Notes:

• Shipped 2017-02-11 via United Package (shipping order only)
• No purchase order on file for order 10436  consistent invoice/shipment pair, but incomplete procurement trail
• Minor naming differences (ship-to vs invoice contact) do not affect product or financial match

Suggested demo flow (1015 min presentation)

1. Architecture: ingest pipeline ? artifacts ? MCP tools
2. Structured lookup: order 10687 (get_order_documents)
3. Gap analysis: missing POs (find_document_gaps)
4. Semantic search: contract supply terms (search_knowledge_base)
5. Metadata query: inventory periods (list_inventory_reports)
6. Cross-doc reasoning: shipment vs invoice 10436

Manual validation checklist

• Order 10687 -> invoice + shipping (no PO in dataset)
• Order 10436 -> invoice + shipping, no PO
• Contract TotalEnergies -> semantic search returns contract chunks
• 7 inventory reports indexed for periods 2016-07 through 2018-01
• JPG invoices require Tesseract (brew install tesseract, then re-run ingest)

Project layout

CASEWARE/
|-- data/
|-- artifacts/             # generated index (gitignored)
|-- scripts/ingest.py
|-- mcp_server/server.py
`-- src/caseware_kb/

AI-assisted development note

This project was built with Cursor AI assistance. Parts assisted by AI: project scaffold, pipeline modules, MCP server, README, and debugging the tool naming collision in mcp_server/server.py.

Validation: python scripts/ingest.py, MCP client smoke tests, and the live demo queries documented above via Cursor + caseware-kb MCP.