CorpusGate

CorpusGate

An LLM-ready private document gateway powered by MarkItDown and MCP.

Convert, index and retrieve private documents for AI tools without sending document content tothird-party services.

CorpusGate is a general-purpose, self-hosted Document MCP Server for individuals andteams that need controlled AI-tool access to documents on their own infrastructure. MarkItDownconverts supported files into reusable Markdown. The gateway then chunks and indexes that Markdown,and MCP returns only the relevant, source-attributed chunks under server-enforced budgets.

Self-hosting keeps source documents, generated Markdown, queries, metadata, and indexes under theoperator's control. Token reduction comes from bounded retrieval and chunk selection—not fromMarkItDown alone. This project is not a chatbot, an LLM answer generator, a contract-analysisproduct, a SaaS platform, or a user-facing document panel.

Features

• PDF, DOCX, PPTX, XLSX, TXT, Markdown, and HTML conversion through Microsoft MarkItDown.
• Persistent Markdown cache, token-aware heading-preserving chunks, and SHA-256 deduplication.
• SQLite FTS5/BM25 lexical search in the lightweight default installation.
• Optional CPU-only multilingual semantic search and RRF hybrid retrieval using local embeddings.
• Bounded MCP responses with source/position metadata, cursors, deduplication, and neighbor limits.
• REST API-key and MCP Bearer authentication, safe UUID storage, path/symlink protections, ratelimits, and structured content-safe logs.
• Hardened Docker Compose deployment for AMD64/ARM64, Oracle Cloud, Tailscale, or Caddy HTTPS.
• Setup helper, operational doctor/scan/reindex/backup commands, versioned SQLite schema, and CI.

How it works

REST upload or read-only inbox scan
        │
        ├─ type, signature, size, path, and free-space validation
        ├─ UUID storage + SHA-256 ── unchanged? ── reuse cached/indexed record
        │
        └─ MarkItDown ──> persistent Markdown ──> token-aware chunks
                                                   │
                              ┌────────────────────┴────────────────────┐
                              │                                         │
                    SQLite FTS5 / BM25                      optional local embeddings
                              │                                  + private Qdrant
                              └────────────────────┬────────────────────┘
                                                   │
                               ranking → dedup → token/char budget → MCP

The code keeps parser, storage, repository, chunking, embedding, vector-store, and retrievalcontracts behind interfaces without turning the single-server product into a distributed system.Documents remain the primary data source; Markdown and vector indexes can be rebuilt.

Supported formats

Encrypted, corrupted, scanned-image-only, or converter-unsupported files fail safely withoutstopping other documents.

Quick start

Requirements: Docker Engine with Compose v2 and OpenSSL. Host Python is not required.

git clone https://github.com/mustafa0zdemir/corpusgate.git
cd corpusgate
./corpusgate init
./corpusgate up
curl --fail http://127.0.0.1:8000/health
./corpusgate doctor

./corpusgate init creates persistent/inbox folders, copies .env.example only when .env does notexist, generates separate random REST/MCP credentials without printing them, checks Docker/Composeand the selected port, and validates Compose. It never overwrites an existing .env.

The equivalent manual flow is to copy .env.example to .env, replace both credentialplaceholders with different openssl rand -hex 32 values, create documents/, and rundocker compose up -d. Never commit .env.

Optional local semantic/hybrid retrieval is also one operation after initialization:

./corpusgate init --semantic
./corpusgate up --semantic

The first semantic start downloads the model into a persistent cache and then starts the gatewayoffline with Qdrant on the internal Docker network. Later starts reuse both model and vectorvolumes. The lexical installation does not install or run either semantic component.

Add documents

The simplest operator workflow uses the read-only host inbox:

cp examples/documents/* documents/
./corpusgate scan
./corpusgate list-documents

The scan skips hidden/system/temporary files, unsupported types, directories, and symlinks. Inputfiles remain in documents/; private UUID copies are stored in the persistent source volume.For a complete lexical→semantic/hybrid→MCP walkthrough, use thesynthetic demo.

For a one-file workflow that AI tools can invoke without putting file bytes into the model context,stream the local file directly to the running REST API (requires curl):

./corpusgate upload /absolute/path/to/document.pdf

The command reads the REST key from CORPUSGATE_CLIENT_API_KEY, CORPUSGATE_API_KEY, or the local.env, never prints it, rejects redirects and insecure remote HTTP, and returns only the API'supload metadata. For a remote private server, pass--url https://YOUR-NODE.YOUR-TAILNET.ts.net.

REST upload is available for applications:

export CORPUSGATE_CLIENT_API_KEY='value-from-your-env'
curl --fail -X POST http://127.0.0.1:8000/api/v1/documents \
  -H "X-API-Key: ${CORPUSGATE_CLIENT_API_KEY}" \
  -F 'file=@examples/documents/private-network-guide.md'

REST also provides paginated metadata, Markdown, chunks, lexical search, and deletion under/api/v1/documents. Interactive OpenAPI documentation is at /docs; protected operations stillrequire X-API-Key.

Connect an MCP client

The remote endpoint is https://YOUR_PRIVATE_OR_PUBLIC_HOST/mcp and every MCP request needs:

Authorization: Bearer YOUR_MCP_TOKEN

Use Tailscale Serve as the recommended private route. Caddy HTTPS is the public alternative; thegateway port remains bound to host loopback in both cases. See the verified field mapping,Inspector command, Tailscale/HTTPS examples, and troubleshooting inthe MCP connection guide. Do not copy an unverified client-specific JSONwrapper or save a token in source control.

MCP tools

Retrieval items consistently include document_id, document_name, chunk_id, heading,position, relevance/rank fields, bounded content, content_length, and retrieval-modemetadata. Empty searches return an empty items list, applied budgets, metrics, and no cursor.Invalid modes, cursors, filters, document IDs, or over-limit values produce controlled tool errors.Upload and delete remain REST-only.

Recommended flow:

AI tool → search_document(query, top_k=3, max_tokens=600)
        → ranked chunks + source positions + actual retrieval mode
        → optional bounded get_document_section

Lexical, semantic, and hybrid search

• lexical is the production default: SQLite FTS5 with heading-weighted BM25 preserves exactidentifiers and phrases without another service.
• semantic embeds queries/chunks locally with the configurable multilingual CPU model and storesvectors in private Qdrant.
• hybrid merges independent lexical and semantic ranks with Reciprocal Rank Fusion; duplicatechunks are returned once and exact lexical matches are not discarded.
• lexical_fallback is reported when semantic/hybrid was requested but the optional local model,vector store, or index is unavailable and fallback is enabled.

The default model is Apache-2.0-licensedsentence-transformers/paraphrase-multilingual-MiniLM-L12-v2,a 384-dimensional multilingual model used through FastEmbed/ONNX on CPU. See model replacement,offline transfer, reindex rules, measurements, and Oracle memory guidance insemantic search.

Token optimization

MarkItDown makes diverse files consistently parseable; it does not by itself guarantee fewertokens. The gateway reduces returned context by caching conversion once, ranking chunks, omittingduplicates, enforcing top_k, max_chars, and estimated max_tokens, limiting neighbors, andpaginating long result sets. It never exposes a default MCP tool that returns a complete documentor raw file.

Token counts are a local deterministic estimate, not a provider-specific billing tokenizer. Therepeatable synthetic measurement and its exact scope are inthe retrieval report; no universal saving percentage isclaimed.

Security and privacy

• No telemetry, document text, query text, cloud embedding API, or required LLM provider.
• Source files use UUID paths; filename traversal, absolute paths, symlink escapes, hidden/tempfiles, MIME/signature mismatches, archive expansion, upload size, and low disk are checked.
• REST uses API keys; remote MCP uses constant-time compared Bearer tokens from environment or aDocker secret. Multiple current/previous tokens allow rotation.
• Structured logs contain allowlisted operation metadata, never document content, credentials,complete queries, or client-visible stack traces.
• The gateway container is non-root, capability-free, no-new-privileges, read-only-root, and hasexplicit writable volumes/tmpfs plus resource/log limits.
• Base Compose publishes only 127.0.0.1:8000; Qdrant is internal-only. Public deployment requiresCaddy TLS and keeps Bearer authentication/rate/response limits.

Stored data comprises private UUID source copies, generated Markdown, SQLite metadata/chunks/FTS,optional local vectors/model cache, backups, and operator configuration. To erase data, deletedocuments through REST first; remove persistent volumes only after an explicit backup and shutdown.See SECURITY.md for private vulnerability reporting.

Oracle Cloud deployment

The recommended Oracle Ubuntu deployment binds the app to loopback and uses Tailscale Serve fortailnet-only HTTPS. A Caddy public profile is documented for cases that require a domain. OracleSecurity Lists/NSGs must never expose TCP 8000 or Qdrant 6333.

VM preparation, AMD64/Ampere ARM64 notes, Docker installation, filesystem ownership, secrets,firewalls, Tailscale/Caddy, logging, update, backup, restore, and troubleshooting are in theOracle deployment guide.

Configuration

All application environment variables, defaults, requirements, ranges, examples, and securityeffects are listed in the configuration contract and .env.example.Startup rejects missing/short credentials, invalid ports/paths, impossible chunk/budgetrelationships, unsupported retrieval modes, and invalid semantic vector-store configurationwithout echoing secret values.

Operational commands:

./corpusgate version
./corpusgate status
./corpusgate doctor
./corpusgate mcp-smoke
./corpusgate upload /absolute/path/to/document.pdf
./corpusgate scan
./corpusgate reindex
./corpusgate reindex --semantic
./corpusgate list-documents --limit 20 --offset 0

Backup and restore

./corpusgate backup
./corpusgate restore /backups/corpusgate-backup-TIMESTAMP.tar.gz --confirm-restore

Restore replaces current persistent data and therefore requires an explicit confirmation flag anda stopped writer in production. Backups include private source storage, Markdown cache, atransactionally copied SQLite database, manifest, and secret-free config example. Keep .env andtoken files in a separate encrypted secret backup. Vector data is rebuildable from chunks.

Update and rollback

Learn the current version, backup, select the reviewed tag/image, run the versioned idempotentmigration, restart, check ready/MCP, and retain the backup until validation completes. A databasecreated by a newer incompatible application is rejected instead of silently modified.

Exact commands and the safe rollback/restore path are inupdate and rollback. Never run docker compose down -v during anormal update.

The repository also contains a manual, approval-gated GHCR workflow. Stable tag, moving minor, andlatest behavior are defined in the container publishing policy;no image has been published by this sprint.

Troubleshooting

• ./corpusgate doctor: validates config, storage permissions, SQLite/schema, disk, optional model/vectorstate, service readiness, and version without dumping secrets.
• 401: use REST X-API-Key or MCP Authorization: Bearer, not the other credential type.
• Host rejection: add the exact Tailscale/domain host to CORPUSGATE_ALLOWED_HOSTS and recreate gateway.
• 507: free disk or review the reserved disk threshold before retrying ingestion.
• lexical_fallback: inspect model cache and Qdrant health; lexical retrieval remains available.
• Conversion failure: confirm supported extension, MIME/signature, UTF-8/Office archive integrity,size, encryption, and whether the PDF contains text.
• Logs: ./corpusgate logs --tail=100; sanitize output before sharing.

See SUPPORT.md and the deployment-specific troubleshooting guide before opening anissue.

Compatibility

Untested platforms are not presented as supported. Review the release checklistbefore publishing artifacts.

Limitations

• Single-node SQLite is not a high-availability or multi-writer database.
• Upload conversion is synchronous in 0.1.0; large documents may need longer client/proxytimeouts.
• No OCR, cloud storage adapter, user accounts, UI, answer generation, reranker, fine-tuning, orSaaS control plane.
• Approximate token budgets may differ from a specific LLM tokenizer.
• Semantic model download needs temporary outbound access unless the cache is transferred offline.

Roadmap

• Background conversion jobs without making Redis mandatory for single-node users.
• Optional PostgreSQL/pgvector and object-storage adapters behind existing interfaces.
• More converter metadata extraction and operator-controlled OCR adapter.
• Signed releases, SBOM/provenance, expanded cross-architecture and upgrade fixtures.

Contributing

Read CONTRIBUTING.md, follow CODE_OF_CONDUCT.md, add tests,and use only synthetic non-sensitive fixtures. Security reports must use the private route inSECURITY.md, never a public issue.

License

CorpusGate is available under the existing MIT License. Third-partylibraries and the optional embedding model retain their own licenses.