Gnosis MCP Turn your docs into a searchable knowledge base for AI agents.pip install, ingest, serve. Quick Start · Git History · Web Crawl · Backends · Editors · Tools · Embeddings · Full Reference Ingest docs → Search with highlights → Stats overview → Serve to AI agents --- ### Without a docs server - LLMs hallucinate API signatures that don't exist - Entire files dumped into context — 3,000 to 8,000+ tokens each - Architecture decisions buried across dozens of files ### With Gnosis MCP - `search_docs` returns ranked, highlighted excerpts (~600 tokens) - Real answers grounded in your actual documentation - Works across hundreds of docs instantly --- ## Features - **Zero config** — SQLite by default, `pip install` and go - **Hybrid search** — keyword (BM25) + semantic (local ONNX embeddings, no API key) - **Git history** — ingest commit messages as searchable context (`ingest-git`) - **Web crawl** — ingest documentation from any website via sitemap or link crawl - **Multi-format** — `.md` `.txt` `.ipynb` `.toml` `.csv` `.json` + optional `.rst` `.pdf` - **Auto-linking** — `relates_to` frontmatter creates a navigable document graph - **Watch mode** — auto-re-ingest on file changes - **PostgreSQL ready** — pgvector + tsvector when you need scale ## Quick Start ```bash pip install gnosis-mcp gnosis-mcp ingest ./docs/ # loads docs into SQLite (auto-created) gnosis-mcp serve # starts MCP server ``` That's it. Your AI agent can now search your docs. **Want semantic search?** Add local embeddings — no API key needed: ```bash pip install gnosis-mcp[embeddings] gnosis-mcp ingest ./docs/ --embed # ingest + embed in one step gnosis-mcp serve # hybrid search auto-activated ``` Test it before connecting to an editor: ```bash gnosis-mcp search "getting started" # keyword search gnosis-mcp search "how does auth work" --embed # hybrid semantic+keyword gnosis-mcp stats # see what was indexed ``` Try without installing (uvx) ```bash uvx gnosis-mcp ingest ./docs/ uvx gnosis-mcp serve ``` ## Web Crawl Dry-run discovery → Crawl & ingest → Search crawled docs → SSRF protection Ingest docs from any website — no local files needed: ```bash pip install gnosis-mcp[web] # Crawl via sitemap (best for large doc sites) gnosis-mcp crawl https://docs.stripe.com/ --sitemap # Depth-limited link crawl with URL filter gnosis-mcp crawl https://fastapi.tiangolo.com/ --depth 2 --include "/tutorial/*" # Preview what would be crawled gnosis-mcp crawl https://docs.python.org/ --dry-run # Force re-crawl + embed for semantic search gnosis-mcp crawl https://docs.sveltekit.dev/ --sitemap --force --embed ``` Respects `robots.txt`, caches with ETag/Last-Modified for incremental re-crawl, and rate-limits requests (5 concurrent, 0.2s delay). Crawled pages use the URL as the document path and hostname as the category — searchable like any other doc. ## Git History Turn commit messages into searchable context — your agent learns *why* things were built, not just *what* exists: ```bash gnosis-mcp ingest-git . # current repo, all files gnosis-mcp ingest-git /path/to/repo --since 6m # last 6 months only gnosis-mcp ingest-git . --include "src/*" --max-commits 5 # filtered + limited gnosis-mcp ingest-git . --dry-run # preview without ingesting gnosis-mcp ingest-git . --embed # embed for semantic search ``` Each file's commit history becomes a searchable markdown document stored as `git-history/`. The agent finds it via `search_docs` like any other doc — no new tools needed. Incremental re-ingest skips files with unchanged history. ## Editor Integrations Add the server config to your editor — your AI agent gets `search_docs`, `get_doc`, and `get_related` tools automatically: ```json { "mcpServers": { "docs": { "command": "gnosis-mcp", "args": ["serve"] } } } ``` | Editor | Config file | |--------|------------| | **Claude Code** | `.claude/mcp.json` (or [install as plugin](#claude-code-plugin)) | | **Cursor** | `.cursor/mcp.json` | | **Windsurf** | `~/.codeium/windsurf/mcp_config.json` | | **JetBrains** | Settings > Tools > AI Assistant > MCP Servers | | **Cline** | Cline MCP settings panel | VS Code (GitHub Copilot) — slightly different key Add to `.vscode/mcp.json` (note: `"servers"` not `"mcpServers"`): ```json { "servers": { "docs": { "command": "gnosis-mcp", "args": ["serve"] } } } ``` Also discoverable via the VS Code MCP gallery — search `@mcp gnosis` in the Extensions view. For remote deployment, use Streamable HTTP: ```bash gnosis-mcp serve --transport streamable-http --host 0.0.0.0 --port 8000 ``` ## REST API > v0.10.0+ — Enable native HTTP endpoints alongside MCP on the same port. ```bash gnosis-mcp serve --transport streamable-http --rest ``` Web apps can now query your docs over plain HTTP — no MCP protocol required. | Endpoint | Description | |----------|-------------| | `GET /health` | Server status, version, doc count | | `GET /api/search?q=&limit=&category=` | Search docs (auto-embeds with local provider) | | `GET /api/docs/{path}` | Get document by file path | | `GET /api/docs/{path}/related` | Get related documents | | `GET /api/categories` | List categories with counts | **Environment variables:** | Variable | Description | |----------|-------------| | `GNOSIS_MCP_REST=true` | Enable REST API (same as `--rest`) | | `GNOSIS_MCP_CORS_ORIGINS` | CORS allowed origins: `*` or comma-separated list | | `GNOSIS_MCP_API_KEY` | Optional Bearer token auth | **Examples:** ```bash # Health check curl http://127.0.0.1:8000/health # Search curl "http://127.0.0.1:8000/api/search?q=authentication&limit=5" # With API key curl -H "Authorization: Bearer sk-secret" "http://127.0.0.1:8000/api/search?q=setup" ``` ## Backends | | SQLite (default) | SQLite + embeddings | PostgreSQL | |---|---|---|---| | **Install** | `pip install gnosis-mcp` | `pip install gnosis-mcp[embeddings]` | `pip install gnosis-mcp[postgres]` | | **Config** | Nothing | Nothing | Set `GNOSIS_MCP_DATABASE_URL` | | **Search** | FTS5 keyword (BM25) | Hybrid keyword + semantic (RRF) | tsvector + pgvector hybrid | | **Embeddings** | None | Local ONNX (23MB, no API key) | Any provider + HNSW index | | **Multi-table** | No | No | Yes (`UNION ALL`) | | **Best for** | Quick start, keyword-only | Semantic search without a server | Production, large doc sets | **Auto-detection:** Set `GNOSIS_MCP_DATABASE_URL` to `postgresql://...` and it uses PostgreSQL. Don't set it and it uses SQLite. Override with `GNOSIS_MCP_BACKEND=sqlite|postgres`. PostgreSQL setup ```bash pip install gnosis-mcp[postgres] export GNOSIS_MCP_DATABASE_URL="postgresql://user:pass@localhost:5432/mydb" gnosis-mcp init-db # create tables + indexes gnosis-mcp ingest ./docs/ # load your markdown gnosis-mcp serve ``` For hybrid semantic+keyword search, also enable pgvector: ```sql CREATE EXTENSION IF NOT EXISTS vector; ``` Then backfill embeddings: ```bash gnosis-mcp embed # via OpenAI (default) gnosis-mcp embed --provider ollama # or use local Ollama ``` ## Claude Code Plugin For Claude Code users, install as a plugin to get the MCP server plus slash commands: ```bash claude plugin marketplace add nicholasglazer/gnosis-mcp claude plugin install gnosis ``` This gives you: | Component | What you get | |-----------|-------------| | **MCP server** | `gnosis-mcp serve` — auto-configured | | **`/gnosis:search`** | Search docs with keyword or `--semantic` hybrid mode | | **`/gnosis:status`** | Health check — connectivity, doc stats, troubleshooting | | **`/gnosis:manage`** | CRUD — add, delete, update metadata, bulk embed | The plugin works with both SQLite and PostgreSQL backends. Manual setup (without plugin) Add to `.claude/mcp.json`: ```json { "mcpServers": { "gnosis": { "command": "gnosis-mcp", "args": ["serve"] } } } ``` For PostgreSQL, add `"env": {"GNOSIS_MCP_DATABASE_URL": "postgresql://..."}`. ## Tools & Resources Gnosis MCP exposes 6 tools and 3 resources over [MCP](https://modelcontextprotocol.io/). Your AI agent calls these automatically when it needs information from your docs. | Tool | What it does | Mode | |------|-------------|------| | `search_docs` | Search by keyword or hybrid semantic+keyword | Read | | `get_doc` | Retrieve a full document by path | Read | | `get_related` | Find linked/related documents | Read | | `upsert_doc` | Create or replace a document | Write | | `delete_doc` | Remove a document and its chunks | Write | | `update_metadata` | Change title, category, tags | Write | Read tools are always available. Write tools require `GNOSIS_MCP_WRITABLE=true`. | Resource URI | Returns | |-----|---------| | `gnosis://docs` | All documents — path, title, category, chunk count | | `gnosis://docs/{path}` | Full document content | | `gnosis://categories` | Categories with document counts | ### How search works ```bash # Keyword search — works on both SQLite and PostgreSQL gnosis-mcp search "stripe webhook" # Hybrid search — keyword + semantic (requires [embeddings] or pgvector) gnosis-mcp search "how does billing work" --embed # Filtered — narrow results to a specific category gnosis-mcp search "auth" -c guides ``` When called via MCP, the agent passes a `query` string for keyword search. With embeddings configured, search automatically combines keyword and semantic results using Reciprocal Rank Fusion. Results include a `highlight` field with matched terms in `` tags. ## Embeddings Embeddings enable semantic search — finding docs by meaning, not just keywords. **Local ONNX (recommended)** — zero-config, no API key: ```bash pip install gnosis-mcp[embeddings] gnosis-mcp ingest ./docs/ --embed # ingest + embed in one step gnosis-mcp embed # or embed existing chunks separately ``` Uses [MongoDB/mdbr-leaf-ir](https://huggingface.co/MongoDB/mdbr-leaf-ir) (~23MB quantized, Apache 2.0). Auto-downloads on first run. **Remote providers** — OpenAI, Ollama, or any OpenAI-compatible endpoint: ```bash gnosis-mcp embed --provider openai # requires GNOSIS_MCP_EMBED_API_KEY gnosis-mcp embed --provider ollama # uses local Ollama server ``` **Pre-computed vectors** — pass `embeddings` to `upsert_doc` or `query_embedding` to `search_docs` from your own pipeline. ## Configuration All settings via environment variables. Nothing required for SQLite — it works with zero config. | Variable | Default | Description | |----------|---------|-------------| | `GNOSIS_MCP_DATABASE_URL` | SQLite auto | PostgreSQL URL or SQLite file path | | `GNOSIS_MCP_BACKEND` | `auto` | Force `sqlite` or `postgres` | | `GNOSIS_MCP_WRITABLE` | `false` | Enable write tools | | `GNOSIS_MCP_TRANSPORT` | `stdio` | Transport: `stdio`, `sse`, or `streamable-http` | | `GNOSIS_MCP_EMBEDDING_DIM` | `1536` | Vector dimension for init-db | All configuration variables **Database:** `GNOSIS_MCP_SCHEMA` (public), `GNOSIS_MCP_CHUNKS_TABLE` (documentation_chunks), `GNOSIS_MCP_LINKS_TABLE` (documentation_links), `GNOSIS_MCP_SEARCH_FUNCTION` (custom search on PG). **Search & chunking:** `GNOSIS_MCP_CONTENT_PREVIEW_CHARS` (200), `GNOSIS_MCP_CHUNK_SIZE` (4000), `GNOSIS_MCP_SEARCH_LIMIT_MAX` (20). **Connection pool (PostgreSQL):** `GNOSIS_MCP_POOL_MIN` (1), `GNOSIS_MCP_POOL_MAX` (3). **Webhooks:** `GNOSIS_MCP_WEBHOOK_URL`, `GNOSIS_MCP_WEBHOOK_TIMEOUT` (5s). **Embeddings:** `GNOSIS_MCP_EMBED_PROVIDER` (openai/ollama/custom/local), `GNOSIS_MCP_EMBED_MODEL`, `GNOSIS_MCP_EMBED_DIM` (384), `GNOSIS_MCP_EMBED_API_KEY`, `GNOSIS_MCP_EMBED_URL`, `GNOSIS_MCP_EMBED_BATCH_SIZE` (50). **Column overrides:** `GNOSIS_MCP_COL_FILE_PATH`, `GNOSIS_MCP_COL_TITLE`, `GNOSIS_MCP_COL_CONTENT`, `GNOSIS_MCP_COL_CHUNK_INDEX`, `GNOSIS_MCP_COL_CATEGORY`, `GNOSIS_MCP_COL_AUDIENCE`, `GNOSIS_MCP_COL_TAGS`, `GNOSIS_MCP_COL_EMBEDDING`, `GNOSIS_MCP_COL_TSV`, `GNOSIS_MCP_COL_SOURCE_PATH`, `GNOSIS_MCP_COL_TARGET_PATH`, `GNOSIS_MCP_COL_RELATION_TYPE`. **Logging:** `GNOSIS_MCP_LOG_LEVEL` (INFO). Custom search function (PostgreSQL) Delegate search to your own PostgreSQL function for custom ranking: ```sql CREATE FUNCTION my_schema.my_search( p_query_text text, p_categories text[], p_limit integer ) RETURNS TABLE ( file_path text, title text, content text, category text, combined_score double precision ) ... ``` ```bash GNOSIS_MCP_SEARCH_FUNCTION=my_schema.my_search ``` Multi-table mode (PostgreSQL) Query across multiple doc tables: ```bash GNOSIS_MCP_CHUNKS_TABLE=documentation_chunks,api_docs,tutorial_chunks ``` All tables must share the same schema. Reads use `UNION ALL`. Writes target the first table. CLI reference ``` gnosis-mcp ingest [--dry-run] [--force] [--embed] Load files into database gnosis-mcp ingest-git [--since] [--max-commits] [--include] [--exclude] [--dry-run] [--embed] gnosis-mcp crawl [--sitemap] [--depth N] [--include] [--exclude] [--dry-run] [--force] [--embed] gnosis-mcp serve [--transport stdio|sse|streamable-http] [--ingest PATH] [--watch PATH] gnosis-mcp search [-n LIMIT] [-c CAT] [--embed] Search docs gnosis-mcp stats Document, chunk, and embedding counts gnosis-mcp check Verify DB connection + sqlite-vec gnosis-mcp embed [--provider P] [--model M] [--dry-run] Backfill embeddings gnosis-mcp init-db [--dry-run] Create tables + indexes gnosis-mcp export [-f json|markdown|csv] [-c CAT] Export documents gnosis-mcp diff Preview changes on re-ingest ``` How ingestion works `gnosis-mcp ingest` scans a directory for supported files and loads them into the database: - **Multi-format** — Markdown native; `.txt`, `.ipynb`, `.toml`, `.csv`, `.json` auto-converted. Optional: `.rst` (`[rst]` extra), `.pdf` (`[pdf]` extra) - **Smart chunking** — splits by H2 headings (H3/H4 for oversized sections), never splits inside code blocks or tables - **Frontmatter** — extracts `title`, `category`, `audience`, `tags` from YAML frontmatter - **Auto-linking** — `relates_to` in frontmatter creates bidirectional links for `get_related` - **Auto-categorization** — infers category from parent directory name - **Incremental** — content hashing skips unchanged files (`--force` to override) - **Watch mode** — `gnosis-mcp serve --watch ./docs/` auto-re-ingests on changes Architecture ``` src/gnosis_mcp/ ├── backend.py DocBackend protocol + create_backend() factory ├── pg_backend.py PostgreSQL — asyncpg, tsvector, pgvector ├── sqlite_backend.py SQLite — aiosqlite, FTS5, sqlite-vec hybrid search (RRF) ├── sqlite_schema.py SQLite DDL — tables, FTS5, triggers, vec0 virtual table ├── config.py Config from env vars, backend auto-detection ├── db.py Backend lifecycle + FastMCP lifespan ├── server.py FastMCP server — 6 tools, 3 resources, auto-embed queries ├── ingest.py File scanner + converters — multi-format, smart chunking ├── crawl.py Web crawler — sitemap/BFS, robots.txt, ETag caching ├── parsers/ Non-file ingest sources (git history, future: schemas) │ └── git_history.py Git log → markdown documents per file ├── watch.py File watcher — mtime polling, auto-re-ingest ├── schema.py PostgreSQL DDL — tables, indexes, search functions ├── embed.py Embedding providers — OpenAI, Ollama, custom, local ONNX ├── local_embed.py Local ONNX embedding engine — HuggingFace model download └── cli.py CLI — serve, ingest, crawl, search, embed, stats, check ``` ## Available On [MCP Registry](https://registry.modelcontextprotocol.io) (feeds VS Code MCP gallery and GitHub Copilot) · [PyPI](https://pypi.org/project/gnosis-mcp/) · [mcp.so](https://mcp.so) · [Glama](https://glama.ai) · [cursor.directory](https://cursor.directory) ## AI-Friendly Docs | File | Purpose | |------|---------| | [`llms.txt`](llms.txt) | Quick overview — what it does, tools, config | | [`llms-full.txt`](llms-full.txt) | Complete reference in one file | | [`llms-install.md`](llms-install.md) | Step-by-step installation guide | ## Performance Benchmarked on SQLite (in-memory) with keyword search (FTS5 + BM25): | Corpus | QPS | p50 | p95 | p99 | Hit Rate | |--------|-----|-----|-----|-----|----------| | 100 docs / 300 chunks | ~9,800 | 0.09ms | 0.16ms | 0.18ms | 100% | | 500 docs / 1,500 chunks | ~3,500 | 0.24ms | 0.51ms | 0.82ms | 100% | Install size: ~23MB with `[embeddings]` (ONNX model). Base install is ~5MB. Run the benchmark yourself: ```bash python tests/bench/bench_search.py # 100 docs, 1000 queries python tests/bench/bench_search.py --docs 500 # larger corpus python tests/bench/bench_search.py --json # machine-readable output ``` 550+ tests, 10 eval cases (90% hit rate, 0.85 MRR on sample corpus). All tests run without a database. ## Development ```bash git clone https://github.com/nicholasglazer/gnosis-mcp.git cd gnosis-mcp python -m venv .venv && source .venv/bin/activate pip install -e ".[dev]" pytest # 550+ tests, no database needed ruff check src/ tests/ ``` All tests run without a database. Keep it that way. Good first contributions: new embedding providers, export formats, ingestion for new file types (via optional extras). Open an issue first for larger changes. ## Sponsors If Gnosis MCP saves you time, consider [sponsoring the project](https://github.com/sponsors/nicholasglazer). ## License [MIT](LICENSE)