{"repo":"sisig-ai/doctor","free":true,"listed":false,"github":"https://github.com/sisig-ai/doctor","clone":"git clone https://github.com/sisig-ai/doctor.git","description":"Doctor is a tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents.","language":"Python","stars":463,"topics":[],"license":"MIT","category":"mcp-servers","readme_excerpt":"🩺 Doctor A tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents for better and more up-to-date reasoning and code generation. --- 🔍 Overview Doctor provides a complete stack for: - Crawling web pages using crawl4ai with hierarchy tracking - Chunking text with LangChain - Creating embeddings with OpenAI via litellm - Storing data in DuckDB with vector search support - Exposing search functionality via a FastAPI web service - Making these capabilities available to LLMs through an MCP server - Navigating crawled sites with hierarchical site maps --- 🏗️ Core Infrastructure 🗄️ DuckDB - Database for storing document data and embeddings with vector search capabilities - Managed by unified Database class 📨 Redis - Message broker for asynchronous task processing 🕸️ Crawl Worker - Processes crawl jobs - Chunks text - Creates embeddings 🌐 Web Server - FastAPI service exposing endpoints - Fetching, searching, and viewing data - Exposing the MCP server --- 💻 Setup ⚙️ Prerequisites - Docker and Docker Compose - Python 3.10+ - uv (Python package manager) - OpenAI API key 📦 Installation 1. Clone this repository 2. Set up environment variables: 3. Run the stack: --- 👁 Usage 1. Go to http://localhost:9111/docs to see the OpenAPI docs 2. Look for the /fetch url endpoint and start a crawl job by providing a URL 3. Use /job progress to see the current job status 4. Configure your editor to use http://localhost:9111/mcp as an MCP server --- ☁️","default_branch":null,"files":null,"tree":[],"storefront":"/r/sisig-ai","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/sisig-ai/doctor/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}