{"repo":"laurentvv/crawl4ai-mcp-llm","free":true,"listed":false,"github":"https://github.com/laurentvv/crawl4ai-mcp-llm","clone":"git clone https://github.com/laurentvv/crawl4ai-mcp-llm.git","description":"Web crawling tool that integrates with AI assistants via the MCP","language":"Python","stars":42,"topics":["crawl4ai","mcp","python3","ai-tools","web-crawler"],"license":"MIT","category":"mcp-servers","readme_excerpt":"Web Crawler MCP A powerful web crawling tool that integrates with AI assistants via the MCP (Model Context Protocol). This project allows AI assistants to crawl websites, extract dynamic content, navigate through links, and save structured Markdown files directly. 📋 Features - Native integration with AI assistants via MCP - Return scraped Markdown content directly to the AI - Extracts and surfaces internal/external links for AI navigation - Website crawling with configurable depth - Detailed crawl result statistics - Error and not found page handling - Advanced Scraping Capabilities : - Magic Mode : Bypass anti-bots (like Cloudflare) and simulate real browser behavior - Targeted Extraction : Fetch only what you need using CSS selectors - Custom JavaScript : Execute code before extraction (clicks, scrolls, form fills) - Persistent Sessions : Keep cookies and state across requests for authenticated sites - SPA Support : Wait for dynamic CSS selectors or set explicit pre-extraction delays 🚀 MCP Configuration The simplest and recommended way to use this tool is via uvx , which automatically fetches and runs the latest published version from PyPI. Prerequisites - uv installed on your system. Setup for AI Assistants (e.g., Claude Desktop, Cline) Add the following to your AI Assistant's MCP configuration file (e.g., cline mcp settings.json or claude desktop config.json ): Note for Windows Users : It is highly recommended to specify --python 3.13 to avoid compilation issues with ce","default_branch":null,"files":null,"tree":[],"storefront":"/r/laurentvv","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/laurentvv/crawl4ai-mcp-llm/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}