{"repo":"MaitreyaM/WEB-SCRAPING-MCP","free":true,"listed":false,"github":"https://github.com/MaitreyaM/WEB-SCRAPING-MCP","clone":"git clone https://github.com/MaitreyaM/WEB-SCRAPING-MCP.git","description":"MCP Server leveraging crawl4ai for web scraping and LLM-based content extraction (Markdown, text snippets, smart extraction). Designed for AI agent integration.","language":"Python","stars":27,"topics":[],"license":null,"category":"mcp-servers","readme_excerpt":"Crawl4AI Web Scraper MCP Server This project provides an MCP (Model Context Protocol) server that uses the crawl4ai library to perform web scraping and intelligent content extraction tasks. It allows AI agents (like Claude, or agents built with LangChain/LangGraph) to interact with web pages, retrieve content, search for specific text, and perform LLM-based extraction based on natural language instructions. This server uses: FastMCP: For creating the MCP server endpoint. crawl4ai: For the core web crawling and extraction logic. dotenv: For managing API keys via a .env file. (Optional) Docker: For containerized deployment, bundling Python and dependencies. Features Exposes MCP tools for web interaction: scrape url : Get the full content of a webpage in Markdown format. extract text by query : Find specific text snippets on a page based on a query. smart extract : Use an LLM (currently Google Gemini) to extract structured information based on instructions. Configurable via environment variables (API keys). Includes Docker configuration ( Dockerfile ) for easy, self-contained deployment. Communicates over Server-Sent Events (SSE) on port 8002 by default. Exposed MCP Tools scrape url Scrape a webpage and return its content in Markdown format. Arguments: url (str, required ): The URL of the webpage to scrape. Returns: (str): The webpage content in Markdown format, or an error message. extract text by query Extract relevant text snippets from a webpage that contain a specific searc","default_branch":null,"files":null,"tree":[],"storefront":"/r/MaitreyaM","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/MaitreyaM/WEB-SCRAPING-MCP/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}