{"repo":"mdowis/anansi","free":true,"listed":false,"github":"https://github.com/mdowis/anansi","clone":"git clone https://github.com/mdowis/anansi.git","description":"A self-healing scraper for hostile sites: broken selectors repair themselves, browser rendering kicks in when needed, and a coherent identity layer (Chrome TLS fingerprints, matched personas, vendor-aware Cloudflare/Akamai/DataDome handling) works to slip past bot detection. Ships with an MCP server so any LLM can drive a full crawl by conversation.","language":"Python","stars":109,"topics":["adaptive-scraping","ai-agent","anti-bot","crawler","data-extraction","llm-tools","mcp","mcp-server","pydantic","python"],"license":"Apache-2.0","category":"mcp-servers","readme_excerpt":"The spider that learns. A self-healing web scraper for hostile sites — and it's driveable by any LLM. Every scraper starts working. The question is how long before it breaks. Anansi is built on a different assumption: the web is adversarial and unstable, and your scraper should handle that without your involvement. When a site changes its layout, Anansi finds the data anyway and remembers the fix . When a page needs a browser to render, it switches to one silently . When bot detection gets in the way, it presents a coherent identity — matched TLS fingerprint, persona, and headers — that works to slip past detection instead of tripping it. And when you re-crawl, unchanged pages are skipped before a request is even made. The result is a crawler that survives redesigns, handles hostile sites, and gets better the longer it runs. Ships with an MCP server so any LLM can drive a full crawl through conversation. --- Highlights - Selectors that repair themselves — CSS selectors carry confidence scores; when one breaks, four healing strategies compete and the winner is persisted. How it works → - A browser only when you need one — every response is checked for JS shells and silently retried in a stealth Playwright browser, cached per domain. How it works → - A coherent anti-bot identity — matched TLS/HTTP-2 fingerprint, persona, and headers, with vendor-aware handling of Cloudflare, Akamai, and DataDome. Anti-bot & identity → - Re-crawls that skip the unchanged — ETag, Last-Modified, c","default_branch":null,"files":null,"tree":[],"storefront":"/r/mdowis","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mdowis/anansi/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}