{"repo":"spronta/crawlie","free":true,"listed":false,"github":"https://github.com/spronta/crawlie","clone":"git clone https://github.com/spronta/crawlie.git","description":"Fast, free, open-source technical SEO + GEO crawler: built for humans and agents.","language":"Rust","stars":98,"topics":["aeo","crawler","geo","marketing-tools","mcp","seo","search-optimisation","search-optimization"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"crawlie The fast, free, open-source technical SEO + GEO crawler — built for humans and agents. Crawl any site for broken links, redirects, missing metadata, and 40+ SEO & Generative-Engine checks — with plain-English guidance on every fix. Runs locally, ships a CLI and an MCP server, and costs nothing. Setup · CLI · MCP & agents · Changelog · Use cases · Why I built this · Desktop app · Checks · Compare · Architecture Read the docs → crawlie.dev 💡Share feedback & suggestions here .png) --- Setup The easy way — npm (installs the crawlie CLI and the crawlie-mcp server): The macOS app — grab the signed .dmg from Releases. From source — needs Rust (engine/CLI/MCP) and, for the desktop app, pnpm + Node: How it ships: the CLI + MCP come only through npm — the right native binary installs automatically as a platform package (nothing to download or unblock). The desktop app is the only direct download: a signed, notarized .dmg on Releases. --- How to use (CLI) Output formats: pretty (terminal), json (machine-readable, the default), csv (issues), html (shareable file). Common flags: Flag What it does --- --- --max-pages Cap pages fetched (default 500) --max-depth Max click depth from the seed --concurrency Parallel requests (default 16) --include / --exclude Scope the crawl by URL pattern --no-robots / --no-sitemap / --no-external Turn off robots.txt, sitemap seeding, external link checks --severity error\\ warning\\ notice Only output findings at/above a level --save Save to local","default_branch":null,"files":null,"tree":[],"storefront":"/r/spronta","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/spronta/crawlie/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}