{"repo":"memvid/maw","free":true,"listed":false,"github":"https://github.com/memvid/maw","clone":"git clone https://github.com/memvid/maw.git","description":"Crawl any website into a single searchable file. Query it forever, offline.","language":"TypeScript","stars":38,"topics":["agents","ai-agents","claude","claude-code","crawler","firecrawl","mcp","memory","memvid","mv2"],"license":null,"category":"ai-agents","readme_excerpt":"Crawl any site. Search it forever. Install · Commands · Examples · FAQ --- No more bookmarking docs you'll forget about. No more 47 browser tabs. No more \"I read this somewhere but can't find it.\" Install Or just npx @memvid/maw if you don't want to install anything. The basics That last one needs OPENAI API KEY in your env. The first two work out of the box. What can you crawl? Basically anything. It figures out the right approach automatically: - Single page URL → fetches just that page - Domain root → crawls everything it can find - Local path → reads your files directly - Cloudflare/bot protection → switches to stealth browser Commands Crawl Search Preview before crawling Export Semantic search By default you get keyword search (BM25). It's fast and works well for most things. Want semantic search? Add --embed : Costs about $0.01 per 1000 pages. Your queries will understand meaning, not just keywords. How it handles protected sites 90% of sites work with a simple fetch. The other 10% get a real browser. If that's blocked too, stealth mode usually gets through. You don't have to think about this. It just tries each approach until something works. All the flags Flag What it does Default ------ -------------- --------- -o, --output Output file maw.mv2 -d, --depth How deep to crawl 2 -m, --max-pages Stop after this many pages 150 -c, --concurrency Parallel requests 10 -r, --rate-limit Max requests/second 10 --include Only crawl URLs matching this - --exclude Skip URLs matchin","default_branch":null,"files":null,"tree":[],"storefront":"/r/memvid","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/memvid/maw/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}