{"repo":"SiluPanda/agent-crawl","free":true,"listed":false,"github":"https://github.com/SiluPanda/agent-crawl","clone":"git clone https://github.com/SiluPanda/agent-crawl.git","description":"High performance, lightweight and typesafe library to crawl and scrape web, built for LLM agents.","language":"TypeScript","stars":11,"topics":["agents","ai-sdk","crawler","llm","markdown","openai-agents-sdk","scraper","typescript"],"license":"MIT","category":"ai-agents","readme_excerpt":"AgentCrawl The High-Performance TypeScript Web Scraper for LLM Agents. AgentCrawl is built to be the \"eyes\" of your AI Agents. It fetches web content, strips away the noise (ads, scripts, styles), and returns clean, token-optimized Markdown ready for your LLM context window. It features a Hybrid Engine that starts with extremely fast static scraping and automatically falls back to a headless browser (Playwright) only when necessary for dynamic content or authentication. Features - 🚀 Hybrid Engine : Instant static fetch by default, auto-switch to Headless Browser for dynamic sites. - ⚡ Token Optimized : Returns clean Markdown, stripping 80-90% of tokens (ads, navs, footers). - 🧠 Agent-First : Detects Main Content, removes boilerplate, and extracts semantic structure. - 🔌 Plug-and-Play : Simple API designed for agent runtimes (scrape + crawl). - 🛡️ Production Ready : Built-in caching, retry logic, user-agent rotation, and resource blocking. - 🕵️ Stealth Mode : Optional best-effort browser hardening to reduce common bot-detection fingerprints. - ✅ Predictable Errors : Non-2xx HTTP responses are surfaced as errors instead of silently parsed as success. - 🇹 Type-Safe : 100% TypeScript with Zod validation. Installation CLI AgentCrawl ships with a CLI for quick scraping and crawling from the terminal. Scrape a page to markdown JSON output with metadata Browser mode for JS-rendered pages Structured extraction Crawl multiple pages More options Run agent-crawl --help for the full","default_branch":null,"files":null,"tree":[],"storefront":"/r/SiluPanda","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/SiluPanda/agent-crawl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}