{"repo":"vakovalskii/searcharvester","free":true,"listed":false,"github":"https://github.com/vakovalskii/searcharvester","clone":"git clone https://github.com/vakovalskii/searcharvester.git","description":"Self-hosted search + markdown harvester for AI agents. SearXNG (100+ engines) + FastAPI + trafilatura. Tavily-compatible /search plus /extract with size presets and pagination. One-command Docker Compose.","language":"Python","stars":254,"topics":["agentic-ai","ai-agents","docker-compose","fastapi","llm","llm-tools","markdown","rag","search-api","searxng"],"license":"AGPL-3.0","category":"ai-agents","readme_excerpt":"Searcharvester 🌾 Self-hosted search + extract + deep research for AI agents 📖 Docs: English · Русский · 中文 Three composable HTTP services in a single docker compose up : - /search — Tavily-compatible search via SearXNG (100+ engines) - /extract — URL → clean markdown via trafilatura, with size presets and pagination - /research — deep research agent: give it a question, get back a cited markdown report No API keys, no quotas, fully self-hosted. Pre-built image on GHCR. 🚀 Quick start --- 🧱 Three services, one API 1️⃣ POST /search — Tavily-compatible search Drop-in replacement for the Tavily API: Request body: Response — Tavily schema (see docs/en/api.md ). 2️⃣ POST /extract — URL → clean markdown Takes a URL, fetches the HTML, runs trafilatura for main-content extraction (strips nav/footer/ads, preserves headings, lists, tables, links), returns ready-to-use markdown. Size presets for different context windows: Size Chars Use case --- --- --- s 5 000 Quick summary, small-context LLMs m 10 000 Default agent reading l 25 000 Deep single-page read f full Paginated by 25 000 — read long docs piece by piece Pagination via cache: Cache keyed by md5(url)[:16] , TTL 30 minutes. Cold fetch: 1-3 s; cached page: <50 ms. Useful as a standalone service, not just for the agent — plug it into any LLM pipeline that needs clean page text. 3️⃣ POST /research — deep research agent {query} → orchestrator spawns an ephemeral Hermes Agent container with three skills: Skill Role --- --- searcharv","default_branch":null,"files":null,"tree":[],"storefront":"/r/vakovalskii","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/vakovalskii/searcharvester/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}