{"repo":"openclaw/crawlkit","free":true,"listed":false,"github":"https://github.com/openclaw/crawlkit","clone":"git clone https://github.com/openclaw/crawlkit.git","description":"Shared Go infrastructure for local-first crawler archives.","language":"Go","stars":55,"topics":["crawler","go","infra"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"crawlkit 🧰 — one kit, many crawlers crawlkit is the shared Go library for local-first crawler archives. It gives crawler authors provider-neutral building blocks for config paths, SQLite stores, snapshots, backups, synchronization, search, terminal interfaces, and automation. Provider APIs, authentication, schemas, privacy filters, and user-facing command contracts stay in the downstream crawl apps. Install crawlkit requires Go 1.26.6 or newer. Add the package you need to a Go module. For the quick start below: Install the optional archive controller: The latest module version is available through the Go module proxy. Binary release history and the future signed-archive process are documented in Publishing Crawlkit. Quick start This example opens an in-memory SQLite store with crawlkit's connection defaults, applies a schema, writes a row, and reads it back: Run it from a module that depends on crawlkit : Package map Area Packages What they own --- --- --- Local data config , store , state , cache Runtime paths, SQLite access, sync cursors, and safe cache snapshots Portable archives snapshot , backup , mirror JSONL/Gzip packs, encrypted backups, sidecars, and Git-backed history Search embed , vector Embedding providers, vector encoding, exact search, and result fusion App contracts control , output , progress Machine-readable metadata, output formats, and CI-safe progress logs Remote archives remote Provider-neutral HTTP client and versioned archive protocol User surfaces sc","default_branch":null,"files":null,"tree":[],"storefront":"/r/openclaw","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/openclaw/crawlkit/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}