{"repo":"vifreefly/kimuraframework","free":true,"listed":false,"github":"https://github.com/vifreefly/kimuraframework","clone":"git clone https://github.com/vifreefly/kimuraframework.git","description":"Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.","language":"Ruby","stars":1103,"topics":["kimurai","scraper","crawler","scrapy","headless-chrome","antidetect-browser","ruby","web-scraping","camoufox"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Kimurai: AI-First Web Scraping Framework for Ruby Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs: How it works: 1. On the first request, extract sends the HTML + your schema to an LLM 2. The LLM generates XPath selectors and caches them in google spider.json 3. All subsequent requests use cached XPath — zero AI calls, pure fast Ruby extraction 4. Supports OpenAI, Anthropic, Gemini, or local LLMs via Nukitori Traditional Mode Prefer writing your own selectors? Kimurai works great as a traditional scraper too — with headless antidetect Chromium, Firefox, or simple HTTP requests: Run: $ ruby github spider.rb results.json Okay, that was easy. How about JavaScript rendered websites with dynamic HTML? Let's scrape a page with infinite scroll: Run: $ ruby infinite scroll spider.rb AI Extraction — Configuration Configure your LLM provider to start using AI extraction. The extract method is powered by Nukitori: Selectors are cached in github spider ai.json after the first AI call — all subsequent requests use pure Ruby extraction. Features AI-powered data extraction : Use Nukitori to extract structured data without writing XPath/CSS selectors — just describe what you want, and AI figures out how to extract it Scrape JavaScript rendered websites out of the box Supported engines: Headless Chrom","default_branch":null,"files":null,"tree":[],"storefront":"/r/vifreefly","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/vifreefly/kimuraframework/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}