{"repo":"tinyfish-io/bigset-oss","free":true,"listed":false,"github":"https://github.com/tinyfish-io/bigset-oss","clone":"git clone https://github.com/tinyfish-io/bigset-oss.git","description":"Open-source BigSet — self-hostable live datasets populated by TinyFish web agents","language":"TypeScript","stars":1676,"topics":["open-source","tinyfish","bigset"],"license":"AGPL-3.0","category":"self-hosted-apps","readme_excerpt":"Build and maintain any dataset from the live web, that refreshes regularly --- ⚠️ BigSet is experimental. It works, sometimes surprisingly well, but expect rough edges. We're building in the open and shipping fast. Things will break, improve, and change. Issues and feedback are very welcome. --- What Is BigSet? You type a sentence: \"YC companies that are currently hiring engineers, with their funding stage, location, and number of open roles.\" BigSet infers the schema automatically, sends autonomous agents to research it on the live web, verifies what they find against real sources, deduplicates, and hands you a structured dataset. Download as CSV or XLSX. You can even set a refresh cadence (30 min, 6 hours, 12 hours, daily, weekly) and the agents re-run on schedule, pulling fresh data so the dataset never goes stale. Built on TinyFish APIs. Quick set up guide ( 3 mins) ✨ Why BigSet? At the end of the day, every interaction with the web, whether it's you or your AI agent, ultimately comes down to data. Prices, companies, jobs, research, availability, inventory. The web has all of it, scattered across millions of pages. There are great tools out there for parts of this problem. Scraping frameworks that extract content from URLs you point them at. Search APIs that return ranked results. Pre-built actors for specific sites. Lead gen platforms that produce verified lists of people and companies. They work, and they work well for what they do. But the moment you need something tha","default_branch":null,"files":null,"tree":[],"storefront":"/r/tinyfish-io","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/tinyfish-io/bigset-oss/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}