{"repo":"ShivrajY/MagicBox","free":true,"listed":false,"github":"https://github.com/ShivrajY/MagicBox","clone":"git clone https://github.com/ShivrajY/MagicBox.git","description":"Configuration-driven .NET universal scraper with safe HTTP/browser fetching, CSS/JSONPath extraction, and streaming output.","language":"C#","stars":22,"topics":["cli","csharp","dotnet","html","json","playwright","search-engine","web-scraper","web-scraping","yaml"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"MagicBox Universal Scraper MagicBox is a configuration-driven, asynchronous web-scraping framework for .NET. It turns reviewed YAML adapters into bounded HTML/JSON extraction jobs, with a reusable library, a command-line interface, optional Playwright browser rendering, encrypted local secrets, and streaming JSONL/CSV output. MagicBox is an early public preview. The architecture is implemented and tested, but the project is not yet claiming stable APIs, production guarantees, or universal compatibility with changing third-party websites. Why this project exists Small scrapers often begin as one-off scripts and become difficult to operate: selectors drift, pagination loops indefinitely, browser sessions leak, secrets land in source control, and a failed page can lose the rest of a run. MagicBox puts those concerns in a shared runtime and keeps website-specific behavior in reviewable, schema-validated adapters. The design is deliberately conservative: - adapters contain declarative YAML, not arbitrary F#, JavaScript, shell, or dynamic assemblies; - HTTP requests have cancellation, retries, body limits, redirect validation, pacing, and private-network protection; - browser rendering is optional and isolated from HTTP-only installations; - records stream as they complete, so output is not accumulated in memory; - robots.txt is reported for visibility, while users remain responsible for authorization, terms, rate limits, privacy, and applicable law. Architecture The solution is sp","default_branch":null,"files":null,"tree":[],"storefront":"/r/ShivrajY","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ShivrajY/MagicBox/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}