{"repo":"testomato/minicrawler","free":true,"listed":false,"github":"https://github.com/testomato/minicrawler","clone":"git clone https://github.com/testomato/minicrawler.git","description":"Multiplexing web client supporting HTTP/2 and WHATWG URL compliant parser written in C","language":"C","stars":23,"topics":["crawler","c","cookie","icu","nghttp2","ssl","agpl","http2","multiplexing","parser"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Minicrawler Minicrawler parses URLs, executes HTTP (HTTP/2) requests while handling cookies, network connection management and SSL/TLS protocols. By default it follows redirect locations and returns a full response, final URL, parsed cookies and more. It is designed to handle many requests in parallel in a single thread . It multiplexes connections, running the read/write communication asynchronously. The whole Minicrawler suite is licensed under the AGPL license. URL Library (libminicrawler-url) WHATWG URL Standard compliant parsing and serializing library written in C. It is fast and has only one external dependency – libicu. The library is licensed under the AGPL license. Usage More in test/url.c. Minicrawler Library (libminicrawler) Usage Minicrawler Binary Usage minicrawler [options] [urloptions] url [[url2options] url2]... Options Output header Minicrawler prepends its own header into the output with the following meaning URL : Requested URL Redirected-To : Final absolute URL Redirect-info : Info about each redirect Status : HTTP Status of final response (negative in case of error) -10 Invalid input -9 , -8 DNS error -7 , -6 Connection error -5 SSL/TLS error -4 , -3 Error during sending a HTTP request -2 Error during receiving a HTTP response -1 Decoding or converting error Content-length : Length of the downloaded content in bytes Timeout : Reason of timeout in case of timeout Error-msg : Error message in case of error (negative Status) Content-type : Correct content t","default_branch":null,"files":null,"tree":[],"storefront":"/r/testomato","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/testomato/minicrawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}