{"repo":"jakopako/goskyr","free":true,"listed":false,"github":"https://github.com/jakopako/goskyr","clone":"git clone https://github.com/jakopako/goskyr.git","description":"A configurable command-line web scraper written in go with auto configuration capability","language":"Go","stars":41,"topics":["golang","go","scraper","webscraping"],"license":"GPL-3.0","category":"scrapers-browser-automation","readme_excerpt":"goskyr 1. Quick Start 1. Installation 1. Shell Auto-completion 1. Semi-Automatic Configuration 1. Manual Configuration & Usage 1. Static fields 1. Dynamic fields 1. Fetcher 1. Filters 1. Interaction 1. Pagination 1. Output 1. Build local ML Model for Auto-Config 1. Related Projects 1. Build & Release 1. Contributing 1. Naming 1. Similar Projects This project's goal is to make it easier to scrape list-like structured data from web pages. This could be a list of books from an online book store, a list of plays in a public theater, a list of newspaper articles, etc. Currently, the biggest use-case that I know of is croncert which is also the main motivation behind this project. Next to manually configuring the scraper there is an option of (semi-)automatically generating a configuration file, see quick start and Semi-Automatic Configuration. Machine learning can be leveraged to predict field names more or less accurately, see section Build ML Model for Improved Auto-Config. Quick Start First, install goskyr and then run the following steps to generate a configuration file for the scraper. The configuration file is written to the default location config.yml . Navigation in the interactive terminal window is done with the arrow keys, the return key and the tab key. Note, that different colors are used to show how 'close' certain fields are to each other in the html tree. This should help when there are multiple list-like structures on a web page and you need to figure out which fi","default_branch":null,"files":null,"tree":[],"storefront":"/r/jakopako","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jakopako/goskyr/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}