{"repo":"recommend-games/board-game-scraper","free":true,"listed":false,"github":"https://github.com/recommend-games/board-game-scraper","clone":"git clone https://github.com/recommend-games/board-game-scraper.git","description":"Board game data scraper","language":"Python","stars":29,"topics":["board-games","boardgamegeek","bgg","bgg-rating","scrapers","spider","scrapy","board-game","boardgame","boardgames"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"🎲 Board Game Scraper 🕸 Scraping data about board games from the web. View the data live at Recommend.Games! Install via Sources BoardGameGeek ( bgg ) DBpedia ( dbpedia ) Luding.org ( luding ) Spielen.de ( spielen ) Wikidata ( wikidata ) Run scrapers Requires Python 3. Make sure Pipenv is installed and create the virtual environment: Run a spider like so: where $SPIDER is one of the IDs above. Run all the spiders with the run scrapers.sh script. Get a list of the running scrapers' PIDs with the processes.sh script. You can close all the running scrapers via and resume them later. Tests You can run scrapy check to perform contract tests for all spiders, or scrapy check $SPIDER to test one particular spider. If tests fails, there most likely has been some change on the website and the spider needs updating. Board game datasets If you are interested in using any of the datasets produced by this scraper, take a look at the BoardGameGeek guild. A subset of the data can also be found on Kaggle. Links board-game-scraper: This repository Recommend.Games: board game recommender using the scraped data recommend-games-server: Server code for Recommend.Games board-game-recommender: Recommender code for Recommend.Games","default_branch":null,"files":null,"tree":[],"storefront":"/r/recommend-games","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/recommend-games/board-game-scraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}