{"repo":"manuelzander/ra-scraper","free":true,"listed":false,"github":"https://github.com/manuelzander/ra-scraper","clone":"git clone https://github.com/manuelzander/ra-scraper.git","description":"Scraper for Resident Advisor (https://residentadvisor.net/) events, prices and lineups using scrapy","language":"Python","stars":43,"topics":["python3","scraper","scrapy","python37","spider"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"ra-scraper: Resident Advisor event scraper Scrape Resident Advisor (RA) to find out where and when your favourite (techno) artists are playing. Prerequisites These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. The code is run and tested with Python 3.7.5 on macOS 10.14.6 and Ubuntu 18.04.5. Environment Clone the repo to your local machine. Create a virtual environment for Python 3 with: Activate the virtual environment with: Install the required Python packages with: Artists Within the scraper/artists.txt file specify all artists you want to scrape events for (one per line). Artist names should be in a format such that a valid RA url https://www.residentadvisor.net/dj/ARTIST NAME exists. Running the scraper (spider) This project uses the scrapy Python library (tutorial here). You can find the spider with the name ra artist spider in the file scraper/scraper/spiders/ra artist spider.py . To start the spider and save results locally please run: If you want to recursively scrape events of artists mentioned in the lineup section on the event pages scraped for the initially given artists, set the following value in scraper/scraper/settings.py : This setting can result in a very large number of requests, which can be controlled by CLOSESPIDER ITEMCOUNT or CLOSESPIDER PAGECOUNT . Example output Every run of make build produces three JSON Line files (data models in scraper/scraper/items.py ). The first one c","default_branch":null,"files":null,"tree":[],"storefront":"/r/manuelzander","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/manuelzander/ra-scraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}