{"repo":"stanikol/olx","free":true,"listed":false,"github":"https://github.com/stanikol/olx","clone":"git clone https://github.com/stanikol/olx.git","description":"Web crawler / web scraper micro-service for downloading advertisements from http://olx.ua","language":"Scala","stars":18,"topics":["scala","akka-http","akka-streams","grabber","scraper","parser","microservice","crawler","olx","phone-number"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Parser / Scraper for classified advertisements and phone numbers from site www.olx.ua This application downloads classified ads and phone numbers from www.olx.ua and saves them to H2 Database. It's capable of scraping huge number of ads and phones within very short time. Документація Українською Installation and running 1. Install Docker with docker-compose. 2. Download the source code. Run git clone https://github.com/stanikol/olx 3. Change dir to 'olx/docker'. cd olx/docker 4. Run docker-compose up --build for the first time, then use just docker-compose up . 5. Wait for a while till sbt downloads all the libraries needed and compiles the sources. This may take a while, when running for the first time. 6. Open http://localhost:8080/olx in your browser. 7. Set the search parameters and press \"Start\". 8. H2 database with downloaded ads will be stored in \"olx/db/olxdb.mv.db\" 9. If you want to export results into csv, just run call CSVWRITE('out.csv', 'select from ADS'); from DB page. CSV file is saved to olx/db/out.csv Micro-service You can also send POST requests to http://localhost:8080/olx/run to start downloads. Parameters are: url - OLX search URL name - name of the search count - max number of advertisements to download parsePhones - When true parse and save phones You can send these params with POST requests as form data. Please note, that your search query term should be url encoded. You can use jq, for example: To stop all downloads, run Technology This app relies on ","default_branch":null,"files":null,"tree":[],"storefront":"/r/stanikol","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/stanikol/olx/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}