{"repo":"rafatbiin/newspaper-crawler","free":true,"listed":false,"github":"https://github.com/rafatbiin/newspaper-crawler","clone":"git clone https://github.com/rafatbiin/newspaper-crawler.git","description":"Scrapy based crawler which crawls newspaper.","language":"Python","stars":20,"topics":["python","scrapy","crawler","news","newspaper","newspaper3k","mongodb","scraper","news-aggregator"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Newspaper Crawler Scrapy based crawler which crawls newspaper. Required Dependencies: 1. docker 2. docker-compose Run the Crawler from source: - First clone the repository. as it contains newspaper3k as a submodule of this repository,follow: - Open terminal inside of the project folder and run the docker-compose file: - Use any MongoDB Client to check the DB at Configuration - The logs will be saved in a directory called tmp inside the runner package. - The default crawling depth is set to 2. If you need to change that, you'll find the CRAWL DEPTH environment variable inside the docker-compose.yml file. About the Newspaper3k library It's an awesome library which helps parsing data from a news article.At this moment, it doesn't have Bengali language support. So I tweaked the library to add Bengali language support. I've already created a Pull Request for adding Bengali language support to the library. Development You are welcome to contribute in this project. You can: - create new issues pointing out the bugs. - add new newspaper sources. - add new features to extend the project. Supported sources Available Sources Can Crawl -------------------------------------- :--------: https://www.prothomalo.com/ ✔ https://bangla.bdnews24.com/ ✔ https://www.thedailystar.net/ ✔","default_branch":null,"files":null,"tree":[],"storefront":"/r/rafatbiin","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/rafatbiin/newspaper-crawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}