{"repo":"le1nux/crawly","free":true,"listed":false,"github":"https://github.com/le1nux/crawly","clone":"git clone https://github.com/le1nux/crawly.git","description":"... a crawler that retrieves news articles via RSS feeds.","language":"Python","stars":11,"topics":["crawler","rss-feed","rss","newsfeed","news"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"Crawly ... a crawler written in Python that retrieves news articles via RSS feeds. Description Crawly retrieves RSS feeds in a pre-defined interval and checks whether there are any new RSS items present. If so it immediately downloads the news articles referred to in the new RSS items (link tag). Since some articles may be modified / corrected later on, crawly can download articles in a fixed interval for a fixed number of times. Getting started Clone the repository and create the following directories: Install the required packages by running Start crawly : If you are using the default config the crawler will run two warmup runs in the beginning where it collects all RSS items and marks them as 'known'. Afterwards it will download the RSS feeds in pre-set intervals and check whether there are new items in the RSS feeds. If so it will download the corresponding articles every 10mins for one hour (7 times in total) and store them in website downloads .csv. Also see section Article Storage. Configuration crawly comes with a sane default configuration. To set up feeds to crawl, set the path accordingly: If you keep the default configuration, crawly will retrieve all RSS feeds in 3-minute intervals and download new articles every 10 mins for 1 hour (- 7 times in total). IMPORTANT All temporal settings are in seconds. Always keep defensive crawling intervals; preferably in the minutes area. Article Storage This section assumes that you are using the default config. There are three","default_branch":null,"files":null,"tree":[],"storefront":"/r/le1nux","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/le1nux/crawly/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}