{"repo":"ivbeg/newsworker","free":true,"listed":false,"github":"https://github.com/ivbeg/newsworker","clone":"git clone https://github.com/ivbeg/newsworker.git","description":"Advanced news feeds extractor and finder library. Helps to automatically extract news from websites without RSS/ATOM feeds","language":"Python","stars":87,"topics":["python","news","scraper","rss","rss-feed-parser","news-scraper"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"newsworker Turn any news page into an RSS/Atom feed — even when the site publishes no feed at all. newsworker is a Python 3 library and command-line tool that extracts news feeds from plain HTML pages . It is built for the common case where a site publishes fresh news but offers no RSS/ATOM feed, and where generic \"page change\" monitors are too noisy to be useful. The extracted feed can be emitted as JSON, JSON Feed 1.1, RSS, Atom, CSV, HTML, Markdown, YAML or OPML , so you can plug it straight into a feed reader, a pipeline, or your own storage. --- Table of contents - How it works - Installation - Quick start - Command-line interface - extract — build a feed from a page - serve — local feed server - scan — discover existing feeds - analyze — generate a reusable spec - batch — extract feeds from many pages - Plugins, bridges and async transport - watch — poll a page and deliver only new items - cache — inspect and manage caches - parsedate — inspect date parsing - Settings and caching - Output formats - Library usage - Features - Supported languages - Performance - Limitations - Dependencies - Documentation - Contributing - License - Acknowledgements --- How it works The core idea is simple. Most news pages carry a publication date next to each item — 2017-09-27 , 1 jul 2016 , 18/06/2018 , and hundreds of other variants. newsworker : 1. Finds every date on the page using qddate, a fast pattern-based date parser that recognizes 340+ date formats across many languages. 2. Clus","default_branch":null,"files":null,"tree":[],"storefront":"/r/ivbeg","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ivbeg/newsworker/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}