{"repo":"commoncrawl/news-crawl","free":true,"listed":false,"github":"https://github.com/commoncrawl/news-crawl","clone":"git clone https://github.com/commoncrawl/news-crawl.git","description":"News crawling with StormCrawler - stores content as WARC","language":"Java","stars":375,"topics":["crawler","news","warc","web-crawler","apache-storm","common-crawl","commoncrawl","storm-crawler"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"News Crawler Crawler for news based on StormCrawler. Produces WARC files to be stored as part of the Common Crawl. The data is hosted as AWS Open Data Set – if you want to use the data and not the crawler software please read the announcement of the news dataset. How it works The project is a custom Apache StormCrawler topology. Stock StormCrawler provides the heavy machinery — the HTTP fetcher, robots.txt handling, URL partitioner, WARC writer and OpenSearch status index. This repository adds 10 news-specific bolts and filters on top (parsing Google News sitemaps, detecting RSS/Atom feeds, news-aware URL filtering). Two indexes / stores are involved: - OpenSearch holds the URL status index : every known URL and its state ( DISCOVERED , FETCHED , REDIRECTION , ERROR ). This is the crawler's memory of what to fetch next. - WARC files on local disk ( warc.dir ) hold the actual fetched content — this is the output shipped to Common Crawl. The production crawl is a Storm pipeline defined primarily in conf/crawler.flux (the main topology, launched via Storm Flux). A functionally-equivalent Java implementation of the same pipeline is provided in CrawlTopology . Either way the DAG is the same: Feeds and sitemaps are re-fetched on a schedule to discover new articles; the article URLs they yield are queued in the status index and fetched in turn. Prerequisites JVM 17 or higher Install OpenSearch 2.19.6 Install Apache Storm 2.8.9 Start OpenSearch and Storm Create the OpenSearch indices","default_branch":null,"files":null,"tree":[],"storefront":"/r/commoncrawl","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/commoncrawl/news-crawl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}