{"repo":"palewire/fivethirtyeightindex.com","free":true,"listed":false,"github":"https://github.com/palewire/fivethirtyeightindex.com","clone":"git clone https://github.com/palewire/fivethirtyeightindex.com.git","description":"An index of the pages published at fivethirtyeight.com","language":"Python","stars":12,"topics":["archiving","data-journalism","fivethirtyeight","internet-archive","journalism","media","news","svelte","sveltekit","wayback-machine"],"license":"MIT","category":"data-pipelines","readme_excerpt":"fakethirtyeight The old fivethirtyeight.com was taken offline by its corporate owners. This repo spiders the Wayback Machine (and a few adjacent sources) to build a comprehensive, deduplicated index of every editorial entry FiveThirtyEight ever published, then serves a browse + search UI at fivethirtyeightindex.com. What's in the box layer path purpose --- --- --- Data pipeline src/fakethirtyeight/ , fakethirtyeight CLI Crawl Wayback → classify → curate → enrich → emit static site data Frontend web/ (SvelteKit) Reads web/static/data/articles.{json,csv} and renders the archive Tracked data data/enriched.csv , data/feed- .csv The slow-to-regenerate Wayback enrichment + feed harvest Published data web/static/data/ What the site actually serves How the pipeline works 1. crawl queries the Wayback CDX Server API under fivethirtyeight.com (with matchType=domain , so every subdomain is captured). Sharded by year, resumable from data/state.json . Output: data/shards/cdx- - .csv . 2. sitemaps finds captured sitemap.xml URLs in the index, fetches the latest snapshot of each, parses recursively, and writes a data/shards/sitemap- .csv . 3. feeds walks Wayback's timemap of an Atom/RSS feed URL, parses every captured snapshot, and emits both a sitemap-style shard and a pre-enriched metadata file ( data/feed- .csv ). This is how the NYT-era content was recovered when CDX domain queries were 403'd — see Working around CDX restrictions. 4. merge SURT-dedups every shard into data/index.csv . Th","default_branch":null,"files":null,"tree":[],"storefront":"/r/palewire","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/palewire/fivethirtyeightindex.com/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}