{"repo":"oduwsdl/ipwb","free":true,"listed":false,"github":"https://github.com/oduwsdl/ipwb","clone":"git clone https://github.com/oduwsdl/ipwb.git","description":"InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS","language":"Python","stars":655,"topics":["ipfs","warc","wayback","web-archiving","python","service-worker","memento","memento-rfc","docker"],"license":"MIT","category":"deployment-docker-iac","readme_excerpt":"InterPlanetary Wayback (ipwb) Peer-To-Peer Permanence of Web Archives InterPlanetary Wayback (ipwb) facilitates permanence and collaboration in web archives by disseminating the contents of WARC files into the IPFS network. IPFS is a peer-to-peer content-addressable file system that inherently allows deduplication and facilitates opt-in replication. ipwb splits the header and payload of WARC response records before disseminating into IPFS to leverage the deduplication, builds a CDXJ index with references to the IPFS hashes that are returned, and combines the header and payload from IPFS at the time of replay. InterPlanetary Wayback primarily consists of two scripts: - ipwb/indexer.py - archival indexing script that takes the path to a WARC input, extracts the HTTP headers, HTTP payload (response body), and relevant parts of the WARC-response record header from the WARC specified and creates byte string representations. The indexer then pushes the byte strings into IPFS using a locally running IPFS daemon then creates a CDXJ file with this metadata for replay.py. - ipwb/replay.py - rudimentary replay script to resolve requests for archival content contained in IPFS for replay in the browser. A pictorial representation of the ipwb indexing and replay process: An important aspect of archival replay systems is rewriting various resource references for proper memento reconstruction so that they are dereferenced properly from the archive from around the same datetime as of the root","default_branch":null,"files":null,"tree":[],"storefront":"/r/oduwsdl","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/oduwsdl/ipwb/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}