{"repo":"mikwielgus/forum-dl","free":true,"listed":false,"github":"https://github.com/mikwielgus/forum-dl","clone":"git clone https://github.com/mikwielgus/forum-dl.git","description":"Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC","language":"Python","stars":125,"topics":["python","scraper","forum","discourse","phpbb","simplemachines","data-fetching","internet-archiving","warc"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"forum-dl Forum-dl is a scraper and archiver for forums (including Discourse, PhpBB, SMF), mailing lists, and news aggregators (list). It can be used to extract and archive all posts from individual threads and entire boards into JSONL, Mbox, Maildir, WARC (complete list). The project is currently in alpha stage. Please do not hesitate to file bug reports and feature requests. Installation You can install stable Forum-dl from PIP or directly from the repository. Minimum Python version is 3.10.11. PIP Install the latest stable version from PIP: Repository Clone the repository and install the development branch in editable mode: Usage examples Download a Simple Machines Forum thread in JSONL format: Save all images from the same thread in directory files : Download a PhpBB subboard into JSONL format, write to stdout ( -o - ) and record a WARC file in phpbb.warc : (due to current architectural limitations, forum-dl will scan the first page of each board in the entire forum before downloading the target board. This will be fixed in future releases) Download Hacker News top stories and write them to a Maildir directory hn : - --textify converts HTML to plaintext (useful for text-only mail clients), - --content-as-title puts the beginning of each message's content in its title (useful for mail clients that don't display content in index view), - -f maildir changes the output format to maildir , - -o hn changes the output directory name to hn . What is supported Forum software - Disc","default_branch":null,"files":null,"tree":[],"storefront":"/r/mikwielgus","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mikwielgus/forum-dl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}