{"repo":"RuedigerVoigt/exoskeleton","free":true,"listed":false,"github":"https://github.com/RuedigerVoigt/exoskeleton","clone":"git clone https://github.com/RuedigerVoigt/exoskeleton.git","description":"A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend","language":"Python","stars":22,"topics":["python","python-3","scraping","crawler","crawling-framework","network","database","mariadb","machine-learning"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Exoskeleton New Version Upcoming Exoskeleton 3 will be released in November 2025. Version 2 will not be updated further. Please see the development branch for status quo / updates. Machine Learning and other applications make it necessary to download thousands or sometimes hundreds of thousands of files. Using a high-speed-connection carries the risk to run an involuntary denial-of-service attack on the servers that provide those files and webpages. Exoskeleton is a Python framework that helps you build a crawler / scraper that avoids too high loads on the connection and instead runs permanently and fault tolerant to ultimately download all files. Its main functionalities are: Managing the download queue and document data within a MariaDB database. Avoid processing the same URL more than once. Working through the queue by either downloading files to disk, storing the page source code into a database table, storing the page text, or making PDF-copies of webpages. Managing already downloaded files: Storing multiple versions of a specific file. Assigning labels to downloads, so they can be found and grouped easily. Sending progress reports to the admin. Documentation How To Use Exoskeleton Installation and Requirements Create a Bot Dealing with result pages Avoiding duplicates The Queue: Downloading files / Saving the page code / Creating PDF Bot Behavior Progress Reports via Email File Versions and Labels Using the Blocklist Example Uses Downloading an Archive : A quite complex","default_branch":null,"files":null,"tree":[],"storefront":"/r/RuedigerVoigt","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/RuedigerVoigt/exoskeleton/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}