{"repo":"archivesunleashed/aut","free":true,"listed":false,"github":"https://github.com/archivesunleashed/aut","clone":"git clone https://github.com/archivesunleashed/aut.git","description":"The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.","language":"Scala","stars":158,"topics":["spark","hadoop","webarchives","analysis","apache-spark","scala","digital-humanities","pyspark","dataframe","big-data-analytics"],"license":"Apache-2.0","category":"analytics","readme_excerpt":"The Archives Unleashed Toolkit The Archives Unleashed Toolkit is an open-source platform for analyzing web archives using Apache Spark, and makes use of Sparkling for parsing W/ARC records. The toolkit provides powerful tools for analytics and data processing. It is part of the Archives Unleashed Project. To learn more about the Toolkit and how to use, please see our comprehensive documentation. If you would like a more in-depth look at the project, please check out the following two articles: + Nick Ruest, Jimmy Lin, Ian Milligan, and Samantha Fritz. The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives. Proceedings of the 2020 IEEE/ACM Joint Conference on Digital Libraries (JCDL 2020), Wuhan, China. + Jimmy Lin, Ian Milligan, Jeremy Wiebe, and Alice Zhou. Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives. ACM Journal on Computing and Cultural Heritage , 10(4), Article 22, 2017. Dependencies - Java 11 - Python 3.7.3+ (PySpark) - Scala 2.12+ - Apache Spark (Hadoop 2.7) 3.0.3+ More information on setting up dependencies can be found here. Building Clone the repo: You can then build The Archives Unleashed Toolkit. Usage The Toolkit can be used to submit a variety of extraction jobs with spark-submit , as well used as a library via spark-submit , pyspark , or in your own application. More information on using the Toolkit can be found here. Citing Archives Unleashed How to cite the Archives Unleashed To","default_branch":null,"files":null,"tree":[],"storefront":"/r/archivesunleashed","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/archivesunleashed/aut/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}