{"repo":"uscensusbureau/SABLE","free":true,"listed":false,"github":"https://github.com/uscensusbureau/SABLE","clone":"git clone https://github.com/uscensusbureau/SABLE.git","description":"Scraping Assisted by Learning","language":"Python","stars":36,"topics":["web-scraping","machine-learning","text-classification"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"SABLE Introduction Scraping Assisted by Learning (SABLE) is a collection of tools for web crawling and web scraping. Some elements involve supervised machine learning to classify text. The idea is to discover potential new data sources on the web in Portable Document Format (PDF), apply a text classification model to predict whether the PDF contains useful data, and then scrape data using regular expressions and text analysis. SABLE was initially developed to scrape data from government websites in support of the U.S. Census Bureau's Quarterly Summary of State & Local Tax Revenue (QTAX). For more information about QTAX, including technical documentation and experimental monthly state tax collections, visit https://www.census.gov/programs-surveys/qtax.html. Software SABLE is based on the following open-source software: Linux wget pdftotext) pdftoppm) The Linux command-line utilities wget, pdftotext, and pdftoppm are used to download documents, convert PDFs to TXT format, and convert PDFs to image files, respectively. Apache Nutch (version 1.20) Apache Nutch is a Java-based web crawler and is used to crawl websites, discover PDFs, and compile a training set of documents for model building. Python (version 3.10) scikit-learn NLTK (Natural Language Toolkit) PDFMiner Beautiful Soup pandas tabula Selenium Python and various packages are used to download documents for special cases, scrape data and text from PDFs, and fit and evaluate text classification models. These models are bas","default_branch":null,"files":null,"tree":[],"storefront":"/r/uscensusbureau","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/uscensusbureau/SABLE/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}