{"repo":"VIDA-NYU/auctus","free":true,"listed":false,"github":"https://github.com/VIDA-NYU/auctus","clone":"git clone https://github.com/VIDA-NYU/auctus.git","description":"Mirror from: https://gitlab.com/ViDA-NYU/auctus/auctus","language":"Python","stars":44,"topics":["dataset","search","dataset-search","data-profiling","index","crawling","search-engine"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Auctus ====== This project is a web crawler and search engine for datasets, specifically meant for data augmentation tasks in machine learning. It is able to find datasets in different repositories and index them for later retrieval. Documentation is available here It is divided in multiple components: Libraries Geospatial database datamart geo . This contains data about administrative areas extracted from Wikidata and OpenStreetMap. It lives in its own repository and is used here as a submodule. Profiling library datamart profiler . This can be installed by clients, will allow the client library to profile datasets locally instead of sending them to the server. It is also used by the apiserver and profiler services. Materialization library datamart materialize . This is used to materialize dataset from the various sources that Auctus supports. It can be installed by clients, which will allow them to materialize datasets locally instead of using the server as a proxy. Data augmentation library datamart augmentation . This performs the join or union of two datasets and is used by the apiserver service, but could conceivably be used stand-alone. Core server library datamart core . This contains common code for services. Only used for the server components. The filesystem locking code is separate as datamart fslock for performance reasons (has to import fast). Services Discovery services : those are responsible for discovering datasets. Each plugin can talk to a specific reposit","default_branch":null,"files":null,"tree":[],"storefront":"/r/VIDA-NYU","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/VIDA-NYU/auctus/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}