{"repo":"neuml/paperetl","free":true,"listed":false,"github":"https://github.com/neuml/paperetl","clone":"git clone https://github.com/neuml/paperetl.git","description":"📄 ⚙️ ETL processes for medical and scientific papers","language":"Python","stars":698,"topics":["python","etl","scientific-papers","parse","medical"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"ETL processes for medical and scientific papers ------------------------------------------------------------------------------------------------------------------------------------------------------- paperetl is an ETL library for processing medical and scientific papers. paperetl supports the following sources: - Full PDF articles - PubMed XML - ArXiv XML - Text Encoding Initiative (TEI) XML - CSV with article metadta paperetl supports the following datastores for parsed articles. - SQLite - JSON files - YAML files Additional optional datastores are available. - Elasticsearch Installation The easiest way to install is via pip and PyPI Python 3.10+ is supported. Using a Python virtual environment is recommended. paperetl can also be installed directly from GitHub to access the latest, unreleased features. Additional dependencies PDF parsing relies on an existing GROBID instance to be up and running. It is assumed that this is running locally on the ETL server. This is only necessary for PDF files. - GROBID install instructions - GROBID start service Note: In some cases, the GROBID engine pool can be exhausted, resulting in a 503 error. This can be fixed by increasing concurrency and/or poolMaxWait in the GROBID configuration file. Docker A Dockerfile with commands to install paperetl , all dependencies and scripts are available in this repository. This will bring up a paperetl command shell. Standard Docker commands can be used to copy files over or commands can be run direct","default_branch":null,"files":null,"tree":[],"storefront":"/r/neuml","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/neuml/paperetl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}