{"repo":"harvard-growth-lab/pandas-to-postgres","free":true,"listed":false,"github":"https://github.com/harvard-growth-lab/pandas-to-postgres","clone":"git clone https://github.com/harvard-growth-lab/pandas-to-postgres.git","description":"Copy Pandas DataFrames and HDF5 files to PostgreSQL database","language":"Python","stars":56,"topics":["pandas","postgresql","hdf5","hdf","database","python3","sqlalchemy","sqlalchemy-database","scientific-computing","postgres"],"license":"BSD-3-Clause","category":"databases-storage","readme_excerpt":"Pandas-to-postgres by the Growth Lab at the Harvard Kennedy School This package is part of Harvard Growth Lab’s portfolio of software packages, digital products and interactive data visualizations. To browse our entire portfolio, please visit growthlab.app. To learn more about our research, please visit Harvard Growth Lab’s home page. About Pandas-to-Postgres allows you to bulk load the contents of large dataframes into postgres as quickly as possible. The main differences from pandas' to sql function are: - Uses COPY combined with to csv instead of execute / executemany , which runs much faster for large volumes of data - Uses COPY FROM STDIN with StringIO to avoid IO overhead to intermediate files. This matters in particular for data stored in unusual formats like HDF, STATA, parquet - common in the scientific world. - Chunked loading methods to be able to load larger-than-memory tables. In particular the HDF5 functions load data in chunks directly from the file, easily extendible to other formats that support random access by row range. - Removes indexing overhead by automatically detecting and dropping indexes before load, and then re-creating them afterwards - Allows you to load multiple separate HDF tables in parallel using multiprocessing.Pool - Works around pandas null value representation issues: float pandas columns that have an integer SQL type get converted into an object column with int values where applicable and NaN elsewhere. - Provides hooks to modify data as","default_branch":null,"files":null,"tree":[],"storefront":"/r/harvard-growth-lab","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/harvard-growth-lab/pandas-to-postgres/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}