{"repo":"lisad/phaser","free":true,"listed":false,"github":"https://github.com/lisad/phaser","clone":"git clone https://github.com/lisad/phaser.git","description":"The missing layer for complex data batch integration pipelines","language":"Python","stars":26,"topics":["data","data-integration","etl","etl-pipeline"],"license":"MIT","category":"data-pipelines","readme_excerpt":"phaser A library to simplify automated batch-oriented complex data integration pipelines, by organizing steps and column definitions into phases, and offering utilities for transforms, sorting, validating and viewing changes to data. Goals and Scope This library is designed to help developers run a series of steps on batch-oriented , record-oriented , un-indexed data. A batch of record-oriented data is a set of records that are intended to be processed together, in which each record has more or less the same fields and those fields are the same type across records. Often record-oriented data can be expressed in CSV files, where the first line contains the column names. Record-oriented data can be stored or expressed in various formats and objects including: CSV files Excel files Pandas dataframes JSON files in JSONL or panda's JSON \"records\" format In this project, record consistency is somewhat forgiving. The library does not insist that each record must have a value for every column. Some records may not have some fields, i.e. 'sparse' data. Sparse data may sometimes be represented in a format that isn't columnar (a JSON file in 'record' format might easily contain records in which only fields with values are listed). Sparse record-oriented data should be trivial to handle in this library, although by default checkpoint data will be saved in a columnar CSV that shows all the null values. The goals of Phaser are to offer an opinionated framework for complex data pipelines wi","default_branch":null,"files":null,"tree":[],"storefront":"/r/lisad","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/lisad/phaser/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}