{"repo":"m-lab/etl","free":true,"listed":false,"github":"https://github.com/m-lab/etl","clone":"git clone https://github.com/m-lab/etl.git","description":"M-Lab ingestion pipeline","language":"Go","stars":29,"topics":["etl","pipeline"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"etl branch travis-ci report-card coveralls -------- ----------- ----------- ------------- master integration ETL (extract, transform, load) is a core component of the M-Lab data processing pipeline. The ETL worker is responsible for parsing data archives produced by pusher and publishing M-Lab measurements to BigQuery. Local Development From the command line (or with a browser) make a request to the /v2/worker resource with a filename= parameter that names a valid M-Lab GCS archive. Generating Schema Docs To build a new docker image with the generate schema docs command, run: GKE The universal parser will run in GKE, using a parser node pool, defined in terraform-support. The parser images are built in Cloud Build environment, pushed to gcr.io, and deployed to the data-pipeline cluster. The build trigger can be found with: Migrating to Sink interface The parsers currently use etl.Inserter as the backend for writing records. This API is overly shaped by bigquery, and complicates testing and extension. The row.Sink interface, and row.Buffer define cleaner APIs for the back end and for buffering and annotating. This will streamline migration to Gardener driven table selection, column partitioned tables, and possibly future migration to BigQuery loads instead of streaming inserts. Factories The TaskFactory aggregates a number of other factories for the elements required for a Task. Factory injection is used to generalize ProcessGKETask, and simplify testing. SinkFactory produces ","default_branch":null,"files":null,"tree":[],"storefront":"/r/m-lab","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/m-lab/etl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}