{"repo":"m-lab/etl-gardener","free":true,"listed":false,"github":"https://github.com/m-lab/etl-gardener","clone":"git clone https://github.com/m-lab/etl-gardener.git","description":"Gardener provides services for maintaining and reprocessing mlab data.","language":"Go","stars":22,"topics":["etl","pipeline"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"gardener branch report-card coveralls -------- ------------- ----------- main Gardener provides services for maintaining and reprocessing M-Lab data. Overview The v2 data pipeline depends on the gardener for daily and historical processing. Daily processing is daily around 10:30 UTC to allow time for nodes to upload data and the daily transfer jobs to copy data to the public archives. Historical processing is currently continuous. As soon as one pass has completed, the gardener starts again from its start date. For both of these modes, gardener issues Jobs (dates) to parsers that request them. The parsers will enuemerate all files for that date and parse each, and report status updates to the gardener for the Job date until all are complete. Jobs API Parsers request new date jobs from the gardener via the Jobs API. The API supports four operations: /job - return the next job /update - update the status of a job /error - report a job error /heartbeat - update progress for a job and tell gardener that work is still in progress. These resources are available on the -gardener addr . Status Page Gardener maintains a status page on a separate status port, that summarizes recent jobs, current state, and any errors. Jobs transition through the following stages: Parsing - gardener has issued a job and a parser is working on it. postProcessing - the parser completed parsing a job. Loading - gardener loads the parser output from GCS to a temporary BigQuery table. Deduplicating - gardene","default_branch":null,"files":null,"tree":[],"storefront":"/r/m-lab","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/m-lab/etl-gardener/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}