{"repo":"datamill-co/singer-runner","free":true,"listed":false,"github":"https://github.com/datamill-co/singer-runner","clone":"git clone https://github.com/datamill-co/singer-runner.git","description":"A CLI and library to run Singer Taps and Targets","language":"Python","stars":36,"topics":["singer","etl","python"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"singer-runner Singer Runner manages tap and target processes, as well as metrics, state, and configuration. Features - Run a tap or target - Pass run options via CLI paramters or JSON/YAML config file. - Use local file system or S3 for piping the singer stream, storing state, and storing metrics. - Metric storage, piping, and state storage can be extended / customized by inheriting from the base classes. Usage Requires python 3, tested with python 3.7 Install Run Concepts - Pipes - Pipes move a stream of Singer messages from tap to target. A pipe could be as simple as a local file, a file in S3, or Kafka. - State Storage - Loads/saves the state JSON. - Metrics Storage - Accepts a stream of Singer Metrics Programmatic Usage Singer runner can be used within any python application. The primary functions are in singer runner.runner including: - run tap runs a tap - run target runs a target Classes in the singer runner.metrics , singer runner.pipes , and singer runner.state can be used as arguemnts, along with catalog/config.","default_branch":null,"files":null,"tree":[],"storefront":"/r/datamill-co","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/datamill-co/singer-runner/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}