{"repo":"laserdisc-io/tamer","free":true,"listed":false,"github":"https://github.com/laserdisc-io/tamer","clone":"git clone https://github.com/laserdisc-io/tamer.git","description":"Standalone alternatives to Kafka Connect Connectors","language":"Scala","stars":46,"topics":["zio-streams","kafka","jdbc","scala","fp","functional-programming","zio","doobie","fs2","avro"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Tamer Tamer is a domesticated Kafka source connector. It puts the user of this library in complete control of how data is ingested and what state is preserved (in a compacted Kafka topic). As an example, it allows for a JDBC source to pull a window of data (e.g. 5 minutes), starting from some time in the past, as fast as possible. At every \"pull\", the user can decide what to do next (e.g. should the window be increased/decreased? Should the pull slow down?). Tamer currently supports 4 possible source types: - any SQL data store Doobie can handle (e.g. Postgres, MySQL, etc) - any cloud storage that is AWS S3 compatible - OCI Object Storage - any type of REST API, with AuthN, pagination and HATEOAS support Some sensible defaults are provided out of the box but many customisations are possible. Usage Add one of Tamer's modules as a dependency to your project: See here for a sample application that makes use of Tamer's Db module for ingesting data from a JDBC datasource. End to end testing Database module Basic manual testing is available for the code in the example module tamer.db.DatabaseSimple (and/or tamer.db.DatabaseGeneralized ). This code covers getting data from a synthetic Postgres database. Make sure you have docker installed before proceeding. From the db/local folder launch docker compose up (you can enter docker compose down if you want to start from scratch). After that you should be able to access the Kafka gui from http://localhost:8000. Start the runDatabaseSimpl","default_branch":null,"files":null,"tree":[],"storefront":"/r/laserdisc-io","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/laserdisc-io/tamer/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}