{"repo":"Nextdoor/pg-bifrost","free":true,"listed":false,"github":"https://github.com/Nextdoor/pg-bifrost","clone":"git clone https://github.com/Nextdoor/pg-bifrost.git","description":"PostgreSQL Logical Replication tool into Kinesis, S3 and RabbitMQ","language":"Go","stars":63,"topics":["postgresql","kinesis","rabbitmq","postgres-replication","s3","logical-replication"],"license":"Apache-2.0","category":"databases-storage","readme_excerpt":"pg-bifrost pg-bifrost is a logical decoding tool for PostgresSQL that writes the database's stream of events (creates, inserts, and deletes) to Amazon Kinesis Data Streams, Amazon S3 or RabbitMQ. It is written in a modular manner that allows adding additional sinks such as AWS Firehose, AWS DynamoDB and non-AWS destinations as well. Installation A precompiled binary for Linux AMD 64 can obtained under the github Releases tab. Additionally pg-bifrost can also be found on Docker Hub. Note that the docker image is built and deployed by the circleci build. Example Startup: Input: Output: NOTE time here is not the time this wal message has been replicated, nor is it when the time the message has been transported to the sink. It is the server time as reported in the wal binary. These appear to be uninplemented in Postgres versions 9.5 to 11.2 so they they take the epoch time in the marshalled (JSON) message. As such it does not provide any useful data. This is left in if there is support for this binary field in the future. Usage main replicate Configuration pg-bifrost can be configured using cli arguments, environment variables, and a config file. See --help for a detailed list of options. Memory and Throughput Config Var Description -- -- workers Number of workers that push write data to kinesis. client-buffer-size Number of messages to buffer from postgres batch-flush-timeout Number of milliseconds to wait for new messages before writing a batch batch-queue-depth Number of batch","default_branch":null,"files":null,"tree":[],"storefront":"/r/Nextdoor","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Nextdoor/pg-bifrost/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}