{"repo":"paradigmxyz/sinker","free":true,"listed":false,"github":"https://github.com/paradigmxyz/sinker","clone":"git clone https://github.com/paradigmxyz/sinker.git","description":"Synchronize Postgres to Elasticsearch","language":"Python","stars":74,"topics":["cdc","elasticsearch","postgres"],"license":"Apache-2.0","category":"databases-storage","readme_excerpt":"Sinker: Synchronize Postgres to Elasticsearch What are you sinking about? What is Sinker? Sinker is middleware that synchronizes relational data from a Postgres database to Elasticsearch. It is simple to operate, requires minimal RAM, and handles arbitrarily complex schemas. Sinker is built by Paradigm, and is licensed under the Apache and MIT licenses. For Example In Postgres, you might have a normalized schema like this: - A Student and a Teacher refer to a Person - A Course is taught by a Teacher - Students have and belong to many Courses through the Enrollment join table In Elasticsearch, you might want to index the Course data in an index called courses like this: Now you can easily query Elasticsearch for courses taught by Prof Georgios, or students with high GPAs named Loren. To do this, you need to do two things reliably: 1. Denormalize the normalized data from the five Postgres tables into a single Elasticsearch document with the Course as the parent and the other four tables nested appropriately inside it. 2. Keep the Elasticsearch document in sync with the Postgres data, so that if Abigail changes her name in the database to Abby, it's reflected in the Course- Enrollments- Student- Person.name field. How it Works Sinker transforms the normalized Postgres data into JSON documents stored in a simple key-value materialized view where the key is the Elasticsearch document ID and the value is the JSON document to be stored in Elasticsearch. Sinker creates triggers on th","default_branch":null,"files":null,"tree":[],"storefront":"/r/paradigmxyz","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/paradigmxyz/sinker/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}