{"repo":"jwplayer/southpaw","free":true,"listed":false,"github":"https://github.com/jwplayer/southpaw","clone":"git clone https://github.com/jwplayer/southpaw.git","description":":baseball: Streaming left joins in Kafka for change data capture","language":"Java","stars":53,"topics":["kafka","distributed-systems","debezium","southpaw","rocksdb","java","cdc","change-data-capture","apache-kafka"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Southpaw Overview Southpaw is a tool that creates denormalized records from input records based on hierarchical relationships. These relationships are similar to a LEFT OUTER JOIN defined by the following SQL statement: SELECT ... FROM table a LEFT OUTER JOIN table b on a key = b key In this case 'table b' is a child relationship of 'table a.' 'a key' is equivalent to the parent key and 'b key' is equivalent to the join key in a child relation. Ultimately, one 'table' is the root relation. The record key in each topic for all input and denormalized records is treated as the primary key, which is used by the various indices and within the denormalized entities themselves. Why? While robust tools like Flink or Kafka Streams support joins, they are extremely limited. The typical use case is to enrich a stream of records with another stream that is used as a small lookup table. For Southpaw, we wanted to be able to create denormalized records in a streaming fashion as the input topics receive new records or updates are made to existing records. The results should be similar to running large JOIN queries against a standard SQL DB, but the results should be processed in a streaming fashion. How? Southpaw maintains a state of all records it sees, keeping the latest version of each record. In addition to this, it builds two types of indices. The first type is the parent index. This index tells Southpaw which denormalized records it should create whenever it sees a new or updated chil","default_branch":null,"files":null,"tree":[],"storefront":"/r/jwplayer","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jwplayer/southpaw/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}