{"repo":"pinterest/memq","free":true,"listed":false,"github":"https://github.com/pinterest/memq","clone":"git clone https://github.com/pinterest/memq.git","description":"MemQ is an efficient, scalable cloud native PubSub system","language":"Java","stars":141,"topics":["big-data","distributed-systems","kafka","pubsub"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"MemQ: An efficient, scalable cloud native PubSub system MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and serving architecture similar to Apache Pulsar and Facebook Logdevice; however, it relies on a pluggable replicated storage layer i.e. Object Store / DFS / NFS for storing data. The net result is a PubSub system that: - Handles GB/s traffic - Independently scales, writes, and reads - Doesn’t require expensive rebalancing to handle traffic growth - Is 90% more cost effective than our Kafka footprint Getting Started Checkout our Getting Started Guide Usage MemQ can be a valuable PubSub choice for the following use case families: - Large scale data ingestion - Bulk uploads - Near Real Time Analytics Design Detailed architecture docs can be found here Blog Please checkout the original MemQ blog for details and background. Authors Ambud Sharma Ping Min-Lin See contributors License MemQ distributed under Apache License, Version 2.0","default_branch":null,"files":null,"tree":[],"storefront":"/r/pinterest","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/pinterest/memq/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}