{"repo":"Menziess/slipstream-async","free":true,"listed":false,"github":"https://github.com/Menziess/slipstream-async","clone":"git clone https://github.com/Menziess/slipstream-async.git","description":"Slipstream provides a data-flow model to simplify development of stateful streaming applications.","language":"Python","stars":39,"topics":["kafka","stateful-streaming","data-engineering","data-processing","dataflow","python","slipstream","stream-processing","streaming","streaming-data"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Slipstream Slipstream provides a data-flow model to simplify development of stateful streaming applications. Usage Slipstream components interoperate with basic python building blocks: - Any -thing can be passed around as data - Any Callable may be used as a sink - AsyncIterables act as sources - Parallelize through handle A many-to-many relation is established by passing multiple sources / sinks. Quickstart Install Slipstream along with aiokafka (latest): Spin up a local Kafka broker with docker-compose.yml, using localhost:29091 to connect: Copy-paste this snippet. Features - slipstream.handle : bind streams (iterables) and sinks (callables) to user defined handler functions - slipstream.stream : start streaming - slipstream.Topic : consume from (iterable), and produce to (callable) kafka using aiokafka - slipstream.Cache : store data to disk using rocksdict - slipstream.Conf : set global kafka configuration (can be overridden per topic). Produce is fail-fast by default; set produce retries / produce retry backoff to retry retriable broker errors (sources pause via Conf.signal iterables while retrying) - slipstream.codecs.JsonCodec : serialize and deserialize json messages - slipstream.Checkpoint : recover from stream downtimes ( @handle(..., depends on=leader) )","default_branch":null,"files":null,"tree":[],"storefront":"/r/Menziess","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Menziess/slipstream-async/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}