{"repo":"goldsky-io/streamling","free":true,"listed":false,"github":"https://github.com/goldsky-io/streamling","clone":"git clone https://github.com/goldsky-io/streamling.git","description":"Data streaming runtime focused on performance, consistency, and extensibility. Write plugins in Rust or WASM and process data with data guarantees.","language":"Rust","stars":88,"topics":["arrow","data-science","datafusion","etl","etl-pipeline","streaming","rust","fair-source"],"license":null,"category":"data-pipelines","readme_excerpt":"Streamling Streamling is a columnar streaming runtime for real-time and historical data processing, extended through dynamically linked plugins. It powers Goldsky Turbo, where thousands of user-created pipelines run in production for banks, hedge funds, and prediction markets. Install with one command (see Quick start) or read more at streamling.dev. Contents - Why Streamling - Quick start - Common patterns - Development setup - Reference - Topology - Dynamic Tables - Side Outputs - Checkpointing - State Backends - Error Handling - Upsert Semantics - Plugin System - Profiling - Benchmarking - Telemetry and Metrics Why Streamling - Plugin system on a columnar data plane. Custom sources, transforms, and sinks are separate Rust crates dynamically linked at runtime. No forking or recompiling the engine. Every stage exchanges Arrow RecordBatches and processes millions of rows per second on one node. In benchmarks against Flink, Streamling delivered roughly 40x the throughput per GB of memory on a Kafka workload. - Transactional correctness. At-least-once delivery through checkpointing. Checkpoint state lives in any Postgres database. - Declarative pipelines, built-in connectors. Define topologies in YAML. Kafka, Postgres, ClickHouse, files, and webhooks ship in a single binary with fast startup. - Lean in production. Allocator and Arrow-native decode work cut one customer workload writing million-row batches into ClickHouse from 31 GB of memory to 4 GB. A pipeline has three sectio","default_branch":null,"files":null,"tree":[],"storefront":"/r/goldsky-io","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/goldsky-io/streamling/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}