{"repo":"chrislusf/gleam","free":true,"listed":false,"github":"https://github.com/chrislusf/gleam","clone":"git clone https://github.com/chrislusf/gleam.git","description":"Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.","language":"Go","stars":3562,"topics":["distributed-computing","map-reduce","golang","distributed-systems"],"license":"Apache-2.0","category":"dev-tools","readme_excerpt":"Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built in Go, and the user defined computation can be written in Go, Unix pipe tools, or any streaming programs. High Performance Pure Go mappers and reducers have high performance and concurrency. Data flows through memory, optionally to disk. Multiple map reduce steps are merged together for better performance. Memory Efficient Gleam does not have the common GC problem that plagued other languages. Each executor runs in a separated OS process. The memory is managed by the OS. One machine can host many more executors. Gleam master and agent servers are memory efficient, consuming about 10 MB memory. Gleam tries to automatically adjust the required memory size based on data size hints, avoiding the try-and-error manual memory tuning effort. Flexible The Gleam flow can run standalone or distributed. Adjustable in memory mode or OnDisk mode. Easy to Customize The Go code is much simpler to read than Scala, Java, C++. One Flow, Multiple ways to execute Gleam code defines the flow, specifying each dataset(vertex) and computation step(edge), and build up a directed acyclic graph(DAG). There are multiple ways to execute the DAG. The default way is to run locally. This works in most cases. Here we mostly talk about the distributed mode. Distributed Mode The distributed mode has several names to explain: Master, Agent, Executor, Driver. Gleam","default_branch":null,"files":null,"tree":[],"storefront":"/r/chrislusf","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/chrislusf/gleam/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}