{"repo":"apache/uniffle","free":true,"listed":false,"github":"https://github.com/apache/uniffle","clone":"git clone https://github.com/apache/uniffle.git","description":"Uniffle is a high performance, general purpose Remote Shuffle Service.","language":"Java","stars":453,"topics":["mapreduce","shuffle","spark","remote-shuffle-service","rss","tez"],"license":"Apache-2.0","category":"deployment-docker-iac","readme_excerpt":"Apache Uniffle Uniffle is a high performance, general purpose remote shuffle service for distributed computing engines. It provides the ability to push shuffle data into centralized storage service, changing the shuffle style from \"local file pull-like style\" to \"remote block push-like style\". It brings in several advantages like supporting disaggregated storage deployment, super large shuffle jobs, and high elasticity. Currently it supports [Apache Spark][1], [Apache Hadoop MapReduce][2] and [Apache Tez][3]. [1]: https://spark.apache.org [2]: https://hadoop.apache.org [3]: https://tez.apache.org Architecture Uniffle cluster consists of three components, a coordinator cluster, a shuffle server cluster and an optional remote storage (e.g., HDFS). Coordinator will collect the status of shuffle servers and assign jobs based on some strategy. Shuffle server will receive the shuffle data, merge them and write to storage. Depending on different situations, Uniffle supports Memory & Local, Memory & Remote Storage(e.g., HDFS), Memory & Local & Remote Storage(recommendation for production environment). Shuffle Process with Uniffle Spark driver ask coordinator to get shuffle server for shuffle process Spark task write shuffle data to shuffle server with following step: 1. Send KV data to buffer 2. Flush buffer to queue when buffer is full or buffer manager is full 3. Thread pool get data from queue 4. Request memory from shuffle server first and send the shuffle data 5. Shuffle server ","default_branch":null,"files":null,"tree":[],"storefront":"/r/apache","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/apache/uniffle/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}