{"repo":"bcongdon/corral","free":true,"listed":false,"github":"https://github.com/bcongdon/corral","clone":"git clone https://github.com/bcongdon/corral.git","description":"🐎 A serverless MapReduce framework written for AWS Lambda","language":"Go","stars":694,"topics":["aws-lambda","mapreduce-framework","mapreduce","serverless"],"license":"MIT","category":"deployment-docker-iac","readme_excerpt":"🐎 corral Serverless MapReduce Corral is a MapReduce framework designed to be deployed to serverless platforms, like AWS Lambda. It presents a lightweight alternative to Hadoop MapReduce. Much of the design philosophy was inspired by Yelp's mrjob -- corral retains mrjob's ease-of-use while gaining the type safety and speed of Go. Corral's runtime model consists of stateless, transient executors controlled by a central driver. Currently, the best environment for deployment is AWS Lambda, but corral is modular enough that support for other serverless platforms can be added as support for Go in cloud functions improves. Corral is best suited for data-intensive but computationally inexpensive tasks, such as ETL jobs. More details about corral's internals can be found in this blog post. Contents: --- - Examples - Deploying in Lambda - AWS Credentials - Configuration - Configuration Settings - Framework Settings - Lambda Settings - Command Line Flags - Environment Variables - Config Files - Architecture - Input Files / Splits - Mappers - Partition / Shuffle - Reducers / Output - Contributing - Running Tests - License - Previous Work / Attributions Examples Every good MapReduce framework needs a WordCount™ example. Here's how to write a \"word count\" in corral: This can be invoked locally by building/running the above source and adding input files as arguments: By default, job output will be stored relative to the current directory. We can also input/output to S3 by pointing to an S3","default_branch":null,"files":null,"tree":[],"storefront":"/r/bcongdon","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/bcongdon/corral/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}