{"repo":"kevwan/go-stash","free":true,"listed":false,"github":"https://github.com/kevwan/go-stash","clone":"git clone https://github.com/kevwan/go-stash.git","description":"go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.","language":"Go","stars":1223,"topics":["logstash","elk","kafka","elasticsearch"],"license":"MIT","category":"data-pipelines","readme_excerpt":"English 简体中文 go-stash go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch. go-stash is about 5x throughput more than logstash, and easy to deploy, only one executable file. Quick Start Install Quick Start - With binary - With docker, make sure the path of config file is correct. The config.yaml example is as follows: Details input Conns The number of links to kafka, the number of links is based on the number of cores of the CPU, usually <= the number of cores of the CPU. Consumers The number of open threads per connection, the calculation rule is Conns Consumers, not recommended to exceed the total number of slices, for example, if the topic slice is 30, Conns Consumers <= 30 Processors The number of threads to process data, depending on the number of CPU cores, can be increased appropriately, the recommended configuration: Conns Consumers 2 or Conns Consumers 3, for example: 60 or 90 MinBytes MaxBytes The default size of the data block from kafka is 1M 10M. If the network and IO are better, you can adjust it higher. Offset Optional last and first, the default is last, which means read data from kafka from the beginning Filters - Action: drop - Delete flag: The data that meets this condition will be removed when processing and will not be entered into es - According to the delete condition, specify the value of the key field and Value, the Type field can be con","default_branch":null,"files":null,"tree":[],"storefront":"/r/kevwan","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/kevwan/go-stash/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}