{"repo":"ozontech/file.d","free":true,"listed":false,"github":"https://github.com/ozontech/file.d","clone":"git clone https://github.com/ozontech/file.d.git","description":"A blazing fast tool for building data pipelines: read, process and output events. Our community: https://t.me/file_d_community","language":"Go","stars":500,"topics":["pipeline","processing","go","input","output","actions","events","logs","observability","tracing"],"license":"BSD-3-Clause","category":"analytics","readme_excerpt":"Overview file.d is a blazing fast tool for building data pipelines: read, process, and output events. Primarily developed to read from files, but also supports numerous input/action/output plugins. ⚠ Although we use it in production, it still isn't v1.0.0 . Please, test your pipelines carefully on dev/stage environments. Contributing file.d is an open-source project and contributions are very welcome! Please make sure to read our contributing guide before creating an issue and opening a PR! Motivation Well, we already have several similar tools: vector, filebeat, logstash, fluend-d, fluent-bit, etc. Performance tests state that best ones achieve a throughput of roughly 100MB/sec. Guys, it's 2023 now. HDDs and NICs can handle the throughput of a few GB/sec and CPUs processes dozens of GB/sec . Are you sure 100MB/sec is what we deserve? Are you sure it is fast? Main features Fast: more than 10x faster compared to similar tools Predictable: it uses pooling, so memory consumption is limited Reliable: doesn't lose data due to commitment mechanism Container / cloud / kubernetes native Simply configurable with YAML Prometheus-friendly: transform your events into metrics on any pipeline stage Vault-friendly: store sensitive info and get it for any pipeline parameter Well-tested and used in production to collect logs from Kubernetes cluster with 3000+ total CPU cores Performance On MacBook Pro 2017 with two physical cores file.d can achieve the following throughput: 1.7GB/s in files d","default_branch":null,"files":null,"tree":[],"storefront":"/r/ozontech","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ozontech/file.d/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}