{"repo":"streamthoughts/kafka-connect-file-pulse","free":true,"listed":false,"github":"https://github.com/streamthoughts/kafka-connect-file-pulse","clone":"git clone https://github.com/streamthoughts/kafka-connect-file-pulse.git","description":"🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka","language":"Java","stars":350,"topics":["kafka","kafka-connect","grok-filters","etl","file-streaming","kafka-producer","kafka-connector","avro","csv","xml"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Kafka Connect File Pulse Connect FilePulse is a multipurpose, scalable and reliable, Kafka Connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka™. It provides capabilities for reading files from: local-filesystem , Amazon S3 , Azure Storage and Google Cloud Storage . Motivation In organizations, data is frequently exported, shared and integrated from legacy systems through the use of files in a wide variety of formats (e.g. CSV, XML, JSON, Avro, etc.). Dealing with all of these formats can quickly become a real challenge for enterprise that usually end up with a complex and hard to maintain data integration mess. A modern approach consists in building a scalable data streaming platform as a central nervous system to decouple applications from each other. Apache Kafka™ is one of the most widely used technologies to build such a system. The Apache Kafka project packs with Kafka Connect a distributed, fault tolerant and scalable framework for connecting Kafka with external systems. The Connect File Pulse project aims to provide an easy-to-use solution, based on Kafka Connect, for streaming any type of data file with the Apache Kafka™ platform. Some of the features of Connect File Pulse are inspired by the ingestion capabilities of Elasticsearch and Logstash . 🚀 Key Features Overview Connect FilePulse provides a set of built-in features for streaming files from multiple filesystems into Kafka. This includes, among other things: Su","default_branch":null,"files":null,"tree":[],"storefront":"/r/streamthoughts","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/streamthoughts/kafka-connect-file-pulse/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}