{"repo":"linkedin/data-integration-library","free":true,"listed":false,"github":"https://github.com/linkedin/data-integration-library","clone":"git clone https://github.com/linkedin/data-integration-library.git","description":"The Data Integration Library project provides a library of generic components based on a multi-stage architecture for data ingress and egress.","language":"Java","stars":34,"topics":["data-integration","data-ingestion","data-ingest","data-egress","gobblin"],"license":"BSD-2-Clause","category":"data-pipelines","readme_excerpt":"Data Integration Library LinkedIn Data Integration Library (DIL) is a collection of generic data integration components that can be mix-and-matched to form powerful ready-to-use connectors, which can then be used by data integration frameworks like Apache Gobblin or event processing frameworks like Apache Kafka to ingress or egress data between cloud services or APIs. Highlights - Generic components: data transmission protocol components and data format components are generically designed without one depending on another, greatly relieved the challenges in handling the variety of cloud APIs and services. - Multistage architecture: data integration is never a one-step process, the library inherently supports multi-staged integration processes so that complex data integration scenarios can be handled with simple generic components. - Bidirectional transmission: ingress and egress are just business logic in DIL, both work the same way and use the same set of configurations, as ingress to one end is egress to the other end. - Extensible compression and encryption: users can easily add pluggable and extensible data compression and encryption algorithms. - Flexible pagination: DIL supports a wide range of pagination methods to break large payloads to small chunks. Common Patterns used in production - Asynchronous bulk ingestion from Rest APIs, like Salesforce.com, to Data Lake (HDFS, S3, ADLS) - Data upload to Rest APIs, like Google API, with tracking of responses - Ingest data fro","default_branch":null,"files":null,"tree":[],"storefront":"/r/linkedin","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/linkedin/data-integration-library/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}