{"repo":"aws-samples/aws-cdk-managed-elkk","free":true,"listed":false,"github":"https://github.com/aws-samples/aws-cdk-managed-elkk","clone":"git clone https://github.com/aws-samples/aws-cdk-managed-elkk.git","description":"Managed ELKK stack implemented with the AWS CDK","language":"Python","stars":42,"topics":["aws-cdk","elasticsearch","kafka","amazon-s3","amazon-msk","amazon-elasticsearch-service","aws-fargate","logstash","filebeat","kibana"],"license":"MIT-0","category":"data-pipelines","readme_excerpt":"Amazon Managed ELKK This repository contains an implementation example of a managed ELKK stack using the AWS Cloud Development Kit. This example uses Python. Table of Contents 1. Context 2. Prerequisites 3. Amazon Virtual Private Cloud 4. Amazon Managed Streaming for Apache Kafka 5. Filebeat 6. Amazon Elasticsearch Service 7. Kibana 8. Amazon Athena 9. Logstash 10. Clean up Context The ELKK stack is a pipeline of services to support real-time reporting and analytics. Amazon services can provide a managed ELKK stack using the services Amazon Elasticsearch Service, Logstash on Amazon EC2 or on Amazon Elastic Container Services and Amazon Managed Streaming for Kafka. Kibana is included as a capability of the Amazon Elasticsearch Service. As part of a holistic solution Logstash in addition to outputting logs to Amazon Elasticsearch outputs the log to Amazon S3 for longer term storage. Amazon Athena can be used to directly query files in Amazon S3. Components Filebeat agents will be used to collect the logs from the application/host systems, and publish the logs to Amazon MSK. Filebeat agents are deployed on an Amazon EC2 instance to simulate log generation. Amazon Managed Streaming for Kafka (Amazon MSK) is used as a buffering layer to handle the collection of logs and manage the back-pressure from downstream components in the architecture. The buffering layer will provide recoverability and extensibility in the platform. The Logstash layer will perform a dual-purpose of reading ","default_branch":null,"files":null,"tree":[],"storefront":"/r/aws-samples","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/aws-samples/aws-cdk-managed-elkk/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}