{"repo":"Dicklesworthstone/automatic_log_collector_and_analyzer","free":true,"listed":false,"github":"https://github.com/Dicklesworthstone/automatic_log_collector_and_analyzer","clone":"git clone https://github.com/Dicklesworthstone/automatic_log_collector_and_analyzer.git","description":"Replace Splunk in your small company with this one weird trick!","language":"Python","stars":429,"topics":["ingestion","log","logging","splunk"],"license":null,"category":"analytics","readme_excerpt":"Automatically Download and Analyze Log Files from Remote Machines This application is designed to collect and analyze logs from remote machines hosted on Amazon Web Services (AWS) and other cloud hosting services. Note : This application was specifically designed for use with Pastel Network's log files. However, it can be easily adapted to work with any log files by modifying the parsing functions, data models, and specifying the location and names of the log files to be downloaded. It is compatible with log files stored in a standard format, where each entry is on a separate line and contains a timestamp, a log level, and a message. The application has been tested with log files several gigabytes in size from dozens of machines and can process all of it in minutes. It is designed for Ubuntu 22.04+, but can be adapted for other Linux distributions. Customization To adapt this application for your own use case, refer to the included sample log files and compare them to the parsing functions in the code. You can also modify the data models to store log entries as desired. Features The application consists of various Python scripts that perform the following functions: Connect to Remote Machines : Using the boto3 library for AWS instances and an Ansible inventory file for non-AWS instances, the application establishes SSH connections to each remote machine. Download and Parse Log Files : Downloads specified log files from each remote machine and parses them. The parsed log entri","default_branch":null,"files":null,"tree":[],"storefront":"/r/Dicklesworthstone","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Dicklesworthstone/automatic_log_collector_and_analyzer/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}