{"repo":"fkie-cad/Logprep","free":true,"listed":false,"github":"https://github.com/fkie-cad/Logprep","clone":"git clone https://github.com/fkie-cad/Logprep.git","description":"log data pre processing, generation and shipping in python","language":"Python","stars":37,"topics":["logdata","preprocessing","python","kafka","opensearch","etl","soar","sre","log","loggenerator"],"license":"LGPL-2.1","category":"data-pipelines","readme_excerpt":"Logprep Introduction Logprep allows to collect, process and forward log messages from various data sources. Log messages are being read and written by so-called connectors. Currently, connectors for Kafka, Opensearch, S3, HTTP and JSON(L) files exist. The log messages are processed in serial by a pipeline of processors, where each processor modifies an event that is being passed through. The main idea is that each processor performs a simple task that is easy to carry out. Once the log message is passed through all processors in the pipeline the resulting message is sent to a configured output connector. Logprep is primarily designed to process log messages. Generally, Logprep can handle JSON messages, allowing further applications besides log handling. - About Logprep - Installation - Deployment Examples - Event Generation - Documentation - Container signatures - Container SBOM - Contributing - License - Changelog About Logprep Pipelines Logprep processes incoming log messages with a configured pipeline that can be spawned multiple times via multiprocessing. The following chart shows a basic setup that represents this behaviour. The pipeline consists of three processors: the Dissector , Geo-IP Enricher and the Dropper . Each pipeline runs concurrently and takes one event from it's Input Connector . Once the log messages is fully processed the result will be forwarded to the Output Connector , after which the pipeline will take the next message, repeating the processing cycle","default_branch":null,"files":null,"tree":[],"storefront":"/r/fkie-cad","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/fkie-cad/Logprep/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}