{"repo":"thevxn/xilt","free":true,"listed":false,"github":"https://github.com/thevxn/xilt","clone":"git clone https://github.com/thevxn/xilt.git","description":"A utility for parsing Common and Combined Log Format (CLF) log files.","language":"Go","stars":32,"topics":["go","golang","log","log-parser","log-parsers","log-parsing","parser","sqlite","fortheloveofcode"],"license":"MIT","category":"databases-storage","readme_excerpt":"xilt A utility for parsing Common and Combined Log Format (CLF) log files and storing them in SQLite for further analysis. Built with concurrency in mind. Usage Install Run - Default logFilePath = ./access.log - Default dbFilePath = ./logs.db Flags Indexes Currently, if the -i flag is used, the following indexes are created: - IP - Method - Referer - Route - TimestampUTC - TimestampUTC + IP This may be subject to change in the future depending on further fine-tuning and configurable parameters extension. Example logs.txt : Run xilt -i logs.txt logs.db : The resulting logs.db file: Benchmark Currently, a basic benchmark is available using the following hardware: - AMD Ryzen™ 7 5800X3D - 32 GB DDR4 RAM A large log file containing 17 379 453 logs ( 1.63 GB in size) takes 1 minute, 55 seconds on average to be parsed and stored in SQLite if not using indexes. Index creation on a table of such size prolongs the duration to 4 minutes. All logs in the file are the same: 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] \"GET /apache pb.gif HTTP/1.0\" 200 2326 . Without indexes: With indexes: Memory Usage Using the default presets ( batchSize=5000 , avgLogSize=0.001 , maxMemUsage=100 ) to parse the aforementioned benchmark file leads to an average RAM usage of 85 MB. Build & Run Locally","default_branch":null,"files":null,"tree":[],"storefront":"/r/thevxn","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/thevxn/xilt/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}