{"repo":"iamkaroko/tx-processor","free":true,"listed":false,"github":"https://github.com/iamkaroko/tx-processor","clone":"git clone https://github.com/iamkaroko/tx-processor.git","description":"High-throughput Go transaction processor with PostgreSQL analytics, Redis caching, and a query API","language":"Go","stars":17,"topics":["analytics","concurrency","docker","go","golang","ndjson","postgresql","redis","rest-api","worker-pool"],"license":"MIT","category":"databases-storage","readme_excerpt":"Tx Processor tx-processor processes newline-delimited JSON transaction data, aggregates per-user analytics concurrently in Go, stores the results in PostgreSQL, and exposes an HTTP API for querying the processed analytics. It is designed around worker pools, worker-local aggregation, bulk database writes, and optional Redis-backed read caching. Table of Contents - Overview - Approach - Trade-offs - Project Structure - Prerequisites - Running the Application - API Endpoints - Testing Overview The processor reads an NDJSON file of transactions, fans the work out across multiple workers, aggregates analytics in memory, and flushes the final result to PostgreSQL in bulk. The API layer exposes endpoints for: - total orders per user - total spending per user - top users by order volume - anomaly detection Redis can be enabled to speed up repeated single-user reads. Approach Transaction Format Transactions are stored as NDJSON : one JSON object per line. This allows the processor to stream records incrementally instead of loading the entire file into memory. Compared with a JSON array, this keeps memory usage low and makes the approach suitable for large datasets. Processing Model The processor uses a buffered channel and a pool of worker goroutines. - One goroutine reads lines from the file. - Lines are pushed onto a buffered channel. - Multiple workers consume lines concurrently. - Each worker aggregates into its own private in-memory map. - After all workers finish, the worker-lo","default_branch":null,"files":null,"tree":[],"storefront":"/r/iamkaroko","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/iamkaroko/tx-processor/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}