{"repo":"yassnemo/iot-data-and-anomaly-detection-ml-system","free":true,"listed":false,"github":"https://github.com/yassnemo/iot-data-and-anomaly-detection-ml-system","clone":"git clone https://github.com/yassnemo/iot-data-and-anomaly-detection-ml-system.git","description":"Built with 100% open-source technologies, capable of processing 10,000+ sensors at 1 message/second with ML-powered anomaly detection.","language":"Python","stars":17,"topics":["anomaly-detection","iot","kafka","real-time","spark-streaming","docker","ml"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Real-Time IoT Anomaly Detection System A production-ready system for detecting anomalies in 10,000+ IoT sensors using Apache Kafka, Spark Structured Streaming, TensorFlow, and MinIO. 📖 Read the full documentation site → An interactive, end-to-end walkthrough of the architecture, streaming pipeline, ML model, and results. Source lives in docs/ and is published with GitHub Pages. Architecture - Data Ingestion : Async Kafka producer simulating 10K sensors @ 1 msg/sec - Stream Processing : Spark Structured Streaming with sliding window feature extraction - ML Model : LSTM Autoencoder for anomaly detection - Storage : MinIO for features/scores (Parquet) - Serving : TensorFlow Serving for real-time inference - Monitoring : Prometheus + Grafana for metrics Quick Start (Docker Compose) Access Services - Grafana : http://localhost:3000 (admin/admin) - Prometheus : http://localhost:9090 - MinIO Console : http://localhost:9001 (minioadmin/minioadmin) - Spark UI : http://localhost:8080 Kubernetes (k3s) Deployment Scaling Guide Kafka Partitions Spark Executors Edit docker-compose.yml : Producer Throughput Tuning for 95% Detection 1. Data Quality : Ensure training data includes diverse anomaly patterns 2. Model Hyperparameters : - lstm units : 128-256 - sequence length : 50-100 - threshold percentile : 95-99 3. Feature Engineering : Adjust window size (T=60s) and slide (S=10s) 4. Training : Use more epochs (50-100) with early stopping Configuration Environment Variables Create .env file: ","default_branch":null,"files":null,"tree":[],"storefront":"/r/yassnemo","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/yassnemo/iot-data-and-anomaly-detection-ml-system/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}