{"repo":"nitish9413/open_auto_loader","free":true,"listed":false,"github":"https://github.com/nitish9413/open_auto_loader","clone":"git clone https://github.com/nitish9413/open_auto_loader.git","description":"OpenAutoLoader: A lightweight, open-source alternative to Databricks Auto Loader. Built with Polars and SQLite for efficient, incremental file ingestion.","language":"Python","stars":16,"topics":["autoloader","data-engineering","databrick","delta-lake","delta-rs","etl","polars","polars-dataframe","python"],"license":"MIT","category":"data-pipelines","readme_excerpt":"🚀 OpenAutoLoader OpenAutoLoader is a high-performance, incremental data ingestion engine. It bridges the gap between raw cloud storage and production-ready Delta Lakes using the lightning-fast Polars Rust engine . Stop writing complex Spark jobs for simple file ingestion. OpenAutoLoader provides a \"Databricks-style\" Auto Loader experience in a lightweight Python package. ----- 💡 Why OpenAutoLoader? Traditional ingestion often requires heavy JVM clusters (Spark) or manual file tracking. OpenAutoLoader changes that: Zero-Spark Overhead : Runs on standard Python environments with Rust-level performance. Exactly-Once Processing : Integrated SQLite checkpointing ensures no duplicate data, even if a job restarts. Schema First : Automatically infers, saves, and enforces JSON schema contracts to prevent data corruption. Cloud Native : A single API for Local, S3, Azure Blob (ABFSS), and GCS. ----- 🛠️ Installation ----- 🚀 Quick Start: S3 to Delta Lake ----- 🏗️ Architecture: How it Works 1. Scanner : Uses fsspec to identify new files since the last successful batch id . 2. Schema Guard : Checks the file header against the stored JSON contract in schema path . 3. Polars Engine : Streams the data using sink delta() , minimizing memory footprint. 4. Metadata Injection : Automatically adds batch id , processed at , and source file to every row for full auditability. 5. Committer : Updates the SQLite checkpoint only after a successful Delta write. ----- 📋 Compatibility Matrix Feature L","default_branch":null,"files":null,"tree":[],"storefront":"/r/nitish9413","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/nitish9413/open_auto_loader/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}