{"repo":"arpitbbhayani/s4db","free":true,"listed":false,"github":"https://github.com/arpitbbhayani/s4db","clone":"git clone https://github.com/arpitbbhayani/s4db.git","description":"A lightweight embedded key-value store where keys and values are strings. Data is written to numbered binary files on disk and synced to S3. Values are Snappy-compressed. An in-memory index tracks the exact file and byte offset for every live key, so reads never scan - they seek directly.","language":"Python","stars":37,"topics":["bitcask","database","embedded-database","kv-store","s3","s4db"],"license":"MIT","category":"databases-storage","readme_excerpt":"s4db - Simple DB on S3 A lightweight embedded key-value store where keys are strings and values are strings or bytes. Data is written to numbered binary files on disk and synced to S3. Values are Snappy-compressed when compression helps. An in-memory index tracks the exact file and byte offset for every live key, so reads never scan - they seek directly. Concurrent writers are safe: each writer's files live in their own namespace, and index commits are compare-and-swapped (per-key last-writer-wins). Installation s4db requires python-snappy , which links against the native Snappy C library. Quick start On init , the index is downloaded from S3 into memory. If no index exists, the database starts empty. No local directory is created or used until a write operation ( put / delete ) is called. When to use s4db s4db fits workloads that need durable key-value semantics on ephemeral compute - without a running database. Good fits - Lambda / serverless state - load index on cold start ( 50 ms S3 RTT), mutate in memory, upload() once before return. No VPC, no connections. - Batch pipeline checkpoints - write processed keys as you go, upload() periodically. Restart resumes from the existing index. - Read-heavy config / lookup tables - write once, download() on each worker at startup, all reads from local disk at 0.009 ms. - ETL joins - pre-load a lookup table into s4db, upload to S3, workers download at startup. 0.009 ms median lookup vs 50 ms per S3 GET. - Experiment tracking - log me","default_branch":null,"files":null,"tree":[],"storefront":"/r/arpitbbhayani","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/arpitbbhayani/s4db/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}