{"repo":"Starofall/S3HyperSync","free":true,"listed":false,"github":"https://github.com/Starofall/S3HyperSync","clone":"git clone https://github.com/Starofall/S3HyperSync.git","description":"S3HyperSync is a high-performance, memory-efficient, and cost-effective tool for synchronizing files between S3-compatible storage services.","language":"Scala","stars":102,"topics":["aws","clouflare","s3","s3cmd"],"license":"MIT","category":"deployment-docker-iac","readme_excerpt":"S3HyperSync S3HyperSync is a high-performance, memory-efficient, and cost-effective tool for synchronizing files between S3-compatible storage services. Optimized for speed, reliability, and minimizing AWS costs, it's ideal for large-scale data synchronization and backup tasks. Utilizing Pekko, it adopts a stream-only approach to maintain low memory requirements. Origin Developed for creating daily backups of huge S3 buckets with millions of files and terabyte of data to an seperate AWS account. Cost Effective To sync large S3 buckets, S3HyperSync compares directories using two iterator streams for the source and target, reducing the need for costly GetObject requests, especially for DEEP ARCHIVE storage. It minimizes expensive MultiPart uploads, as they count as multiple PutObject calls. Performance Performance tests on AWS Fargate show iteration speeds between 8,000 to 100,000 files per second with the UUID booster feature. Copy speeds reach around 600MB/s on a c6gn.4xlarge or 800 files per second for smaller files. UUID Booster The UUID booster feature can be used if data is suffixed with a uuid. E.g. s3://bucket/videos/$UUID In this case the tool creates 16 iterators and processes them in parallel for extremly fast bucket comparison. Installation Download the JAR file from the Release Section or build it yourself with sbt assembly. Usage Contributing We welcome contributions from the community. If you find a bug or have a feature request, please open an issue on GitHub. I","default_branch":null,"files":null,"tree":[],"storefront":"/r/Starofall","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Starofall/S3HyperSync/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}