{"repo":"larrabee/s3sync","free":true,"listed":false,"github":"https://github.com/larrabee/s3sync","clone":"git clone https://github.com/larrabee/s3sync.git","description":"Really fast sync tool for S3","language":"Go","stars":570,"topics":["sync","s3","golang","multithreading","amazon","ceph-radosgw","sync-s3-bucket"],"license":"GPL-3.0","category":"dev-tools","readme_excerpt":"S3Sync Really fast sync tool for S3 Features Multi-threaded file downloading/uploading Can sync to multiple ways: S3 to local FS Local FS to S3 S3 to S3 Retrying on errors Live statistics Rate limiting by objects Rate limiting by bandwidth Flexible filters by extension, Content-Type, ETag and object mtime Key feature: very high speed. Avg listing speed around 5k objects/sec for S3. With 128 workers we get avg sync speed around 2k obj/sec (small objects 1-20 kb) (limited by 1Gb uplink). Limitations Each object is loaded into RAM. So you need RAM. If you don't have enough RAM, you can use swap. A large (32-64 Gb) swap on SSD does not affect the tool performance. This happened because the tool was designed to synchronize billions of small files and optimized for this workload. To avoid this you can use streaming storage drivers (now available only for S3 and FS). It's uses less RAM, but slower on small objects. Usage Examples: Sync Amazon S3 bucket to FS: Sync S3 bucket with custom endpoint to FS: Sync directory (/test) from Amazon S3 bucket to FS: Sync directory from local FS to Amazon S3: Sync directory from local FS to Amazon S3 bucket directory: Sync one Amazon bucket to another Amazon bucket: Sync S3 bucket with custom endpoint to another bucket with custom endpoint: Sync one Amazon bucket directory to another Amazon bucket: Use streaming S3 transfers (reduces memory, might be slower for small files): SOURCE and TARGET should be a directory. Syncing of single file are not s","default_branch":null,"files":null,"tree":[],"storefront":"/r/larrabee","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/larrabee/s3sync/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}