1
mirror of https://github.com/rclone/rclone synced 2024-11-26 04:07:22 +01:00
rclone/backend/s3
Nick Craig-Wood 7e6fac8b1e s3: re-implement multipart upload to fix memory issues
There have been quite a few reports of problems with the multipart
uploader using too much memory and not retrying possible errors.

Before this change the multipart uploader used the s3manager
abstraction in the AWS SDK.  There are numerous bug reports of this
using up too much memory.

This change re-implements a much simplified version of the s3manager
code specialized for rclone's purposes.

This should use much less memory and retry chunks properly.

See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563
See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405
See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2020-01-03 22:19:28 +00:00
..
s3_test.go s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier 2019-09-14 09:18:55 +01:00
s3.go s3: re-implement multipart upload to fix memory issues 2020-01-03 22:19:28 +00:00
v2sign.go s3: fix v2 signer on files with spaces - fixes #2438 2018-10-14 00:10:29 +01:00