1
mirror of https://github.com/rclone/rclone synced 2024-11-21 22:50:16 +01:00
rclone/backend/s3
Nick Craig-Wood 2db0e23584 backends: change OpenChunkWriter interface to allow backend concurrency override
Before this change the concurrency used for an upload was rather
inconsistent.

- if size below `--backend-upload-cutoff` (default 200M) do single part upload.

- if size below `--multi-thread-cutoff` (default 256M) or using streaming
  uploads (eg `rclone rcat) do multipart upload using
  `--backend-upload-concurrency` to set the concurrency used by the uploader.

- otherwise do multipart upload using `--multi-thread-streams` to set the
  concurrency.

This change makes the default for the concurrency used be the
`--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger
than the `--backend-upload-concurrency` then that will be used instead.

This means that if the user sets `--backend-upload-concurrency` then it will be
obeyed for all multipart/multi-thread transfers and the user can override them
all with `--multi-thread-streams`.

See: #7056
2023-09-03 11:47:05 +01:00
..
gen_setfrom.go s3: refactor to use generated code instead of reflection to copy structs 2022-08-05 16:45:23 +01:00
s3_internal_test.go mount: fix mount failure on macOS with on the fly remote 2023-06-23 14:12:03 +01:00
s3_test.go s3: empty directory markers - #3453 2023-04-28 14:31:05 +01:00
s3.go backends: change OpenChunkWriter interface to allow backend concurrency override 2023-09-03 11:47:05 +01:00
setfrom.go s3: refactor to use generated code instead of reflection to copy structs 2022-08-05 16:45:23 +01:00
v2sign.go s3: v3sign: add missing subresource delete 2023-05-14 11:25:52 +01:00