33 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	57f5ad188b	operations: fix incorrect modtime on some multipart transfers ... In this commit 6a0a54ab97 operations: fix missing metadata for multipart transfers to local disk We broke the setting of modification times when doing multipart transfers from a backend which didn't support metadata to a backend which did support metadata. This was fixed by setting the "mtime" in the metadata if it was missing.	2024-06-08 17:44:11 +01:00
Nick Craig-Wood	6a0a54ab97	operations: fix missing metadata for multipart transfers to local disk ... Before this change multipart downloads to the local disk with --metadata failed to have their metadata set properly. This was because the OpenWriterAt interface doesn't receive metadata when creating the object. This patch fixes the problem by using the recently introduced Object.SetMetadata method to set the metadata on the object after the download has completed (when using --metadata). If the backend we are copying to is using OpenWriterAt but the Object doesn't support SetMetadata then it will write an ERROR level log but complete successfully. This should not happen at the moment as only the local backend supports metadata and OpenWriterAt but it may in the future. It also adds a test to check metadata is preserved when doing multipart transfers. Fixes #7424	2024-05-14 12:51:03 +01:00
Nick Craig-Wood	dd0e5b9a7f	operations: use built in io.OffsetWriter for go1.20	2024-01-15 16:22:07 +00:00
Nick Craig-Wood	d5d28a7513	operations: fix overwrite of destination when multi-thread transfer fails ... Before this change, if a multithread upload failed (let's say the source became unavailable) rclone would finalise the file first before aborting the transfer. This caused the partial file to be written which would overwrite any existing files. This was fixed by making sure we Abort the transfer before Close-ing it. This updates the docs to encourage calling of Abort before Close and updates writerAtChunkWriter to make sure that works properly. This also reworks the tests to detect this and to make sure we upload and download to each multi-thread capable backend (we were only downloading before which isn't a full test). Fixes #7071	2023-11-24 11:19:58 +00:00
Nick Craig-Wood	1f9a79ef09	operations: use less memory when doing multithread uploads ... For uploads which are coming from disk or going to disk or going to a backend which doesn't need to seek except for retries this doesn't buffer the input. This dramatically reduces rclone's memory usage. Fixes #7350	2023-11-20 18:07:05 +00:00
Vitor Gomes	48a8bfa6b3	operations: fix OpenOptions ignored in copy if operation was a multiThreadCopy	2023-10-11 11:19:03 +01:00
Pat Patterson	71f2883562	operations: ensure concurrency is no greater than the number of chunks - fixes #7299	2023-09-13 12:12:27 +01:00
Nick Craig-Wood	2db0e23584	backends: change OpenChunkWriter interface to allow backend concurrency override ... Before this change the concurrency used for an upload was rather inconsistent. - if size below `--backend-upload-cutoff` (default 200M) do single part upload. - if size below `--multi-thread-cutoff` (default 256M) or using streaming uploads (eg `rclone rcat) do multipart upload using `--backend-upload-concurrency` to set the concurrency used by the uploader. - otherwise do multipart upload using `--multi-thread-streams` to set the concurrency. This change makes the default for the concurrency used be the `--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger than the `--backend-upload-concurrency` then that will be used instead. This means that if the user sets `--backend-upload-concurrency` then it will be obeyed for all multipart/multi-thread transfers and the user can override them all with `--multi-thread-streams`. See: #7056	2023-09-03 11:47:05 +01:00
Nick Craig-Wood	df5d92d709	operations: fix terminology in multi-thread copy	2023-08-25 16:31:31 +01:00
Nick Craig-Wood	1b5b36523b	operations: fix accounting for multi-thread transfers ... This stops the accounting moving in large chunks and gets it to move as the data is read out of the buffer.	2023-08-24 17:51:36 +01:00
Nick Craig-Wood	2f424ceecf	operations: don't buffer when a backend implements OpenWriterAt ... In this case we don't seek on errors so no need for seeking.	2023-08-24 17:51:36 +01:00
Nick Craig-Wood	d4cff1ae19	operations: add abort on exit to multithread copy	2023-08-24 12:39:27 +01:00
Nick Craig-Wood	f5753369e4	operations: multipart: don't buffer transfers to local disk #7056	2023-08-24 12:39:27 +01:00
Nick Craig-Wood	0d0bcdac31	fs: add context.Ctx to ChunkWriter methods ... WriteChunk in particular needs a different context from that which OpenChunkWriter was used with so add it to all the methods.	2023-08-24 12:39:27 +01:00
Nick Craig-Wood	f3bd02f0ef	operations: fix and tidy multithread code ... - fix docs and error messages for multithread - use sync/errgroup built in concurrency limiting - re-arrange multithread code - don't continue multi-thread uploads if one part fails	2023-08-24 12:39:27 +01:00
Nick Craig-Wood	b4e3332e02	fs: introduces aliases for OpenWriterAtFn and OpenChunkWriterFn	2023-08-24 12:39:27 +01:00
Vitor Gomes	181fecaec3	multithread: refactor multithread operation to use OpenChunkWriter if available #7056 ... If the feature OpenChunkWriter is not available, multithread tries to create an adapter from OpenWriterAt to OpenChunkWriter.	2023-08-12 17:55:01 +01:00
Paulo Schreiner	fcb912a664	fs: allow setting a write buffer for multithread ... when multi-thread downloading is enabled, rclone used to send a write to disk after every read, resulting in a lot of small writes to different locations of the file. depending on the underlying filesystem or device, it can be more efficient to send bigger writes.	2023-06-23 18:44:43 +01:00
Nick Craig-Wood	1f9c962183	operations: reopen downloads on error when using check --download and cat ... Before this change, some parts of operations called the Open method on objects directly, and some called NewReOpen to make an object which can re-open itself on errors. This adds a new function operations.Open which should be called instead of fs.Object.Open to open a reliable stream of data and changes all call sites to use that. This means `rclone check --download` and `rclone cat` will re-open files on failures. See: https://forum.rclone.org/t/does-rclone-support-retries-for-check-when-using-download-flag/38641	2023-06-10 14:42:29 +01:00
Nick Craig-Wood	46a323ae14	operations: Don't use multi-thread copy if the backend doesn't support it #6915	2023-05-09 17:40:58 +01:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Nick Craig-Wood	421585dd72	accounting: add context to Account and propagate changes #3257 ... This is preparation for getting the Accounting to check the context, buf first we need to get it in place. Since this is one of those changes that makes lots of noise, this is in a seperate commit.	2020-07-28 16:41:17 +01:00
Tim Gallant	9bf3d3da4c	fs: add UploadHeaders, DownloadHeaders to Update/Put/Open options	2020-04-23 11:07:21 +01:00
Nick Craig-Wood	7f15cc9556	operations: make ReOpen and NewReOpen public for re-use elsewhere	2020-04-19 15:18:49 +01:00
Nick Craig-Wood	bde0334bd8	operations: fix setting the timestamp on Windows for multithread copy ... Before this fix we attempted to set the modification time on the file when it was open. This works fine on Linux but not on Windows. The test was also incorrect testing the source file rather than the destination file. This closes the file before setting the modification time and fixes the tests. Fixes #3994	2020-02-24 17:30:09 +00:00
Nick Craig-Wood	eb087a3b04	operations: disable multi thread copy for local to local copies #3419 ... ...unless --multi-thread-streams has been set explicitly	2019-08-14 15:07:38 +01:00
Nick Craig-Wood	57d5de6fba	build: fix up package paths after repo move ... git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`	2019-07-28 18:47:38 +01:00
Aleksandar Jankovic	be0464f5f1	accounting: change stats interface ... This is done to make clear ownership over accounting object and prepare for removing global stats object. Stats elapsed time calculation has been altered to account for actual transfer time instead of stats creation time.	2019-07-28 14:48:19 +01:00
Aleksandar Jankovic	7b2b396d37	context: fix errgroup interaction with context ... Fixes #3307	2019-07-01 11:51:51 +01:00
Aleksandar Jankovic	f78cd1e043	Add context propagation to rclone ... - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.	2019-06-19 11:59:46 +01:00
Nick Craig-Wood	7d70e92664	operations: enable multi threaded downloads - Fixes #2252 ... This implements the --multi-thread-cutoff and --multi-thread-streams flags to control multi thread downloading to the local backend.	2019-05-11 23:35:19 +01:00