Commit Graph

7925 Commits

Author SHA1 Message Date
Nick Craig-Wood 39db8caff1 cache,chunker,combine,compress,crypt,hasher,union: implement MkdirMetadata and related Features 2024-02-28 16:09:04 +00:00
nielash 0297542f6b cache,chunker,combine,compress,crypt,hasher,union: implement DirSetModTime (if supported by wrapped remote) 2024-02-28 16:09:04 +00:00
nielash 17c0ecc72c sftp: implement DirSetModTime 2024-02-28 16:09:04 +00:00
nielash cbcb295185 drive: implement DirSetModTime 2024-02-27 19:59:13 +00:00
nielash 67e3725205 local: implement DirSetModTime 2024-02-27 19:59:13 +00:00
Nick Craig-Wood 61d76ae47d fstests: add integration tests for Directory Metadata and ModTime 2024-02-27 19:59:13 +00:00
Nick Craig-Wood fd1ca2dfe8 fs: allow Metadata calls to be called with Directory or Object
This involved adding the Fs() method to DirEntry as it is needed in
the metadata mapper.

Unspecialised fs.Dir objects will return a new fs.Unknown from their
Fs() methods as they are not specific to any given Fs.
2024-02-27 10:56:19 +00:00
Nick Craig-Wood e1032f693f fs: add DirWrapper for wrapping Directory-s with optional methods 2024-02-27 10:56:19 +00:00
Nick Craig-Wood a4cadd1128 fs: add Directory Metadata flags for backends and interfaces
Add backend flags
- ReadDirMetadata
- WriteDirMetadata
- WriteDirSetModTime
- UserDirMetadata
- DirModTimeUpdatesOnWrite

Add Metadata/SetMetadata for directories.

Add MkdirMetadata optional feature
2024-02-27 10:56:19 +00:00
nielash 6da52d76a7 fs: implement DirSetModTime optional feature 2024-02-22 11:13:54 +00:00
Nick Craig-Wood 71a1bbb2be errcount: factor errcount abstraction from operations 2024-02-22 11:13:54 +00:00
Nick Craig-Wood 8f0e9f9f6b mega: fix panic with go1.22
Before this fix rclone would crash with

    panic: encoding alphabet includes duplicate symbols

When compiled with go1.22. This was fixed upstream in

https://github.com/t3rm1n4l/go-mega/issues/48

And this just pulls in the fix.

Fixes #7639
2024-02-21 18:41:44 +00:00
Nick Craig-Wood 072d1f10ab serve webdav: fix --baseurl without leading /
The webdav server needs the prefix passed to it with a leading /
otherwise it does not remove it properly.

The docs state that a leading slash is optional so this patch adds one
if not present.

See: https://forum.rclone.org/t/cant-rename-files-in-rclone-serve-webdav-with-baseurl-maybe-wrong-handling-of-move-request-method/44637
2024-02-21 18:08:44 +00:00
Nick Craig-Wood 5014348229 Add Anders Swanson to contributors 2024-02-21 18:08:44 +00:00
Nick Craig-Wood ed78ac7c92 Add Joe Cai to contributors 2024-02-21 18:08:44 +00:00
Nick Craig-Wood 53d873d60d Add Dan McArdle to contributors 2024-02-21 18:08:44 +00:00
Nick Craig-Wood f2c35fdec6 Add Gabriel Ramos to contributors 2024-02-21 18:08:44 +00:00
Nick Craig-Wood 1c69b20ed7 Add Jack Provance to contributors 2024-02-21 18:08:44 +00:00
nielash 547c635552 mailru: add override for TestApplyTransforms - #7591
mailru is unable to handle filenames with certain combining characters (for
example: йěáñ), and is therefore incapable of testing ApplyTransforms. (It is
also therefore incapable of fully supporting --no-unicode-normalization.)

The same override is applied to chunker when wrapping mailru.
2024-02-21 18:02:19 +00:00
nielash f0d9117ff3 linkbox: add override for TestFixCase - #7591
linkbox already has an override for TestCaseInsensitiveMoveFile, and being able
to handle case-insensitive moves is a prerequisite for TestFixCase.
2024-02-21 18:02:19 +00:00
nielash 9d2bd163c7 opendrive: fix moving file/folder within the same parent dir - #7591
Before this change, moving (renaming) a file or folder to a different name
within the same parent directory would fail, due to using the wrong API
operation ("/file/move_copy.json" and "/folder/move_copy.json", instead of the
separate "/file/rename.json" and "/folder/rename.json" that opendrive has for
this purpose.)

After this change, Move and DirMove check whether the move is within the same
parent dir. If so, "rename" is used. If not, "move_copy" is used, like before.
2024-02-21 18:02:19 +00:00
Anders Swanson db8fb5ceda oracleobjectstorage: supports workload identity authentication for OKE
Signed-off-by: Anders Swanson <anders.swanson@oracle.com>
2024-02-20 16:25:59 +00:00
Joe Cai a1e66cc5e8 swift: Avoid unnecessary container versioning check
Container versioning check is only needed for non-empty large objects.
2024-02-20 15:52:25 +00:00
nielash 7b8bbe531e nfsmount: fix --volname being ignored #7503
Before this change, nfsmount ignored the --volname flag. After this change, the --
volname flag is respected, making it possible to set a custom volume name.

macOS users should note that Finder will show the correct volume name in most
places, but a notable exception is the sidebar, which will show "localhost".
This seems to be a system limitation (at least without `sudo`), but see the
discussion at https://github.com/rclone/rclone/issues/7503#issuecomment-1933997678
for some possible workarounds.
2024-02-18 05:08:59 -05:00
nielash 0e2f1d64e3 nfsmount: fix exit after external unmount #7503
Before this change, if a user unmounted externally (for example, via the Finder
UI), rclone would not be aware of this and wait forever to exit -- effectively
causing a deadlock that would require Ctrl+C to terminate.

After this change, when the handler detects an external unmount, it calls a
function which allows rclone to cleanly shutdown the VFS and exit.
2024-02-18 05:08:59 -05:00
nielash 5638a3841f serve nfs: fix writing files via Finder on macOS - fixes #7503
Before this change, writing files to an `nfsmount` via Finder on macOS would
cause critical errors, rendering `nfsmount` effectively unusable on macOS. This
change fixes the issue so that writes via Finder should be possible.

The issue was primarily caused by the handler's HandleLimit being set to -1. -1 is
the correct default for a NullAuthHandler, but not for a CachingHandler, which
interprets -1 not as "no limit" but as "no cache".

This change sets a high default of 1000000, and gives the user control over it
with a new --nfs-cache-handle-limit flag (available in both `serve nfs` and
`nfsmount`. A minimum of 5 is enforced, as any lower than this will be
insufficient to support directory listing.
2024-02-18 05:08:59 -05:00
Dan McArdle 6986a43b68 bisync: delete flushCache() function from tests
The flushCache() function has a bug that causes it to never actually
flush the cache. Specifically, it checks whether DirCacheFlush is nil,
but never calls it.

The tests are already passing without flushing the dir cache, so this
commit just deletes flushCache() and its call sites.

Fixes rclone/rclone#7623
2024-02-18 04:14:51 -05:00
Oksana Zhykina 11c6489fd1 quatrix: add option to skip project folders 2024-02-18 07:38:19 +01:00
Gabriel Ramos 43823bc925
webdav: reduce priority of chunks upload log 2024-02-18 07:29:23 +01:00
dependabot[bot] a3b661be0d build(deps): bump golangci/golangci-lint-action from 3 to 4
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3 to 4.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-18 07:25:50 +01:00
Jack Provance f113c68b13
docs: Fix a heading level in webdav.md documentation (#7631)
This fixes a heading problem under the "Provider Notes" section.
2024-02-18 07:16:23 +01:00
nielash 137f7f62fb sync: use operations.DirMove instead of sync.MoveDir for --fix-case - #7591
This should be more efficient for the purposes of --fix-case, as operations.DirMove
accepts `srcRemote` and `dstRemote` arguments, while sync.MoveDir does not.

This also factors the two-step-move logic to operations.DirMoveCaseInsensitive, so
that it is reusable by other commands.
2024-02-13 15:07:41 -05:00
nielash dfe76570a1 operations: skip backends incapable of testing TestApplyTransforms - #7591
This adds a step to detect whether the backend is capable of supporting the
feature, and skips the test if not. A backend can be incapable if, for example,
it is non-case-preserving or automatically converts NFD to NFC.
2024-02-13 15:07:41 -05:00
nielash f4c058e13e bisync: use global --retries and --retries-sleep flags instead of overriding 2024-02-12 13:24:54 -05:00
nielash 407a0f3733 cmd: refactor --retries and --retries-sleep to global config
This change moves the --retries and --retries-sleep flags/variables from cmd to
config (consistent with --low-level-retries), so that they can be more easily
referenced from subcommands.
2024-02-12 13:24:54 -05:00
nielash b14269fd23 bisync: add support for --retries-sleep - fixes #7555
Before this change, bisync supported --retries but not --retries-sleep.
This change adds support for --retries-sleep.
2024-02-12 13:24:54 -05:00
nielash 76b7bcd4d7 bisync: reset errors between retries
Before this change, in the event of a retryable error, bisync would always retry
the maximum number of times allowed by the `--retries` flag, even if one of the
retries was successful. This change fixes the issue, so that bisync moves on
after the first successful retry.
2024-02-12 13:24:54 -05:00
nielash 782ab3f582 bisync: clean up docs
(as the flags in docs/content/bisync.md do not update automatically, unlike
docs/content/commands/rclone_bisync.md)
2024-02-12 13:24:54 -05:00
nielash 9c6325c131 backend: rename variables to fix CI lint test failures 2024-02-12 12:49:00 -05:00
Volodymyr 2abeda5961
quatrix: fix Content-Range header
This change does not actually affect uploads. Just to be right according to definition of Content-Range in
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range#range-end
2024-02-09 16:44:45 +00:00
nielash 885a543023 operations: use --download for TestApplyTransforms #7591
This makes it possible to run the test even on remotes without MD5 support.
2024-02-08 16:08:05 +00:00
nielash f3680d222c operations: fix TestCaseInsensitiveMoveFileDryRun on chunker integration tests #7591
It appears that ci.DryRun = true affects the behavior of r.WriteObject on
chunker only, and no other remotes. This change puts a quick bandaid on it by
setting it later on in the test, but perhaps the underlying issue warrants a
closer look at some point... is chunker checking ci.DryRun itself in a way that
no other remote does? If so, should it? (Does this break encapsulation?)
2024-02-08 16:08:02 +00:00
nielash d2b37cf61e operations: fix case-insensitive moves in operations.Move #7591
Before this change, operations.moveOrCopyFile had a special section to detect
and handle changing case of a file on a case insensitive remote, but
operations.Move did not. This caused operations.Move to fail for certain
backends that are incapable of renaming a file in-place to an equal-folding name.
(Not all case-insensitive backends have this limitation -- for example, Dropbox
does but macOS local does not.)

After this change, the special two-part-move section from
operations.moveOrCopyFile is factored out to its own function,
moveCaseInsensitive, which is then called from both operations.moveOrCopyFile
and operations.Move.
2024-02-08 16:07:57 +00:00
Nick Craig-Wood 83f61a9cfb s3: GCS provider: fix server side copy of files bigger than 5G
GCS gives NotImplemented errors for multi-part server side copies. The
threshold for these is currently set just below 5G so any files bigger
than 5G that rclone attempts to server side copy will fail.

This patch works around the problem by adding a quirk for GCS raising
--s3-copy-cutoff to the maximum. This means that rclone will never use
multi-part copies for files in GCS. This includes files bigger than
5GB which (according to AWS documentation) must be copied with
multi-part copy. However this seems to work with GCS.

See: https://forum.rclone.org/t/chunker-uploads-to-gcs-s3-fail-if-the-chunk-size-is-greater-than-the-max-part-size/44349/
See: https://issuetracker.google.com/issues/323465186
2024-02-08 14:53:30 +00:00
Nick Craig-Wood b206496f63 b2: clarify exactly what --b2-download-auth-duration does in the docs
See: https://forum.rclone.org/t/what-does-b2-download-auth-duration-mean/44504/
2024-02-08 09:39:53 +00:00
Nick Craig-Wood 24fdecf107 ftp: fix mkdir with rsftp which is returning the wrong code
On a successfull MKD, rsftp seems to return code 250 whereas we and
the RFC expects 257.

This patch makes rclone accept 250 here as well.

See: https://forum.rclone.org/t/rclone-pop-up-an-i-o-error-when-creating-a-folder-in-a-mounted-ftp-drive/44368/3
2024-02-07 22:09:56 +00:00
Nick Craig-Wood 9bd7262dfc Add DanielEgbers to contributors 2024-02-07 22:09:56 +00:00
DanielEgbers a0dff2dd9c
Seafile: Fix download/upload error when FILE_SERVER_ROOT is relative
A seafile server can be configured to use a relative URL as
FILE_SERVER_ROOT in order to support more than one hostname/ip. (see
https://github.com/haiwen/seahub/issues/3398#issuecomment-506920360 )

The previous backend implementation always expected an absolute
download/upload URL, resulting in an "unsupported protocol scheme"
error.

With this commit it supports both absolute and relative.
2024-02-05 11:48:51 +00:00
Nick Craig-Wood 91b54aafcc rc: add srcFs and dstFs to core/stats and core/transferred stats
Before this change it wasn't possible to see where transfers were
going from and to in core/stats and core/transferred.

When use in rclone mount in particular this made interpreting the
stats very hard.
2024-02-02 11:43:10 +00:00
Nick Craig-Wood 81a29e6895 Add Thomas Müller to contributors 2024-02-02 11:43:10 +00:00