1
mirror of https://github.com/rclone/rclone synced 2024-12-04 14:35:20 +01:00
Commit Graph

7785 Commits

Author SHA1 Message Date
Nick Craig-Wood
df6c573c99 Add Gachoud Philippe to contributors 2024-03-22 17:36:04 +00:00
Nick Craig-Wood
b7c06e5eb9 Add racerole to contributors 2024-03-22 17:36:04 +00:00
Nick Craig-Wood
e0e9ac50d3 Add John-Paul Smith to contributors 2024-03-22 17:36:04 +00:00
YukiUnHappy
f68d962c86 onedrive: make server-side copy to work in more scenarios 2024-03-22 17:29:38 +00:00
kapitainsky
6232cc123f docs: Proton Drive, correct typo
Proton Drive correct typo
2024-03-22 16:36:21 +00:00
Gachoud Philippe
a33576af7d
docs: drive: corrected relative path of scopes to absolute
and added some links to the reference
2024-03-22 12:26:34 +00:00
kapitainsky
2591703494 docs: clarify shell_type = none and ssh = behaviour
Discussed on the forum:

https://forum.rclone.org/t/can-rclone-be-made-to-work-with-an-sftp-server-confining-users-to-an-sftp-jail-and-no-login/44931
2024-03-21 15:08:56 +01:00
Kyle Reynolds
7803b4ed6c
fs: improve JSON Unmarshalling for Duration
Enhanced the UnmarshalJSON method for the Duration type to correctly
handle the special string 'off' and ensure large integers are parsed
accurately without floating-point rounding errors. This resolves
issues with setting and removing the MinAge filter through the rclone
rc command.

Fixes #3783

Co-authored-by: Kyle Reynolds <kyle.reynolds@bridgerphotonics.com>
2024-03-13 18:08:59 +00:00
racerole
00fb847662
docs: remove repeated words 2024-03-13 17:12:39 +00:00
Thomas Müller
c7bfadd10a owncloud: add config owncloud_exclude_mounts which allows to exclude mounted folders when listing remote resources 2024-03-13 17:09:10 +00:00
John-Paul Smith
ca903b9872
drive: backend query command
This command executes a list query in Google Drive’s native query
language and returns a JSON dump of matches. It’s useful for locating
files quickly in folders with a large number of files, where rclone’s
normal list command is slow due to client-side filtering.
2024-03-11 20:16:13 +00:00
Nick Craig-Wood
b7783f75a4 Start v1.67.0-DEV development 2024-03-10 12:14:00 +00:00
Nick Craig-Wood
b6013a5e68 Version v1.66.0 2024-03-10 11:22:43 +00:00
Nick Craig-Wood
b7422a4fc8 docs: update metadata docs with Move and Copy support 2024-03-09 14:13:18 +00:00
nielash
9b650d3517 hasher: look for cached hash if passed hash unexpectedly blank
Before this change, Hasher did not check whether a "passed hash" (hashtype
natively supported by the wrapped backend) returned from a backend was blank,
and would sometimes return a blank hash to the caller even when a non-blank hash
was already stored in the db. This caused issues with, for example, Google
Drive, which has SHA1 / SHA256 hashes for some files but not others
(https://rclone.org/drive/#sha1-or-sha256-hashes-may-be-missing) and sometimes also
does not have hashes for very recently modified files.

After this change, Hasher will check if the received "passed hash" is
unexpectedly blank, and if so, it will continue to try other enabled methods,
such as retrieving a value from the database, or possibly regenerating it.

https://forum.rclone.org/t/hasher-with-gdrive-backend-does-not-return-sha1-sha256-for-old-files/44680/9?u=nielash
2024-03-09 11:58:02 +00:00
nielash
ff0acfb568 hasher: fix error from trying to stop an already-stopped db
Before this change, Hasher would sometimes try to stop a bolt db that was
already stopped, resulting in an error. This change fixes the issue by checking
first whether the db is already stopped.

https://forum.rclone.org/t/hasher-with-gdrive-backend-does-not-return-sha1-sha256-for-old-files/44680/11?u=nielash
2024-03-09 11:58:02 +00:00
Nick Craig-Wood
ac830ddd42 sync: don't sync directory modtimes from backends which don't have directories
Some backends (like s3, swift, gcs, azureblob) don't have directories
(this can be overridden on some using the directory markers feature).

It therefore makes no sense to sync directory times from them as they
will all be a value made up by rclone (--default-time)

We use the feature flag CanHaveEmptyDirectories to mark backends
without real directory support and disable the directory modification
time syncing on those.
2024-03-09 11:28:15 +00:00
Nick Craig-Wood
f491efc85d sync: fix integration tests on chunker
The tests added in this commit needed a tweak for chunker

8c69455c37 sync: don't set dir modtimes if already set
2024-03-08 15:04:35 +00:00
Nick Craig-Wood
fcb182efce docs: add current sponsor logos in 2024-03-08 15:04:35 +00:00
nielash
1473de3f04 onedrive: add metadata support
This change adds support for metadata on OneDrive. Metadata (including
permissions) is supported for both files and directories.

OneDrive supports System Metadata (not User Metadata, as of this writing.) Much
of the metadata is read-only, and there are some differences between OneDrive
Personal and Business (see table in OneDrive backend docs for details).

Permissions are also supported, if --onedrive-metadata-permissions is set. The
accepted values for --onedrive-metadata-permissions are read, write, read,write, and
off (the default). write supports adding new permissions, updating the "role" of
existing permissions, and removing permissions. Updating and removing require
the Permission ID to be known, so it is recommended to use read,write instead of
write if you wish to update/remove permissions.

Permissions are read/written in JSON format using the same schema as the
OneDrive API, which differs slightly between OneDrive Personal and Business.
(See OneDrive backend docs for examples.)

To write permissions, pass in a "permissions" metadata key using this same
format. The --metadata-mapper tool can be very helpful for this.

When adding permissions, an email address can be provided in the User.ID or
DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an
ObjectID can be provided in User.ID. At least one valid recipient must be
provided in order to add a permission for a user. Creating a Public Link is also
supported, if Link.Scope is set to "anonymous".

Note that adding a permission can fail if a conflicting permission already
exists for the file/folder.

To update an existing permission, include both the Permission ID and the new
roles to be assigned. roles is the only property that can be changed.

To remove permissions, pass in a blob containing only the permissions you wish
to keep (which can be empty, to remove all.)

Note that both reading and writing permissions requires extra API calls, so if
you don't need to read or write permissions it is recommended to omit --onedrive-
metadata-permissions.

Metadata and permissions are supported for Folders (directories) as well as
Files. Note that setting the mtime or btime on a Folder requires one extra API
call on OneDrive Business only.

OneDrive does not currently support User Metadata. When writing metadata, only
writeable system properties will be written -- any read-only or unrecognized keys
passed in will be ignored.

TIP: to see the metadata and permissions for any file or folder, run:

rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read

See the OneDrive backend docs for a table of all the supported metadata
properties.
2024-03-08 14:48:54 +00:00
Nick Craig-Wood
4e07a72dc7 fs: Implement --no-update-dir-modtime to disable setting modification times on dirs 2024-03-07 17:20:24 +00:00
Nick Craig-Wood
99acee7ba0 operations: remove stray debug 2024-03-07 17:15:43 +00:00
Nick Craig-Wood
bda4f25baa s3: support metadata setting and mapping on server side Copy
Before this change the backend would not run the metadata mapper and
it would ignore metadata set when doing server side copies.
2024-03-07 14:44:45 +00:00
Nick Craig-Wood
9f2ce2c7fc drive: support metadata setting and mapping on server side Move,Copy
Before this change the backend would not run the metadata mapper and
it would ignore metadata set when doing server side moves or copies.
2024-03-07 14:44:45 +00:00
Nick Craig-Wood
6e85a39e99 local: support metadata setting and mapping on server side Move
Before this change the backend would not run the metadata mapper and
it would ignore metadata set when doing server side moves.
2024-03-07 14:44:45 +00:00
Nick Craig-Wood
24b4148b5e fs: add MetadataAsOpenOptions 2024-03-07 14:44:45 +00:00
Nick Craig-Wood
41b1250eaf fstests: add tests for Metadata on server side Move and Copy 2024-03-07 14:44:45 +00:00
Nick Craig-Wood
339d3e8ee6 netstorage,quatrix,seafile: fix Root to return correct directory when pointing to a file
This fixes the TestIntegration/FsMkdir/FsPutFiles/FsIsFile/FsRoot
integration test.
2024-03-07 14:44:45 +00:00
Nick Craig-Wood
5750795324 protondrive: fix encoding of Root method
This fixes the TestIntegration/FsMkdir/FsPutFiles/FsIsFile/FsRoot
integration test.
2024-03-07 14:44:45 +00:00
Nick Craig-Wood
cdcb8b2a0a Add huajin tong to contributors 2024-03-07 14:44:45 +00:00
huajin tong
b1ae7df556
docs: fix some comments
Signed-off-by: thirdkeyword <fliterdashen@gmail.com>
2024-03-07 12:57:15 +00:00
nielash
431524445e combine: fix operations.DirMove across upstreams - fixes #7661
Before this change, operations.DirMove would fail when moving a directory, if
the src and dest were on different upstreams of a combine remote.

The issue only affected operations.DirMove, and not sync.MoveDir, because they
checked for server-side-move support in different ways.

MoveDir checks by just trying it and seeing what error comes back. This works
fine for combine because combine returns fs.ErrorCantDirMove which MoveDir
understands what to do with.

DirMove, however, only checked whether the function pointer is nil. This is an
unreliable way to check for combine, because combine does advertise support for
DirMove, despite not always being able to do it.

This change fixes the issue by checking the returned error in a manner similar
to sync.MoveDir and falling back to individual file moves (copy + delete)
depending on which error was returned.
2024-03-07 11:11:46 +00:00
nielash
252562d00a combine: fix CopyDirMetadata error on upstream root
Before this change, operations.CopyDirMetadata would fail with: `internal error:
expecting directory string from combine root '' to have SetMetadata method:
optional feature not implemented` if the dst was the root directory of a combine
upstream. This is because combine was returning a *fs.Dir, which does not
satisfy the fs.SetMetadataer interface.

While it is true that combine cannot set metadata on the root of an upstream
(see also #7652), this should not be considered an error that causes sync to do
high-level retries, abort without doing deletes, etc.

This change addresses the issue by creating a new type of DirWrapper that is
allowed to fail silently, for exceptional cases such as this where certain
special directories have more limited abilities than what the Fs usually
supports.

It is possible that other similar wrapping backends (Union?) may need this same
fix.
2024-03-07 11:09:07 +00:00
nielash
6a72cfd6e1 operations: fix typo in log messages
I assume this must be a typo as %T of dir would only ever print "string"
2024-03-07 11:09:07 +00:00
nielash
354ea6fff3 docs: update to reflect dir modtime/metadata support 2024-03-07 11:09:07 +00:00
nielash
8c69455c37 sync: don't set dir modtimes if already set
Before this change, directory modtimes (and metadata) were always synced from
src to dst, even if already in sync (i.e. their modtimes already matched.) This
potentially required excessive API calls, made logs noisy, and was potentially
problematic for backends that create "versions" or otherwise log activity
updates when modtime/metadata is updated.

After this change, a new DirsEqual function is added to check whether dirs are
equal based on a number of factors such as ModifyWindow and sync flags in use.
If the dirs are equal, the modtime/metadata update is skipped.

For backends that require setDirModTimeAfter, the "after" sync is performed only
for dirs that could have been changed by the sync (i.e. dirs containing files
that were created/updated.)

Note that dir metadata (other than modtime) is not currently considered by
DirsEqual, consistent with how object metadata is synced (only when objects are
unequal for reasons other than metadata).

To sync dir modtimes and metadata unconditionally (the previous behavior), use
--ignore-times.
2024-03-07 09:57:11 +00:00
nielash
fd8faeb0e6 vfs: fix unicode normalization on macOS - fixes #7072
Before this change, the VFS layer did not properly handle unicode normalization,
which caused problems particularly for users of macOS. While attempts were made
to handle it with various `-o modules=iconv` combinations, this was an imperfect
solution, as no one combination allowed both NFC and NFD content to
simultaneously be both visible and editable via Finder.

After this change, the VFS supports `--no-unicode-normalization` (default `false`)
via the existing `--vfs-case-insensitive` logic, which is extended to apply to both
case insensitivity and unicode normalization form.

This change also adds an additional flag, `--vfs-block-norm-dupes`, to address a
probably rare but potentially possible scenario where a directory contains
multiple duplicate filenames after applying case and unicode normalization
settings. In such a scenario, this flag (disabled by default) hides the
duplicates. This comes with a performance tradeoff, as rclone will have to scan
the entire directory for duplicates when listing a directory. For this reason,
it is recommended to leave this disabled if not needed. However, macOS users may
wish to consider using it, as otherwise, if a remote directory contains both NFC
and NFD versions of the same filename, an odd situation will occur: both
versions of the file will be visible in the mount, and both will appear to be
editable, however, editing either version will actually result in only the NFD
version getting edited under the hood. `--vfs-block-norm-dupes` prevents this
confusion by detecting this scenario, hiding the duplicates, and logging an
error, similar to how this is handled in `rclone sync`.
2024-03-06 16:12:13 +00:00
Kyle Reynolds
dcdbad3554
bisync: clarify file operation directions in dry-run logs - fixes #7029
Before this change, NOTICE log messages during bisync dry runs were unclear as
to the direction of the skipped operation (Path1 to 2 vs. 2 to 1.) This change
adjusts the cmd/bisync/log.go indent function to be more expressive about
direction.
2024-03-06 09:26:53 -05:00
Nick Craig-Wood
effad3fe4b build: fix CVE-2024-24786 by upgrading google.golang.org/protobuf
See: https://pkg.go.dev/vuln/GO-2024-2611
2024-03-06 12:42:38 +00:00
Nick Craig-Wood
692af42858 operations: fix TestSetDirModTime for backends with SetDirModTime but not Metadata 2024-03-01 11:39:21 +00:00
Nick Craig-Wood
1693d7ad0f sftp: set DirModTimeUpdatesOnWrite to fix integration tests 2024-03-01 11:29:08 +00:00
Nick Craig-Wood
3bb9394ae5 operations: fix TestMkdirModTime test
This was failing on backends that didn't support metadata but did
support setting directory modtimes.
2024-03-01 11:18:24 +00:00
Nick Craig-Wood
be39e99918 sync: fix TestMoveEmptyDirectories so they work on backends which don't support DirModTimes 2024-03-01 10:56:48 +00:00
Nick Craig-Wood
6e28edeb9a cache: fix crash in tests which assumed local could Purge 2024-02-29 17:55:36 +00:00
Nick Craig-Wood
d50572b108 operations: add operations/hashsum to the rc as rclone hashsum equivalent
Fixes #7569
2024-02-29 16:21:42 +00:00
Nick Craig-Wood
0b8689dc28 rc: Add GetFsNamedFileOK to get an fs which could also be a file 2024-02-29 16:21:42 +00:00
Nick Craig-Wood
5994fcfed8 fs/cache: add PutErr to add an fs.Fs with an fs.ErrorIsFile error to the cache 2024-02-29 16:21:41 +00:00
Nick Craig-Wood
e3f6f68885 lib/cache: add PutErr to put a value with an error into the cache 2024-02-29 16:21:41 +00:00
Nick Craig-Wood
6ff1b6c505 local: delete backend implementation of Purge to speed up and make stats
In this commit (2014 for v1.02) Purge was implemented for the local
backend:

1527e64ee7 local: Implement Purger interface

This appeared to be implemented just to make a Purge and doesn't
appear to do anything useful.

It is in fact significatly worse than the rclone fallback purge since
it doesn't operate in parallel or update stats.

This patch removes the Purge routine for a consequent speed up and
showing of stats.

See: https://forum.rclone.org/t/progress-flag-for-rclone-purge/44416
2024-02-29 15:04:51 +00:00
Nick Craig-Wood
4a049c12fe copyurl: add troubleshooting section to the docs
See: https://forum.rclone.org/t/copyurl-fails-with-stream-error-wget-and-curl-works/44382/2
2024-02-29 14:58:12 +00:00