github/rclone
1
Fork 0
mirror of https://github.com/rclone/rclone synced 2024-12-26 18:23:45 +01:00
Code Releases Activity
5,150 Commits 182 Branches 114 Tags 310 MiB
84c785bc36
Commit Graph

1063 Commits

Author SHA1 Message Date
Nick Craig-Wood
bbe791a886 swift: update github.com/ncw/swift to v2.0.0 
The update to v2 of the swift library introduces a context parameter
to each function. This required a lot of mostly mechanical changes
adding context parameters.

See: https://github.com/ncw/swift/issues/159
See: https://github.com/ncw/swift/issues/161
 2021-02-03 20:23:37 +00:00
Nick Craig-Wood
1545ace8f2 build: remove go1.13 build constraints 2021-02-03 17:46:08 +00:00
Nick Craig-Wood
bcac8fdc83 Use http.NewRequestWithContext where possible after go1.13 minimum version 2021-02-03 17:41:27 +00:00
Nick Craig-Wood
9710ded60f b2: automatically raise upload cutoff to avoid spurious error 
Before this change, if --b2-chunk-size was raised above 200M then this
error would be produced:

    b2: upload cutoff: 200M is less than chunk size 1G

This change automatically reaises --b2-upload-cutoff to be the value
of --b2-chunk-size if it is below it, which stops this error being
generated.

Fixes #4475
 2021-02-03 16:29:32 +00:00
Nick Craig-Wood
8b41dfa50a s3: add --s3-no-head parameter to minimise transactions on upload 
See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
 2021-02-02 10:07:48 +00:00
Nick Craig-Wood
f66928a846 drive: fix copyid command with a bare filename: can't use empty string as a path 
Before this change, running

    rclone backend copyid drive: ID file.txt

Failed with the error

    command "copyid" failed: failed copying "ID" "file.txt": can't use empty string as a path

This fixes the problem.
 2021-02-01 15:17:25 +00:00
Nick Craig-Wood
3b1122c888 azureblob: require go1.14+ to compile due to SDK changes 2021-01-30 18:01:12 +00:00
Nick Craig-Wood
189ef5f257 azureblob: fix memory usage by upgrading the SDK and implementing a TransferManager 
In the Azure SDK there was a bug which caused excessive memory use
when doing repeated transfers:

https://github.com/Azure/azure-storage-blob-go/issues/233

This patch updates the SDK to v0.13.0 which allowed us to implement a
custom TransferManager which integrates with rclone's memory pool.

This fixes the excessive memory consumption.

See: https://forum.rclone.org/t/ask-for-settings-recommendation-for-azureblob/21505/
 2021-01-30 17:26:59 +00:00
NyaMisty
15da53696e
onedrive: add support for china region operated by 21vianet #4963 (#4963) 
fixes #3804
obsoletes #3973
obsoletes #4072
 2021-01-29 23:04:21 +03:00
Pau Rodriguez-Estivill
f0bf9cfda1 drive: add xdg office icons to xdg desktop files 2021-01-28 17:12:48 +00:00
Nguyễn Hữu Luân
671dd047f7
swift: ensure partially uploaded large files are uploaded unless --swift-leave-parts-on-error 
This makes sure that partially uploaded large files are removed
unless the `--swift-leave-parts-on-error` flag is supplied.

- refactor swift.go
- add unit test for swift with chunk
- add unit test for large object with fail case
- add "-" to white list char during encode.
 2021-01-28 17:09:41 +00:00
Nicolas Rueff
f5af761466
gphotos: new flag --gphotos-include-archived - fixes #4728 
New flag --gphotos-include-archived to download and view archived media when needed.
 2021-01-28 16:51:31 +00:00
Nick Craig-Wood
e6a9f005d6 sftp: implement --sftp-use-fstat 
See: https://forum.rclone.org/t/sftp-fails-to-sync-to-local-failed-to-copy-file-does-not-exist/21759
 2021-01-28 16:07:26 +00:00
Yury Stankevich
b569dc11a0 hdfs: support kerberos authentication #42 2021-01-27 18:16:58 +00:00
Riccardo Iaconelli
df4e6079f1 local: new flag --local-zero-size-links to fix sync on some virtual filesystems 
Assume the Stat size of links is zero (and read them instead)

On some virtual filesystems (such ash LucidLink), reading a link size via a
Stat call always returns 0.
However, on unix it reads as the length of the text in the link. This may
cause errors like this when syncing:

    Failed to copy: corrupted on transfer: sizes differ 0 vs 13

Setting this flag causes rclone to read the link and use that as the size of
the link instead of 0 which in most cases fixes the problem.

Fixes #4950

Signed-off-by: Riccardo Iaconelli <riccardo@kde.org>
 2021-01-27 18:13:16 +00:00
Louis Koo
cdaea62932
s3: fix copy multipart with v2 auth failing with 'SignatureDoesNotMatch' 
Signed-off-by: zhuc <zhucan.k8s@gmail.com>
 2021-01-27 14:43:02 +00:00
Nick Craig-Wood
78afe01d15 filefabric: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
4eac88babf premiumizeme: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
b4217fabd3 opendrive: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
3e188495f5 sugarsync: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
Nick Craig-Wood
acb9e17eb3 box: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
lluuaapp
35b2ca642c b2: fixed possible crash when accessing Backblaze b2 remote 2021-01-25 17:48:40 +00:00
Nick Craig-Wood
a774f6bfdb qingstor: fix rclone cleanup 
This patch changes to using the default page limit for listing
unfinished multpart uploads rather than 1000. 1000 is the maximum
specified in the docs, but setting anything larger than 200 gives an
error.
 2021-01-21 17:35:31 +00:00
Nick Craig-Wood
d7cd35e2ca qingstor: fix error propagation in CleanUp 
Before this change errors cleaning multiple buckets were passing silently
 2021-01-21 17:35:31 +00:00
buengese
eb090d3544 compress: check type assertion in SetTier - fixes #4941 2021-01-20 22:59:14 +00:00
Nick Craig-Wood
0be69018b8 drive: log that emptying the trash can take some time - fixes #4915 2021-01-19 18:09:36 +00:00
Nick Craig-Wood
9b9ab5f3e8 gcs: Fix Entry doesn't belong in directory "" (same as directory) - ignoring 
This change allows directory markers to be non-zero in size.

See: https://forum.rclone.org/t/public-gcs-bucket-and-entry-doesnt-belong-in-directory-same-as-directory/21753/
 2021-01-19 16:50:37 +00:00
Nick Craig-Wood
072464cbdb gcs: fix anonymous client to use rclone's HTTP client 2021-01-19 16:50:37 +00:00
buengese
45b57822d5 compress: improve testing 2021-01-18 21:42:58 +01:00
buengese
d8984cd37f compress: correctly handle wrapping of remotes without PutStream 
Also fixes ObjectInfo wrapping for Hash and Size - fixes #4928
 2021-01-18 21:42:58 +01:00
Patrik Nordlén
80e63af470
jottacloud: Add support for Telia Cloud (#4930) 2021-01-17 02:38:57 +01:00
Nick Craig-Wood
cef51d58ac jottacloud: fix token refresh failed: is not a regular file error 
Before this change the jottacloud token renewer would run and give the
error:

    Token refresh failed: is not a regular file

This is because the refresh runs on the root and it isn't a file.

This was fixed by ignoring that specific error.

See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173
 2021-01-12 17:09:44 +00:00
Nick Craig-Wood
e0b5a13a13 jottacloud: fix token renewer to fix long uploads 
See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173
 2021-01-11 16:44:11 +00:00
Ivan Andreev
35a4de2030 chunker: fix case-insensitive NewObject, test metadata detection #4902 
- fix test case FsNewObjectCaseInsensitive (PR #4830)
- continue PR #4917, add comments in metadata detection code
- add warning about metadata detection in user documentation
- change metadata size limits, make room for future development
- hide critical chunker parameters from command line
 2021-01-10 22:29:24 +03:00
Ivan Andreev
847625822f chunker: improve detection of incompatible metadata #4917 
Before this patch chunker required that there is at least one
data chunk to start checking for a composite object.

Now if chunker finds at least one potential temporary or control
chunk, it marks found files as a suspected composite object.
When later rclone tries a concrete operation on the object,
it performs postponed metadata read and decides: is this a native
composite object, incompatible metadata or just garbage.
 2021-01-10 21:55:15 +03:00
Nick Craig-Wood
3877df4e62 s3: update help for --s3-no-check-bucket #4913 2021-01-10 17:54:19 +00:00
Denis Neuling
ec73d2fb9a azure-blob-storage: utilize streaming capabilities - #1614 2021-01-10 17:02:42 +00:00
kice
ef2bfb9718
onedrive: Support addressing site by server-relative URL (#4761) 2021-01-09 03:26:42 +08:00
Alex Chen
78a76b0d29
onedrive: remove % and # from the set of encoded characters (#4909) 
onedrive: remove % and # from the set of encoded characters

This fixes #4700, fixes #4184, fixes #2920.
 2021-01-08 12:07:17 +00:00
Nick Craig-Wood
e775328523 ftp,sftp: Make --tpslimit apply - fixes #4906 2021-01-08 10:29:57 +00:00
Nick Craig-Wood
d58fdb10db onedrive: enhance link creation with expiry, scope, type and password 
This change makes the --expire flag in `rclone link` work.

It also adds the new flags

    --onedrive-link-type
    --onedrive-link-scope
    --onedrive-link-password

See: https://forum.rclone.org/t/create-share-link-within-the-organization-only/21498
 2021-01-08 09:22:50 +00:00
Yury Stankevich
71edc75ca6 HDFS (Hadoop Distributed File System) implementation - #42 
This includes an HDFS docker image to use with the integration tests.

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
 2021-01-07 09:48:51 +00:00
Alex Chen
c66b901320
onedrive: (business only) workaround to replace existing file on server-side copy (#4904) 2021-01-06 10:50:37 +08:00
Cnly
00bf40a8ef onedrive: fix server-side copy completely disabled on OneDrive for Business 
This fixes a little problem in PR #4903, which is a fix for #4342
 2021-01-06 02:57:51 +08:00
Alex Chen
b594cb9430
onedrive: fall back to normal copy if server-side copy unavailable (#4903) 
Fixes #4342 by:
* Disabling server-side copy if either drive isn't OneDrive Personal
* Falling back to normal copy if server-side copy fails
 2021-01-05 21:26:00 +08:00
Kerry Su
add7a35e55 b2: docs for download_url with private buckets 
The current authentication scheme works without creating
a public download endpoint for a private bucket as in the B2 official blog.
On the contrary, if the existing authorization header gets duplicated
in the Cloudflare Workers script, one might receive 401 Unauthorized errors.
 2021-01-02 11:33:48 +00:00
Nick Craig-Wood
cb97c2b0d3 azureblob: fix crash on startup 
This was introduced by accidental code deletion in

08b9ede217 azureblob: add support for managed identities
 2020-12-31 18:39:09 +00:00
buengese
66c3f2f31f new backend: zoho workdrive - fixes #4533 2020-12-30 17:56:08 +00:00
Nick Craig-Wood
a854cb9617 webdav: add "Depth: 0" to GET requests to fix bitrix 
See: https://forum.rclone.org/t/bitrix24-de-remote-support/21112/
 2020-12-30 10:14:50 +00:00
Nick Craig-Wood
ba51409c3c sftp: implement keyboard interactive authentication - fixes #4177 
Some ssh servers are set up with keyboard interactive authentication
which previously the sftp backkend was ignoring.
 2020-12-29 19:48:09 +00:00