Commit Graph

6438 Commits

Author SHA1 Message Date
Nick Craig-Wood c917d2d5b4 s3: fix --s3-versions when copying a single object
Before this change, if --s3-versions was enabled, then copying a
single object from a subdirectory would fail.

This was due to an incorrect comparison in the NewFs code.

This fixes the change and introduces a new unit tests.
2022-09-05 18:56:11 +01:00
Nick Craig-Wood 3cb7734eac config: move locking to fix fatal error: concurrent map read and map write
Before this change we assumed that github.com/Unknwon/goconfig was
threadsafe as documented.

However it turns out it is not threadsafe and looking at the code it
appears that making it threadsafe might be quite hard.

So this change increases the lock coverage in configfile to cover the
goconfig uses also.

Fixes #6378
2022-09-05 12:11:06 +01:00
Nick Craig-Wood d08ed7d1e9 ftp: add notes on how to avoid deadlocks with concurrency - fixes #6370 2022-09-05 12:11:06 +01:00
Nick Craig-Wood f279e4ab01 Add Josh Soref to contributors 2022-09-05 12:10:59 +01:00
albertony 35349657cd docs/sftp: document use of chunk_size option in sftp remote paired with serve sftp
Related to 0008cb4934
2022-08-31 00:04:04 +02:00
Josh Soref ce3b65e6dc all: fix spelling across the project
* abcdefghijklmnopqrstuvwxyz
* accounting
* additional
* allowed
* almost
* already
* appropriately
* arise
* bandwidth
* behave
* bidirectional
* brackets
* cached
* characters
* cloud
* committing
* concatenating
* configured
* constructs
* current
* cutoff
* deferred
* different
* directory
* disposition
* dropbox
* either way
* error
* excess
* experiments
* explicitly
* externally
* files
* github
* gzipped
* hierarchies
* huffman
* hyphen
* implicitly
* independent
* insensitive
* integrity
* libraries
* literally
* metadata
* mimics
* missing
* modification
* multipart
* multiple
* nightmare
* nonexistent
* number
* obscure
* ourselves
* overridden
* potatoes
* preexisting
* priority
* received
* remote
* replacement
* represents
* reproducibility
* response
* satisfies
* sensitive
* separately
* separator
* specifying
* string
* successful
* synchronization
* syncing
* šenfeld
* take
* temporarily
* testcontents
* that
* the
* themselves
* throttling
* timeout
* transaction
* transferred
* unnecessary
* using
* webbrowser
* which
* with
* workspace

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-08-30 11:16:26 +02:00
albertony 0008cb4934 docs: document that serve sftp uses chunk size 32 KiB
See #6404
2022-08-30 10:57:25 +02:00
albertony 2ea5b4f0b8 Add YFdyh000 to contributors 2022-08-30 10:26:03 +02:00
YFdyh000 b5818454f7
onedrive: cleanup brand name 2022-08-30 10:23:29 +02:00
albertony 555def2da7 build: add package comments to silence revive linter 2022-08-28 13:43:51 +02:00
albertony 02b7613104 docs/jottacloud: improve description of the standard authentication 2022-08-28 10:31:39 +02:00
albertony b342c6cf9c docs/ftp: improve documentation of anonymous ftp - fixes #5650 2022-08-28 10:22:29 +02:00
albertony 8a6857c295 Add Simon Bos to contributors 2022-08-28 10:19:39 +02:00
albertony 21fd13f10d Add Ryan Morey to contributors 2022-08-28 10:18:33 +02:00
albertony 5cc7797f9e Add anonion to contributors 2022-08-28 10:18:14 +02:00
albertony 8bf2d6b6c8 Add João Henrique Franco to contributors 2022-08-28 10:16:37 +02:00
João Henrique Franco 85eb9776bd
crypt: fix typo in comment
strign -> string
2022-08-22 10:43:54 +02:00
anonion 47539ec0e6
docs: fix minor typo in onedrive docs 2022-08-21 22:09:33 +02:00
Ryan Morey 58b327a9f6
docs: fix typo in filter pattern example 2022-08-18 21:14:26 +02:00
Simon Bos 1107da7247 dlna: specify SSDP interface names from command line 2022-08-13 12:06:03 +01:00
Nick Craig-Wood 8d1fff9a82 local: obey file filters in listing to fix errors on excluded files
Fixes #6376
2022-08-11 12:23:06 +01:00
Nick Craig-Wood 2c5923ab1a filter: make sure we check --files-from when looking for a single file 2022-08-11 12:20:17 +01:00
Nick Craig-Wood 1ad22b8881 gcs: add --gcs-endpoint flag and config parameter
See: https://forum.rclone.org/t/how-to-modify-google-cloud-storage-endpoint-uri/32342
2022-08-09 17:33:21 +01:00
Nick Craig-Wood 0501773db1 azureblob,b2,s3: fix chunksize calculations producing too many parts
Before this fix, the chunksize calculator was using the previous size
of the object, not the new size of the object to calculate the chunk
sizes.

This meant that uploading a replacement object which needed a new
chunk size would fail, using too many parts.

This fix fixes the calculator to take the size explicitly.
2022-08-09 12:57:38 +01:00
Nick Craig-Wood cb8842941b Add Mark Trolley to contributors 2022-08-09 12:57:38 +01:00
Mark Trolley 5439a2c5c6 docs: fix script installation command on downloads page
Script installation instructions in `downloads.md` differ from those in
`install.md` and fail on MacOS.
2022-08-09 11:58:04 +01:00
Nick Craig-Wood d347ac0154 local: disable xattr support if the filesystems indicates it is not supported
Before this change, if rclone was run with `-M` on a filesystem
without xattr support, it would error out.

This patch makes rclone detect the not supported errors and disable
xattrs from then on. It prints one ERROR level message about this.

See: https://forum.rclone.org/t/metadata-update-local-s3/32277/7
2022-08-09 09:27:56 +01:00
Nick Craig-Wood 9f33eb2e65 Changelog updates from Version 1.59.1 2022-08-08 19:01:11 +01:00
Nick Craig-Wood fe801b8fef Add Joram Schrijver to contributors 2022-08-08 19:01:11 +01:00
albertony 6b158f33a3 serve sftp: document legacy code for checksum detection
See #6351
2022-08-06 20:46:38 +02:00
Joram Schrijver 5a6d233924 dlna: fix SOAP action header parsing - fixes #6354
Changes in github.com/anacrolix/dms changed upnp.ServiceURN to include a
namespace identifier. This identifier was previously hardcoded, but is
now parsed out of the URN. The old SOAP action header parsing logic was
duplicated in rclone and did not handle this field. Resulting responses
included a URN with an empty namespace identifier, breaking clients.
2022-08-06 17:23:37 +01:00
Nick Craig-Wood df513ca90a build: update dependencies 2022-08-05 17:43:53 +01:00
Nick Craig-Wood 49bb640bae accounting: fix panic in core/stats-reset with unknown group - fixes #6327
This also adds tests for the rc commands for stats groups
2022-08-05 17:30:55 +01:00
Nick Craig-Wood 98fd00a655 serve sftp: fix checksum detection - Fixes #6351
Before this change, rclone serve sftp operating with a new rclone
after the md5sum/sha1sum detection was reworked to just run a plain
`md5sum`/`sha1sum` command in

3ea82032e7 sftp: support md5/sha1 with rsync.net #3254

Failed to signal to the remote that md5sum/sha1sum wasn't supported as
in

71e172a139 serve/sftp: support empty "md5sum" and "sha1sum" commands

We unconditionally return good hashes even if the remote being served
doesn't support the hash type in question.

This fix checks the hash type is supported and returns an error

    MD5 hash not supported

When the backend is first contacted this will cause the sftp backend
to detect that the hash type isn't available.

Unfortunately this may have cached the wrong state so editing or
remaking the config may be necessary to fix it.
2022-08-05 17:16:23 +01:00
Nick Craig-Wood 16039b350d fs: fix parsing of times and durations of the form "YYYY-MM-DD HH:MM:SS"
Before this fix, the parsing code gave an error like this

    parsing "2022-08-02 07:00:00" as fs.Time failed: expected newline

This was due to the Scan call failing to read all the data.

This patch fixes that, and redoes the tests
2022-08-05 16:45:23 +01:00
Nick Craig-Wood ebe86c6cec s3: add --s3-decompress flag to download gzip-encoded files
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download.

If --s3-decompress is set, this change erases the length and hash on
compressed objects so they can be downloaded successfully, at the cost
of not being able to check the length or the hash of the downloaded
object.

If --s3-decompress is not set the compressed files will be downloaded
as-is providing compressed objects with intact size and hash
information.

See #2658
2022-08-05 16:45:23 +01:00
Nick Craig-Wood 1f5e7ce598 lib/readers: add GzipReader 2022-08-05 16:45:23 +01:00
Nick Craig-Wood 4b981100db s3: refactor to use generated code instead of reflection to copy structs 2022-08-05 16:45:23 +01:00
Nick Craig-Wood 4344a3e2ea s3: implement --s3-version-at flag - Fixes #1776 2022-08-05 16:45:23 +01:00
Nick Craig-Wood 1542a979f9 s3: refactor f.list() to take an options struct as it had too many parameters 2022-08-05 16:45:23 +01:00
Nick Craig-Wood 81d242473a s3: implement Purge to purge versions and `backend cleanup-hidden` 2022-08-05 16:45:23 +01:00
Nick Craig-Wood 0ae171416f s3: implement --s3-versions flag - See #1776 2022-08-05 16:45:23 +01:00
Nick Craig-Wood a59fa2977d s3: factor different listing versions into separate objects 2022-08-05 16:42:30 +01:00
Nick Craig-Wood 7243918069 s3: implement backend versioning command to get/set bucket versioning 2022-08-05 16:42:30 +01:00
Nick Craig-Wood fa49971d49 docs: move time/duration option docs to the main docs 2022-08-05 16:42:30 +01:00
Nick Craig-Wood 77e3512714 fstests: Make InternalTestFiles so the internal tests know the current state 2022-08-05 16:42:30 +01:00
Nick Craig-Wood 337b43e7e4 fstests: make ReadObject publically accessible 2022-08-05 16:42:30 +01:00
Nick Craig-Wood 6fd9e3d717 build: reformat comments to pass go1.19 vet
See: https://go.dev/doc/go1.19#go-doc
2022-08-05 16:35:41 +01:00
Nick Craig-Wood 876f791ecd Revert "build: lint with go1.18 until golangci-lint is updated"
This reverts commit 2170376d1b.
2022-08-05 16:04:54 +01:00
Nick Craig-Wood 918bd6d3c3 dropox: fix ChangeNotify was unable to decrypt errors
Before this fix, the dropbox backend wasn't decoding the file names
received in changenotify events into rclone standard format.

This meant that changenotify events for filenames which had encoded
characters were failing to be decrypted properly if wrapped in crypt.

See: https://forum.rclone.org/t/rclone-vfs-cache-says-file-name-too-long/31535
2022-08-04 10:26:25 +01:00