github/rclone
1
Fork 0
mirror of https://github.com/rclone/rclone synced 2024-11-17 17:30:37 +01:00
Code Releases Activity
4,917 Commits 179 Branches 114 Tags 282 MiB
c3ac9319f4
Commit Graph

102 Commits

Author SHA1 Message Date
Nick Craig-Wood
76ee3060d1 s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C 
Before this change, small objects uploaded with SSE-AWS/SSE-C would
not have MD5 sums.

This change adds metadata for these objects in the same way that the
metadata is stored for multipart uploaded objects.

See: #1824 #2827
 2020-11-25 12:28:02 +00:00
Nick Craig-Wood
a06f4c2514 s3: fix hashes on small files with aws:kms and sse-c 
If rclone is configured for server side encryption - either aws:kms or
sse-c (but not sse-s3) then don't treat the ETags returned on objects
as MD5 hashes.

This fixes being able to upload small files.

Fixes #1824
 2020-11-25 12:28:02 +00:00
Josh Soref
d09488b829 docs: update: add Tencent 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
0a6196716c docs: style: avoid double-nesting parens 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
a15f50254a docs: grammar: if, then 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
5d4f77a022 docs: grammar: Oxford comma 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
a089de0964 docs: grammar: uncountable: links 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
3068ae8447 docs: grammar: count agreement: files 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
67ff153b0c docs: grammar: article: a-file 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d4f38d45a5 docs: spelling: high-speed 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Christopher Stewart
f3cf6fcdd7
s3: fix spelling mistake 
Fix spelling mistake "patific" => "pacific"
 2020-09-18 12:03:13 +01:00
wjielai
22937e8982
docs: add Tencent COS to s3 provider list - fixes #4468 
* add Tencent COS to s3 provider list.

Co-authored-by: wjielai <wjielai@tencent.com>
 2020-09-08 16:34:25 +01:00
Nick Craig-Wood
510ac341e1 Version v1.53.0 2020-09-02 17:00:18 +01:00
Nick Craig-Wood
d5f4c74697 s3: implement cleanup and backend command to list & remove multipart uploads 
This implements `rclone cleanup` to remove multipart uploads over 24
hours old. It also implements the backend command
`list-multipart-uploads` to see which ones are available and `cleanup`
to delete them with a configurable expiry interval.

See #4302
 2020-07-28 11:37:46 +01:00
Vincent Feltz
f4d7e41f24 s3: add Scaleway provider - fixes #4338 2020-06-13 11:55:37 +01:00
Nick Craig-Wood
844b903595 docs: promote the use of -i/--interactive and "rclone sync -i" everywhere #1574 2020-06-10 12:33:53 +01:00
Nick Craig-Wood
e62c032184 docs: remove manually set dates and use git dates instead 2020-06-01 13:07:46 +01:00
Nick Craig-Wood
6882aeff97 Version v1.52.0 2020-05-27 17:31:10 +01:00
Nick Craig-Wood
41e12114a8 docs: fix validation errors 2020-05-27 17:31:09 +01:00
Nick Craig-Wood
74b8cbfb84 docs: set unsafe HTML parsing to false and fix raw HTML insertion 
This means that markdown files can't contain <thing> any more.
 2020-05-27 17:31:09 +01:00
Nick Craig-Wood
80dab10ec9 docs: Fix generated docs 
In distributed docs
- Make img tags absolute
- Add donate link in docs
- Fix provider table
 2020-05-27 17:31:08 +01:00
Martin Michlmayr
fb169a8b54
doc: fix typos throughout docs 2020-05-19 12:02:44 +01:00
Nick Craig-Wood
43daecd89b Version v1.51.0 2020-02-01 10:40:01 +00:00
Dave Koston
9f99c20232 s3: Add StackPath Object Storage Support 2020-01-31 16:05:44 +00:00
Tennix
15d19131bd s3: use aws web identity role provider 2020-01-05 19:49:31 +00:00
Thomas Kriechbaumer
584e705c0c s3: introduce list_chunk option for bucket listing 
The S3 ListObject API returns paginated bucket listings, with
"MaxKeys" items for each GET call.

The default value is 1000 entries, but for buckets with millions of
objects it might make sense to request more elements per request, if
the backend supports it. This commit adds a "list_chunk" option for
the user to specify a lower or higher value.

This commit does not add safe guards around this value - if a user
decides to request a too large list, it might result in connection
timeouts (on the server or client).

In AWS S3, there is a fixed limit of 1000, some other services might
have one too.  In Ceph, this can be configured in RadosGW.
 2020-01-02 12:15:01 +00:00
Outvi V
db1c7f9ca8 s3: Add new region Asia Patific (Hong Kong) 2020-01-02 11:10:48 +00:00
Marco Molteni
0c9a8cf776 doc: add Scaleway to the S3 table of contents 
Hello, documentation for Scaleway was already there, but the TOC was missing it.
 2019-11-14 21:49:43 +00:00
Aleksandar Jankovic
6b55b8b133 s3: add option for multipart failiure behaviour 
This is needed for resuming uploads across different sessions.
 2019-10-02 16:49:16 +01:00
Fabian Möller
33f129fbbc s3: use lib/encoder 
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
 2019-09-30 22:00:24 +01:00
Nick Craig-Wood
73e010aff9 docs: make the config walkthroughs consistent for each backend 2019-08-26 10:47:17 +01:00
Nick Craig-Wood
93810a739d docs: update fontawesome free to 5.10.2 and fixup broken images 2019-08-25 11:09:19 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Russell Davis
6a3c3d9b89 Update docs on S3 policy to include ListAllMyBuckets permission 
This permission is required for `rclone lsd`.
 2019-07-02 07:56:54 +01:00
Matti Niemenmaa
a6dca4c13f s3: Add INTELLIGENT_TIERING storage class 
For Intelligent-Tiering:
https://aws.amazon.com/s3/storage-classes/#Unknown_or_changing_access
 2019-07-01 18:17:48 +01:00
Nick Craig-Wood
245fed513a Version v1.48.0 2019-06-15 13:55:41 +01:00
Nick Craig-Wood
d533de0f5c docs: clarify that s3 can't access Glacier Vault 2019-06-15 09:56:17 +01:00
Philip Harvey
1a2fb52266 s3: make SetModTime work for GLACIER while syncing - Fixes #3224 
Before this change rclone would fail with

    Failed to set modification time: InvalidObjectState: Operation is not valid for the source object's storage class

when attempting to set the modification time of an object in GLACIER.

After this change rclone will re-upload the object as part of a sync if it needs to change the modification time.

See: https://forum.rclone.org/t/suspected-bug-in-s3-or-compatible-sync-logic-to-glacier/10187
 2019-06-03 15:28:19 +01:00
Animosity022
b88e50cc36 docs: Typo fixes with "a existing" 
Fixed a typo with "a existing" to "an existing"
 2019-05-11 16:49:48 +01:00
Nick Craig-Wood
a0d4fdb2fa Version v1.47.0 2019-04-13 11:01:58 +01:00
Manu
6e86526c9d s3: add support for "Glacier Deep Archive" storage class - fixes #3088 2019-04-11 10:21:41 +01:00
Ben Boeckel
3d424c6e08 docs: fix various typos 2019-03-28 15:12:51 +00:00
Nick Craig-Wood
27714e29c3 s3: note incompatibility with CEPH Jewel - fixes #3015 2019-03-06 11:50:37 +00:00
Nick Craig-Wood
eb85ecc9c4 Version v1.46 2019-02-09 10:42:57 +00:00
Rémy Léone
1e6cbaa355 s3: Add Scaleway to s3 documentation 2019-02-01 17:09:57 +00:00
Nick Craig-Wood
1318c6aec8 s3: Add Alibaba OSS to integration tests and fix storage classes 2019-01-12 20:41:47 +00:00
Nick Craig-Wood
ff0b8e10af s3: Support Alibaba Cloud (Aliyun) OSS 
The existing s3 backend passed all integration tests with OSS provided
`force_path_style = false`.

This makes sure that is so and adds documentation and configuration
for OSS.

Thanks to @luolibin for their work on the OSS backend which we ended
up not needing.

Fixes #1641
Fixes #1237
 2019-01-12 17:28:04 +00:00