github/rclone
1
Fork 0
mirror of https://github.com/rclone/rclone synced 2024-12-24 15:43:45 +01:00
Code Releases Activity

docs: update documentation for --fast-list adding info about ListR

Browse Source
This commit is contained in:
albertony 2023-09-23 18:07:50 +02:00
parent 29baa5888f
commit 195ad98311
1 changed files with 39 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

62
docs/content/docs.md
View File

@ -2121,34 +2121,50 @@ there were IO errors`.
### --fast-list ### ### --fast-list ###
When doing anything which involves a directory listing (e.g. `sync`, When doing anything which involves a directory listing (e.g. `sync`,
`copy`, `ls` - in fact nearly every command), rclone normally lists a `copy`, `ls` - in fact nearly every command), rclone has different
directory and processes it before using more directory lists to strategies to choose from.
process any subdirectories. This can be parallelised and works very
quickly using the least amount of memory.
However, some remotes have a way of listing all files beneath a The basic strategy is to list one directory and processes it before using
directory in one (or a small number) of transactions. These tend to more directory lists to process any subdirectories. This is a mandatory
be the bucket-based remotes (e.g. S3, B2, GCS, Swift). backend feature, called `List`, which means it is supported by all backends.
This strategy uses small amount of memory, and because it can be parallelised
it is fast for operations involving processing of the list results.
If you use the `--fast-list` flag then rclone will use this method for Some backends provide the support for an alternative strategy, where all
listing directories. This will have the following consequences for files beneath a directory can be listed in one (or a small number) of
the listing: transactions. Rclone supports this alternative strategy through an optional
backend feature called [`ListR`](/overview/#listr). You can see in the storage
system overview documentation's [optional features](/overview/#optional-features)
section which backends it is enabled for (these tend to be the bucket-based
ones, e.g. S3, B2, GCS, Swift). This strategy requires fewer transactions
for highly recursive operations, which is important on backends where this
is charged or heavily rate limited. It may be faster (due to fewer transactions)
or slower (because it can't be parallelized) depending on different parameters,
and may require more memory if rclone has to keep the whole listing in memory.
* It **will** use fewer transactions (important if you pay for them) Which listing strategy rclone picks for a given operation is complicated, but
* It **will** use more memory. Rclone has to load the whole listing into memory. in general it tries to choose the best possible. It will prefer `ListR` in
* It *may* be faster because it uses fewer transactions situations where it doesn't need to store the listed files in memory, e.g.
* It *may* be slower because it can't be parallelized for unlimited recursive `ls` command variants. In other situations it will
prefer `List`, e.g. for `sync` and `copy`, where it needs to keep the listed
files in memory, and is performing operations on them where parallelization
may be a huge advantage.
rclone should always give identical results with and without Rclone is not able to take all relevant parameters into account for deciding
`--fast-list`. the best strategy, and therefore allows you to influence the choice in two ways:
You can stop rclone from using `ListR` by disabling the feature, using the
[--disable](#disable-feature-feature) option (`--disable ListR`), or you can
allow rclone to use `ListR` where it would normally choose not to do so due to
higher memory usage, using the `--fast-list` option. Rclone should always
produce identical results either way. Using `--disable ListR` or `--fast-list`
on a remote which doesn't support `ListR` does nothing, rclone will just ignore
it.
If you pay for transactions and can fit your entire sync listing into A rule of thumb is that if you pay for transactions and can fit your entire
memory then `--fast-list` is recommended. If you have a very big sync sync listing into memory, then `--fast-list` is recommended. If you have a
to do then don't use `--fast-list` otherwise you will run out of very big sync to do, then don't use `--fast-list`, otherwise you will run out
memory. of memory. Run some tests and compare before you decide, and if in doubt then
just leave the default, let rclone decide, i.e. not use `--fast-list`.
If you use `--fast-list` on a remote which doesn't support it, then
rclone will just ignore it.
### --timeout=TIME ### ### --timeout=TIME ###