github/rclone
1
Fork 0
mirror of https://github.com/rclone/rclone synced 2024-12-23 14:23:44 +01:00
Code Releases Activity

docs: update documentation for --fast-list adding info about ListR

Browse Source
This commit is contained in:
albertony 2023-09-23 18:07:50 +02:00
parent 29baa5888f
commit 195ad98311
1 changed files with 39 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

62
docs/content/docs.md
View File

@ -2121,34 +2121,50 @@ there were IO errors`.
### --fast-list ###
When doing anything which involves a directory listing (e.g. `sync`,
`copy`, `ls` - in fact nearly every command), rclone normally lists a
directory and processes it before using more directory lists to
process any subdirectories. This can be parallelised and works very
quickly using the least amount of memory.
`copy`, `ls` - in fact nearly every command), rclone has different
strategies to choose from.
However, some remotes have a way of listing all files beneath a
directory in one (or a small number) of transactions. These tend to
be the bucket-based remotes (e.g. S3, B2, GCS, Swift).
The basic strategy is to list one directory and processes it before using
more directory lists to process any subdirectories. This is a mandatory
backend feature, called `List`, which means it is supported by all backends.
This strategy uses small amount of memory, and because it can be parallelised
it is fast for operations involving processing of the list results.
If you use the `--fast-list` flag then rclone will use this method for
listing directories. This will have the following consequences for
the listing:
Some backends provide the support for an alternative strategy, where all
files beneath a directory can be listed in one (or a small number) of
transactions. Rclone supports this alternative strategy through an optional
backend feature called [`ListR`](/overview/#listr). You can see in the storage
system overview documentation's [optional features](/overview/#optional-features)
section which backends it is enabled for (these tend to be the bucket-based
ones, e.g. S3, B2, GCS, Swift). This strategy requires fewer transactions
for highly recursive operations, which is important on backends where this
is charged or heavily rate limited. It may be faster (due to fewer transactions)
or slower (because it can't be parallelized) depending on different parameters,
and may require more memory if rclone has to keep the whole listing in memory.
* It **will** use fewer transactions (important if you pay for them)
* It **will** use more memory. Rclone has to load the whole listing into memory.
* It *may* be faster because it uses fewer transactions
* It *may* be slower because it can't be parallelized
Which listing strategy rclone picks for a given operation is complicated, but
in general it tries to choose the best possible. It will prefer `ListR` in
situations where it doesn't need to store the listed files in memory, e.g.
for unlimited recursive `ls` command variants. In other situations it will
prefer `List`, e.g. for `sync` and `copy`, where it needs to keep the listed
files in memory, and is performing operations on them where parallelization
may be a huge advantage.
rclone should always give identical results with and without
`--fast-list`.
Rclone is not able to take all relevant parameters into account for deciding
the best strategy, and therefore allows you to influence the choice in two ways:
You can stop rclone from using `ListR` by disabling the feature, using the
[--disable](#disable-feature-feature) option (`--disable ListR`), or you can
allow rclone to use `ListR` where it would normally choose not to do so due to
higher memory usage, using the `--fast-list` option. Rclone should always
produce identical results either way. Using `--disable ListR` or `--fast-list`
on a remote which doesn't support `ListR` does nothing, rclone will just ignore
it.
If you pay for transactions and can fit your entire sync listing into
memory then `--fast-list` is recommended. If you have a very big sync
to do then don't use `--fast-list` otherwise you will run out of
memory.
If you use `--fast-list` on a remote which doesn't support it, then
rclone will just ignore it.
A rule of thumb is that if you pay for transactions and can fit your entire
sync listing into memory, then `--fast-list` is recommended. If you have a
very big sync to do, then don't use `--fast-list`, otherwise you will run out
of memory. Run some tests and compare before you decide, and if in doubt then
just leave the default, let rclone decide, i.e. not use `--fast-list`.
### --timeout=TIME ###