From 96c2fdb445471e4c8c5ef0fba898e29e5dc2a6d8 Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Fri, 26 Jun 2020 18:28:33 +0100 Subject: [PATCH] vfs: update VFS help --- vfs/help.go | 150 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 102 insertions(+), 48 deletions(-) diff --git a/vfs/help.go b/vfs/help.go index 1ce66d8a2..4e8be3bb3 100644 --- a/vfs/help.go +++ b/vfs/help.go @@ -3,20 +3,39 @@ package vfs // Help contains text describing file and directory caching to add to // the command help. var Help = ` -### Directory Cache +### VFS - Virtual File System -Using the ` + "`--dir-cache-time`" + ` flag, you can set how long a +This command uses the VFS layer. This adapts the cloud storage objects +that rclone uses into something which looks much more like a disk +filing system. + +Cloud storage objects have lots of properties which aren't like disk +files - you can't extend them or write to the middle of them, so the +VFS layer has to deal with that. Because there is no one right way of +doing this there are various options explained below. + +The VFS layer also implements a directory cache - this caches info +about files and directories (but not the data) in memory. + +### VFS Directory Cache + +Using the ` + "`--dir-cache-time`" + ` flag, you can control long a directory should be considered up to date and not refreshed from the -backend. Changes made locally in the mount may appear immediately or -invalidate the cache. However, changes done on the remote will only -be picked up once the cache expires if the backend configured does not -support polling for changes. If the backend supports polling, changes -will be picked up on within the polling interval. +backend. Changes made through the mount will appear immediately or +invalidate the cache. -Alternatively, you can send a ` + "`SIGHUP`" + ` signal to rclone for -it to flush all directory caches, regardless of how old they are. -Assuming only one rclone instance is running, you can reset the cache -like this: + --dir-cache-time duration Time to cache directory entries for. (default 5m0s) + --poll-interval duration Time to wait between polling for changes. + +However, changes made directoy on the cloud storage by the web +interface or a different copy of rclone will only be picked up once +the directory cache expires if the backend configured does not support +polling for changes. If the backend supports polling, changes will be +picked up on within the polling interval. + +You can send a ` + "`SIGHUP`" + ` signal to rclone for it to flush all +directory caches, regardless of how old they are. Assuming only one +rclone instance is running, you can reset the cache like this: kill -SIGHUP $(pidof rclone) @@ -29,40 +48,41 @@ Or individual files or directories: rclone rc vfs/forget file=path/to/file dir=path/to/dir -### File Buffering +### VFS File Buffering The ` + "`--buffer-size`" + ` flag determines the amount of memory, that will be used to buffer data in advance. -Each open file descriptor will try to keep the specified amount of -data in memory at all times. The buffered data is bound to one file -descriptor and won't be shared between multiple open file descriptors -of the same file. +Each open file will try to keep the specified amount of data in memory +at all times. The buffered data is bound to one open file and won't be +shared. + +This flag is a upper limit for the used memory per open file. The +buffer will only use memory for data that is downloaded but not not +yet read. If the buffer is empty, only a small amount of memory will +be used. -This flag is a upper limit for the used memory per file descriptor. -The buffer will only use memory for data that is downloaded but not -not yet read. If the buffer is empty, only a small amount of memory -will be used. The maximum memory used by rclone for buffering can be up to ` + "`--buffer-size * open files`" + `. -### File Caching +### VFS File Caching -These flags control the VFS file caching options. The VFS layer is -used by rclone mount to make a cloud storage system work more like a -normal file system. +These flags control the VFS file caching options. File caching is +necessary to make the VFS layer appear compatible with a normal file +system. It can be disabled at the cost of some compatibility. -You'll need to enable VFS caching if you want, for example, to read -and write simultaneously to a file. See below for more details. +For example you'll need to enable VFS caching if you want to read and +write simultaneously to a file. See below for more details. -Note that the VFS cache works in addition to the cache backend and you -may find that you need one or the other or both. +Note that the VFS cache is separate from the cache backend and you may +find that you need one or the other or both. --cache-dir string Directory rclone will use for caching. + --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) - --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") + --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) - --vfs-cache-max-size int Max total size of objects in the cache. (default off) + --vfs-write-back duration Time to writeback files after last use when using cache. (default 5s) If run with ` + "`-vv`" + ` rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but @@ -74,9 +94,10 @@ The higher the cache mode the more compatible rclone becomes at the cost of using disk space. Note that files are written back to the remote only when they are -closed so if rclone is quit or dies with open files then these won't -get written back to the remote. However they will still be in the on -disk cache. +closed and if they haven't been accessed for --vfs-write-back +second. If rclone is quit or dies with files that haven't been +uploaded, these will be uploaded next time rclone is run with the same +flags. If using --vfs-cache-max-size note that the cache may exceed this size for two reasons. Firstly because it is only checked every @@ -119,32 +140,65 @@ first. This mode should support all normal file system operations. -If an upload fails it will be retried up to --low-level-retries times. +If an upload fails it will be retried at exponentially increasing +intervals up to 1 minute. #### --vfs-cache-mode full -In this mode all reads and writes are buffered to and from disk. When -a file is opened for read it will be downloaded in its entirety first. +In this mode all reads and writes are buffered to and from disk. When +data is read from the remote this is buffered to disk as well. -This may be appropriate for your needs, or you may prefer to look at -the cache backend which does a much more sophisticated job of caching, -including caching directory hierarchies and chunks of files. +In this mode the files in the cache will be sparse files and rclone +will keep track of which bits of the files it has dowloaded. -In this mode, unlike the others, when a file is written to the disk, -it will be kept on the disk after it is written to the remote. It -will be purged on a schedule according to ` + "`--vfs-cache-max-age`" + `. +So if an application only reads the starts of each file, then rclone +will only buffer the start of the file. These files will appear to be +their full size in the cache, but they will be sparse files with only +the data that has been downloaded present in them. -This mode should support all normal file system operations. +This mode should support all normal file system operations and is +otherwise identical to --vfs-cache-mode writes. -If an upload or download fails it will be retried up to ---low-level-retries times. +### VFS Performance -### Case Sensitivity +These flags may be used to enable/disable features of the VFS for +performance or other reasons. + +In particular S3 and Swift benefit hugely from the --no-modtime flag +(or use --use-server-modtime for a slightly different effect) as each +read of the modification time takes a transaction. + + --no-checksum Don't compare checksums on up/download. + --no-modtime Don't read/write the modification time (can speed things up). + --no-seek Don't allow seeking in files. + --read-only Mount read-only. + +When rclone reads files from a remote it reads them in chunks. This +means that rather than requesting the whole file rclone reads the +chunk specified. This is advantageous because some cloud providers +account for reads being all the data requested, not all the data +delivered. + +Rclone will keep doubling the chunk size requested starting at +--vfs-read-chunk-size with a maximum of --vfs-read-chunk-size-limit +unless it is set to "off" in which case there will be no limit. + + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) + --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default "off") + +Sometimes rclone is delivered reads or writes out of order. Rather +than seeking rclone will wait a short time for the in sequence read or +write to come in. These flags only come into effect when not using an +on disk cach file. + + --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) + --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) + +### VFS Case Sensitivity Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file. -Windows is not like most other operating systems supported by rclone. File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. @@ -155,7 +209,7 @@ file systems case-sensitive but that is not the default The "--vfs-case-insensitive" mount flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the mounted -file system as is. If the flag is "true" (or appears without a value on +file system as-is. If the flag is "true" (or appears without a value on command line), rclone may perform a "fixup" as explained below. The user may specify a file name to open/delete/rename/etc with a case