github/rclone
1
Fork 0
mirror of https://github.com/rclone/rclone synced 2024-12-22 13:03:02 +01:00
Code Releases Activity
5,880 Commits 182 Branches 114 Tags 364 MiB
663b2d9c46
Commit Graph

298 Commits

Author SHA1 Message Date
Nick Craig-Wood
1f6a1cd26d vfs: add test_vfs code for hunting for deadlocks 2018-12-26 09:08:27 +00:00
Nick Craig-Wood
2cfe2354df vfs: fix deadlock between RWFileHandle.close and File.Remove - fixes #2857 
Before this change we took the locks file.mu and file.muRW in an
inconsistent order - after the change we always take them in the same
order to fix the deadlock.
 2018-12-26 09:08:27 +00:00
Nick Craig-Wood
13387c0838 vfs: fix deadlock on concurrent operations on a directory - fixes #2811 
Before this fix there were two paths where concurrent use of a
directory could take the file lock then directory lock and the other
would take the locks in the reverse order.

Fix this by narrowing the locking windows so the file lock and
directory lock don't overlap.
 2018-12-26 09:08:27 +00:00
Nick Craig-Wood
b972dcb0ae rc: implement options/blocks,get,set and register options 2018-11-03 11:32:00 +00:00
Nick Craig-Wood
87e1efa997 mount, vfs: Remove EXPERIMENTAL tags 
rclone mount and the --vfs-cache-mode has been tested extensively by
users now so removing the EXPERIMENTAL tag is appropriate.
 2018-10-06 11:47:46 +01:00
Fabian Möller
1d14972e41 vfs: reduce directory cache cleared by poll-interval 
Reduce the number of nodes purged from the dir-cache when ForgetPath is
called. This is done by only forgetting the cache of the received path
and invalidating the parent folder cache by resetting *Dir.read.

The parent will read the listing on the next access and reuse the
dir-cache of entries in *Dir.items.
 2018-10-02 10:21:14 +01:00
Nick Craig-Wood
59e14c25df vfs: enable rename for nearly all remotes using server side Move or Copy 
Before this change remotes without server side Move (eg swift, s3,
gcs) would not be able to rename files.

After it means nearly all remotes will be able to rename files on
rclone mount with the notable exceptions of b2 and yandex.

This changes checks to see if the remote can do Move or Copy then
calls `operations.Move` to do the actual move.  This will do a server
side Move or Copy but won't download and re-upload the file.

It also checks to see if the destination exists first which avoids
conflicts or duplicates.

Fixes #1965
Fixes #2569
 2018-09-29 14:56:20 +01:00
Fabian Möller
d9a3b26e47 vfs: add vfs/poll-interval rc command 
This command can be used to query the current status of the
poll-interval option and also update the value.
 2018-09-25 14:01:13 +02:00
Fabian Möller
1eec59e091 fs: update ChangeNotifier interface 
This introduces a channel to the ChangeNotify function, which can be
used to update the poll-interval and cleanly exit the polling function.
 2018-09-25 14:01:13 +02:00
Nick Craig-Wood
ef3526b3b8 vfs: fix race condition detected by serve ftp tests 2018-09-17 08:50:34 +01:00
Fabian Möller
6dc5aa7454 docs: clearify buffer-size is per transfer/filehandle 2018-08-17 18:11:40 +01:00
Fabian Möller
552eb8e06b vfs: try to seek buffer on read only files 2018-08-17 18:10:28 +01:00
Fabian Möller
2e80d4c18e vfs: update vfs/refresh rc command documentation 2018-08-07 09:31:12 +01:00
Fabian Möller
6349147af4 vfs: add non recursive mode to vfs/refresh rc command 2018-08-07 09:31:12 +01:00
Fabian Möller
782972088d vfs: add the vfs/refresh rc command 
vfs/refresh will walk the directory tree for the given paths and
freshen the directory cache. It will use the fast-list capability
of the remote when enabled.
 2018-08-07 09:31:12 +01:00
Oleg Kovalov
06c9f76cd2 all: fix go-critic linter suggestions 2018-08-06 21:14:03 +01:00
Fabian Möller
7983b6bdca vfs: enable vfs-read-chunk-size by default 2018-07-29 18:17:05 +01:00
Stefan
2a806a8d8b
mount: only print "File.rename error" if there actually is an error - see #2130 (#2322) 2018-05-29 19:19:17 +02:00
Stefan
67e9ef4547
mount: delay rename if file has open writers instead of failing outright - fixes #2130 (#2249) 2018-05-24 20:45:11 +02:00
Nick Craig-Wood
cb5bd47e61 build: fix errors spotted by ineffassign linter 
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
 2018-05-05 17:32:41 +01:00
Fabian Möller
98bf65c43b vfs: fix ChangeNotify for new or changed folders 
Fixes #2251
 2018-05-05 12:54:03 +01:00
Fabian Möller
bd3ad1ac3e vfs: add option to read source files in chunks 2018-05-05 12:49:42 +01:00
Mateusz Pabian
7c5f87842c vfs: filter files . and .. from readDir output - fixes #2135 2018-04-25 16:09:07 +01:00
Nick Craig-Wood
2b855751fc vfs,mount,cmount: use About to return the correct disk total/used/free 
Disks total, used, free now shows correctly for mount and cmount (eg
`df` for Unix or in the Windows explorer).
 2018-04-18 18:27:34 +01:00
Nick Craig-Wood
2b7957cc74 vfs: Only make the VFS cache if --vfs-cache-mode > Off 
This stops the cache cleaner running unnecessarily and saves
resources.

This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
 2018-04-16 17:06:41 +01:00
Nick Craig-Wood
e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood
2db0c4dd95 vfs: add remote control for directory cache flushing 2018-03-17 10:34:02 +00:00
Nick Craig-Wood
c19e675ca6 vfs: unify locking for RWFileHandle.openPending,.close and File.Delete #2141 
Without this fix the cached file can be removed as the file is being
uploaded or downloaded.  This can cause the directory listings to
become inconsistent (this issue) or data loss (if a retry was needed
in the Copy).

Remove file needs to be excluded from running at the same time as both
openPending and close so it makes sense to unify the locking between
all 3.
 2018-03-15 20:49:07 +00:00
Nick Craig-Wood
a2336ad774 vfs: fix deadlock in mount tests 
This was caused by this sequence of calls

1> file.Release
1> file.close  -> takes the file lock
2> vfs.waitforWriters
2> dir.walk -> takes the dir lock
1> file.setObject
1> dir.addObject -> attempts to take the dir lock - BLOCKS
2> file.activeWriters -> tries to take file lock - BLOCKS - DEADLOCK

The fix is to make activeWriters not take the file lock and use atomic
operations to read the number of writers instead.
 2018-03-09 23:15:38 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it (#2094) 
* fs: rename DirChangeNotify to ChangeNotify

* cache: switch to ChangeNotify

* ChangeNotify: keep order of notifications
 2018-03-08 22:03:34 +02:00
Nick Craig-Wood
0175332987 vfs: fix applying modtime for an open Write Handle 
The symptom of this was that the time set when the file was open was
lost.  This was causing one of the mount tests to fail too.
 2018-03-06 21:58:11 +00:00
Nick Craig-Wood
5795bd7db6 vfs: update cached copy if we know it has changed even if pending opens 
This fixes a problem with Caddy serving corrupted files out of the VFS
cache when the file on the remote changed.
 2018-03-04 11:20:22 +00:00
Nick Craig-Wood
9b011ce7e4 vfs: keep track of number of open RWHandles 2018-03-04 11:20:22 +00:00
Nick Craig-Wood
5e334eedd2 vfs: re-use the File objects when re-reading the directory 
Make it so that d.items is never nil to simplify the code

This should help with inconsistent reads when the source object changes.
 2018-03-04 11:20:22 +00:00
Nick Craig-Wood
7fb53a031c vfs: don't cache the object in read and read/write handles 
This should help with inconsistent reads when the source object changes.
 2018-03-04 11:20:22 +00:00
Nick Craig-Wood
b91bd32489 vfs: Fix TestWriteFileDoubleClose with --vfs-cache-mode >= writes 
This was causing the file to be closed on Flush() instead of Release()
when the file was opened with O_TRUNC.
 2018-02-26 21:26:32 +00:00
Nick Craig-Wood
c3d0f68923 vfs: fix truncation work-around on Windows 
This no longer needs to deal with O_RDONLY and O_TRUNC since we
disallow this earlier.  This also fixes the code to just do it for
O_APPEND, not for everything.
 2018-02-26 19:46:38 +00:00
Nick Craig-Wood
f57e92b9a5 vfs: fix creation of files when truncating #2083 
As spotted by @B4dM4n
 2018-02-26 19:37:58 +00:00
Nick Craig-Wood
baf9ee5cf7 vfs: update cached copy if we know it has changed before using it 
Before this change we would have to wait for the --vfs-cache-max-age
to expire before getting an update.
 2018-02-26 18:00:51 +00:00
Nick Craig-Wood
354f1ad722 vfs: Use operations.Copy instead of CopyFile for efficiency 2018-02-26 17:54:18 +00:00
Nick Craig-Wood
54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC 
Before this change Open("name", os.O_RDONLY|os.O_TRUNC) would have
truncated the file.  This is what Linux does, but is counterintuitive.
POSIX states this is undefined, so return an error in this case
instead.  This preserves the invariant O_RDONLY => file is not
changed.
 2018-02-26 17:04:27 +00:00
Nick Craig-Wood
3282fd26af vfs: clean path names before using them in the cache 
This avoids inconsistent cache behaviour on open("potato/")
close("potato").

The tests were also adjusted to make them more comprehensive.
 2018-02-26 16:59:14 +00:00
Nick Craig-Wood
88d830c7b7 vfs: create cache.opens and use it in place of cache.get to avoid potential race 2018-02-26 16:58:02 +00:00
Nick Craig-Wood
9252224d82 vfs: don't open the file when using a RW handle for a null Seek 
Background: cmd/mount/file.go Open() function does a Seek(0, 1) to see
if the file handle is seekable to set a FUSE hint.  Before this change
the file was downloaded before it needed to be which was inefficient
(and broke beta.rclone.org because HEAD requests caused downloads!).
 2018-02-22 17:28:21 +00:00
Fabian Möller
62e72801be
vfs: fix race between multiple RWFileHandle (#2052) 
Fixes #2034
 2018-02-18 14:12:26 +01:00
Nick Craig-Wood
442334ba61 vfs: disable cache cleaner if --vfs-cache-poll-interval=0 
And use this to disable the cleaner in the cache tests to make them
more reliable
 2018-02-16 14:12:46 +00:00
Nick Craig-Wood
8a9ed57951 vfs: fix another race in cache tests 2018-02-16 12:05:59 +00:00
Nick Craig-Wood
9b800d7184 vfs: fix race in cache tests 2018-02-15 21:34:37 +00:00
Nick Craig-Wood
cd7fd51119 vfs: fix docs - fixes #2067 2018-02-12 11:29:32 +00:00
Stefan Breunig
846bbef1e9 vfs: write 0 bytes when flushing unwritten handles to avoid race conditions in FUSE - fixes #1181 2018-02-11 17:59:13 +00:00
Nick Craig-Wood
04a0a7406b vfs: downgrade "poll-interval is not supported" message to Info 
...to save confusion as it isn't very important
 2018-02-09 07:57:50 +00:00
Nick Craig-Wood
295c3fabec vfs: fill and clean the cache immediately on startup 2018-02-02 12:19:53 +00:00
Nick Craig-Wood
3f8d286a75 vfs: fix cache cleaning on startup 
Previous to this fix the vfs cache wasn't being cleaned properly on
startup as the atimes of the existing files were being ignored.
 2018-02-02 12:06:42 +00:00
Nick Craig-Wood
44276db454 vfs: make -ve sized files appear as 0 size. #320 
This means that Google docs will no longer appear as huge files in
`rclone mount`.  They will not be downloadable, though sometimes
trying twice will work.
 2018-01-31 16:22:05 +00:00
Nick Craig-Wood
dbabb18b0c vfs: Make error messages more informative #2009 2018-01-25 10:33:00 +00:00
Jody Frankowski
5e73acd40a Clean up mount.go and vfs/help.go docs 
* Title cleanups
* Typos
* `rclone mount vs rclone sync/copy` update with `File Caching`
 2018-01-20 18:14:20 +00:00
Nick Craig-Wood
a80d8a21dc vfs: add flags parameter to Dir.Create 2018-01-19 13:18:40 +00:00
Nick Craig-Wood
517bdc719b vfs: make specialized file Open functions private 2018-01-19 11:46:01 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00
Nick Craig-Wood
29d34426bc vfs: fix deletion of in use directories #1860 
This was causing errors if the cache cleaner was called between the
Open and the pendingOpen of a RW file.

The fix was to move the cache open to the Open from the openPending.
 2017-12-15 15:42:49 +00:00
Nick Craig-Wood
38b85e94ea vfs: rename --cache-* options to --vfs-cache-* to save confusion 
..as the backend cache options are all called --cache-* too. Adjust
docs to point out what the vfs cache does vs the backend cache.
 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
c16ac697a9 vfs: keep track of directories in the cache also #1860 
This makes managing empty directories more reliable.
 2017-11-28 20:39:23 +00:00
Nick Craig-Wood
0978957a2e vfs: make sure all 96 combinations of flags for Open work 2017-11-28 20:39:23 +00:00
Nick Craig-Wood
d1b19f975d vfs: remove items from cache when deleted #1860 
Also fixes Error message when items have been deleted from the cache
(eg when Moved) when the cache reaper comes to delete them.
 2017-11-28 16:13:58 +00:00
Nick Craig-Wood
3fb4fe31d2 vfs: make sure write only handles never truncate files they shouldn't 2017-11-20 19:42:25 +00:00
Nick Craig-Wood
76b151984c vfs: cache the size of the object in the read handle 2017-11-20 17:57:13 +00:00
Nick Craig-Wood
60945d0a37 vfs: remove misleading comment 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
b4083b4371 vfs: rename Fsync to Sync and implement Sync on Node and Handle 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9738f8532b vfs: Add FlushDirCache method 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
a5b034a992 vfs: add WaitForWriters to wait until all writers have finished 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
321b6da7af vfs: don't remove file from writers until it is transferred 
This means that the list of active writers is up to date
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
1b22ee5b93 vfs: fix error handling in openPending so it returns the correct error 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eab55ce882 vfs: add open files to directories 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
c560017934 vfs: add Path method to Node and use it to stop reading nil DirEntry 
All DirEntry calls now have been checked for nil or converted to use Path.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
992647b157 vfs: Don't error a r/w file open without cache; delay error until Read called 
If we open a file for r/w without the cache we now always return a
handle and return an error if the file is ever read from.  This fixes
incompatibility with cmount under windows.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
dec21ccf63 vfs, cmount: make truncate work properly in the presence or otherwise of open files 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
e7f2935333 vfs: decode flags in Open/OpenFile for debug 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
8951875c21 vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
05a1e1532b vfs,mount,cmount,serve: Add documentation for vfs caching modes 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
7f20e1d7f3 vfs: add read write files and caching #711 
This adds new flags to mount, cmount, serve *

    --cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
    --cache-mode string              Cache mode off|minimal|writes|full (default "off")
    --cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)
 2017-11-20 12:36:50 +00:00
Nick Craig-Wood
a0cfa0929b vfs: remove un-needed (after introduction of rcat) createInfo struct 2017-11-20 12:00:32 +00:00
Nick Craig-Wood
3fb1e96988 vfs: factor Open logic from Dir.Create into vfs.OpenFile 2017-11-20 12:00:32 +00:00
Fabian Möller
5acb6f47e7 mountlib: log when poll-interval is ineffective 
Notify the user in case poll-interval is used on a unsupported remote
 2017-11-19 20:08:14 +00:00
Nick Craig-Wood
c31556c6d1 vfs: Make sure all public methods are locked in Read and Write Handle 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
2083ac6e2a vfs: add ECLOSED and tidy errors 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
22ee839d05 cmount,vfs: unify Read and Write handles and File and Dir where possible 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
5634659ea3 mount,vfs: unify Read and Write handles in preparation for ReadWrite handles 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
e18122e88b vfs: add tests and subsequent fixes 
* Tests for VFS layer
  * Small fixes found during testing
  * Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
  * Fix nil object bugs on File
 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
8184ec4b70 vfs: add EPERM to errors 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
190367d917 vfs: factor duplicated Open code into vfs from mount/cmount 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
a5dc62f6c1 vfs: Make file handles compatible with OS 
* Implement directory handles
  * Unify OpenFile
  * Add all the methods to match *os.File
  * Add StatParent and Rename methods to VFS
 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
3e0c91ba4b vfs: Move DefaultOpt to vfs and make some methods private 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
7e065440fb vfs: rename Lookup to Stat to be more in keeping with os 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
e8883e9fdb vfs: factor flags into vfsflags and remove global variables 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
1a8f824bad vfs: use os package errors where possible 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
c1aaff220d Factor new vfs module out of cmd/mountlib 
This is an OS style file system abstraction with directory caching
used in mount, cmount, serve webdav and serve http.
 2017-11-03 12:59:59 +00:00