1
mirror of https://github.com/rclone/rclone synced 2024-11-30 09:10:05 +01:00
Commit Graph

108 Commits

Author SHA1 Message Date
Nick Craig-Wood
b5f563fb0f vfs: Ignore Truncate if called with no readers and already the correct size
This fixes FreeBSD which seems to call SetAttr with a size even on
read only files.

This is probably a bug in the FreeBSD FUSE implementation as it
happens with mount and cmount.

See: https://forum.rclone.org/t/freebsd-question/8662/12
2019-02-12 17:27:04 +00:00
Nick Craig-Wood
8ec55ae20b Fix broken flag type tests
Introduced in fc1bf5f931
2019-02-07 16:42:26 +00:00
Nick Craig-Wood
fc1bf5f931 Make flags show up with their proper names, eg SizeSuffix rather than int 2019-02-07 11:57:26 +00:00
Nick Craig-Wood
f27c2d9760 vfs: make cache tests more reliable 2019-02-02 16:26:55 +00:00
Nick Craig-Wood
a43ed567ee vfs: implement --vfs-cache-max-size to limit the total size of the cache 2019-02-02 12:30:10 +00:00
Nick Craig-Wood
53a8b5a275 vfs: Implement renaming of directories for backends without DirMove #2539
Previously to this change, backends without the optional interface
DirMove could not rename directories.

This change uses the new operations.DirMove call to implement renaming
directories which will fall back to Move/Copy as necessary.
2019-01-27 21:26:56 +00:00
Nick Craig-Wood
bd11d3cb62 vfs: Fix panic on rename with --dry-run set - fixes #2911 2019-01-14 12:07:25 +00:00
Nick Craig-Wood
71e98ea584 vfs: fix renaming/deleting open files with cache mode "writes" under Windows
Before this change, renaming and deleting of open files (which can
easily happen due to the asynchronous nature of file systems) would
produce an error, for example saving files with Firefox.

After this change we open files with the flags necessary for open
files to be renamed or deleted.

Fixes #2730
2019-01-11 10:26:34 +00:00
Nick Craig-Wood
7ee7bc87ae vfs: fix tests after --dir-perms changes
This was introduced in 554ee0d963
2019-01-09 09:49:34 +00:00
Nick Craig-Wood
554ee0d963 vfs: add --dir-perms and --file-perms flags - fixes #2897
This allows files to be shown with the execute bit which allows
binaries to be run under Windows and Linux.
2019-01-08 17:29:38 +00:00
Nick Craig-Wood
1f6a1cd26d vfs: add test_vfs code for hunting for deadlocks 2018-12-26 09:08:27 +00:00
Nick Craig-Wood
2cfe2354df vfs: fix deadlock between RWFileHandle.close and File.Remove - fixes #2857
Before this change we took the locks file.mu and file.muRW in an
inconsistent order - after the change we always take them in the same
order to fix the deadlock.
2018-12-26 09:08:27 +00:00
Nick Craig-Wood
13387c0838 vfs: fix deadlock on concurrent operations on a directory - fixes #2811
Before this fix there were two paths where concurrent use of a
directory could take the file lock then directory lock and the other
would take the locks in the reverse order.

Fix this by narrowing the locking windows so the file lock and
directory lock don't overlap.
2018-12-26 09:08:27 +00:00
Nick Craig-Wood
b972dcb0ae rc: implement options/blocks,get,set and register options 2018-11-03 11:32:00 +00:00
Nick Craig-Wood
87e1efa997 mount, vfs: Remove EXPERIMENTAL tags
rclone mount and the --vfs-cache-mode has been tested extensively by
users now so removing the EXPERIMENTAL tag is appropriate.
2018-10-06 11:47:46 +01:00
Fabian Möller
1d14972e41 vfs: reduce directory cache cleared by poll-interval
Reduce the number of nodes purged from the dir-cache when ForgetPath is
called. This is done by only forgetting the cache of the received path
and invalidating the parent folder cache by resetting *Dir.read.

The parent will read the listing on the next access and reuse the
dir-cache of entries in *Dir.items.
2018-10-02 10:21:14 +01:00
Nick Craig-Wood
59e14c25df vfs: enable rename for nearly all remotes using server side Move or Copy
Before this change remotes without server side Move (eg swift, s3,
gcs) would not be able to rename files.

After it means nearly all remotes will be able to rename files on
rclone mount with the notable exceptions of b2 and yandex.

This changes checks to see if the remote can do Move or Copy then
calls `operations.Move` to do the actual move.  This will do a server
side Move or Copy but won't download and re-upload the file.

It also checks to see if the destination exists first which avoids
conflicts or duplicates.

Fixes #1965
Fixes #2569
2018-09-29 14:56:20 +01:00
Fabian Möller
d9a3b26e47 vfs: add vfs/poll-interval rc command
This command can be used to query the current status of the
poll-interval option and also update the value.
2018-09-25 14:01:13 +02:00
Fabian Möller
1eec59e091 fs: update ChangeNotifier interface
This introduces a channel to the ChangeNotify function, which can be
used to update the poll-interval and cleanly exit the polling function.
2018-09-25 14:01:13 +02:00
Nick Craig-Wood
ef3526b3b8 vfs: fix race condition detected by serve ftp tests 2018-09-17 08:50:34 +01:00
Fabian Möller
6dc5aa7454 docs: clearify buffer-size is per transfer/filehandle 2018-08-17 18:11:40 +01:00
Fabian Möller
552eb8e06b vfs: try to seek buffer on read only files 2018-08-17 18:10:28 +01:00
Fabian Möller
2e80d4c18e vfs: update vfs/refresh rc command documentation 2018-08-07 09:31:12 +01:00
Fabian Möller
6349147af4 vfs: add non recursive mode to vfs/refresh rc command 2018-08-07 09:31:12 +01:00
Fabian Möller
782972088d vfs: add the vfs/refresh rc command
vfs/refresh will walk the directory tree for the given paths and
freshen the directory cache. It will use the fast-list capability
of the remote when enabled.
2018-08-07 09:31:12 +01:00
Oleg Kovalov
06c9f76cd2 all: fix go-critic linter suggestions 2018-08-06 21:14:03 +01:00
Fabian Möller
7983b6bdca vfs: enable vfs-read-chunk-size by default 2018-07-29 18:17:05 +01:00
Stefan
2a806a8d8b
mount: only print "File.rename error" if there actually is an error - see #2130 (#2322) 2018-05-29 19:19:17 +02:00
Stefan
67e9ef4547
mount: delay rename if file has open writers instead of failing outright - fixes #2130 (#2249) 2018-05-24 20:45:11 +02:00
Nick Craig-Wood
cb5bd47e61 build: fix errors spotted by ineffassign linter
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
2018-05-05 17:32:41 +01:00
Fabian Möller
98bf65c43b vfs: fix ChangeNotify for new or changed folders
Fixes #2251
2018-05-05 12:54:03 +01:00
Fabian Möller
bd3ad1ac3e vfs: add option to read source files in chunks 2018-05-05 12:49:42 +01:00
Mateusz Pabian
7c5f87842c vfs: filter files . and .. from readDir output - fixes #2135 2018-04-25 16:09:07 +01:00
Nick Craig-Wood
2b855751fc vfs,mount,cmount: use About to return the correct disk total/used/free
Disks total, used, free now shows correctly for mount and cmount (eg
`df` for Unix or in the Windows explorer).
2018-04-18 18:27:34 +01:00
Nick Craig-Wood
2b7957cc74 vfs: Only make the VFS cache if --vfs-cache-mode > Off
This stops the cache cleaner running unnecessarily and saves
resources.

This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
2018-04-16 17:06:41 +01:00
Nick Craig-Wood
e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood
2db0c4dd95 vfs: add remote control for directory cache flushing 2018-03-17 10:34:02 +00:00
Nick Craig-Wood
c19e675ca6 vfs: unify locking for RWFileHandle.openPending,.close and File.Delete #2141
Without this fix the cached file can be removed as the file is being
uploaded or downloaded.  This can cause the directory listings to
become inconsistent (this issue) or data loss (if a retry was needed
in the Copy).

Remove file needs to be excluded from running at the same time as both
openPending and close so it makes sense to unify the locking between
all 3.
2018-03-15 20:49:07 +00:00
Nick Craig-Wood
a2336ad774 vfs: fix deadlock in mount tests
This was caused by this sequence of calls

1> file.Release
1> file.close  -> takes the file lock
2> vfs.waitforWriters
2> dir.walk -> takes the dir lock
1> file.setObject
1> dir.addObject -> attempts to take the dir lock - BLOCKS
2> file.activeWriters -> tries to take file lock - BLOCKS - DEADLOCK

The fix is to make activeWriters not take the file lock and use atomic
operations to read the number of writers instead.
2018-03-09 23:15:38 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it (#2094)
* fs: rename DirChangeNotify to ChangeNotify

* cache: switch to ChangeNotify

* ChangeNotify: keep order of notifications
2018-03-08 22:03:34 +02:00
Nick Craig-Wood
0175332987 vfs: fix applying modtime for an open Write Handle
The symptom of this was that the time set when the file was open was
lost.  This was causing one of the mount tests to fail too.
2018-03-06 21:58:11 +00:00
Nick Craig-Wood
5795bd7db6 vfs: update cached copy if we know it has changed even if pending opens
This fixes a problem with Caddy serving corrupted files out of the VFS
cache when the file on the remote changed.
2018-03-04 11:20:22 +00:00
Nick Craig-Wood
9b011ce7e4 vfs: keep track of number of open RWHandles 2018-03-04 11:20:22 +00:00
Nick Craig-Wood
5e334eedd2 vfs: re-use the File objects when re-reading the directory
Make it so that d.items is never nil to simplify the code

This should help with inconsistent reads when the source object changes.
2018-03-04 11:20:22 +00:00
Nick Craig-Wood
7fb53a031c vfs: don't cache the object in read and read/write handles
This should help with inconsistent reads when the source object changes.
2018-03-04 11:20:22 +00:00
Nick Craig-Wood
b91bd32489 vfs: Fix TestWriteFileDoubleClose with --vfs-cache-mode >= writes
This was causing the file to be closed on Flush() instead of Release()
when the file was opened with O_TRUNC.
2018-02-26 21:26:32 +00:00
Nick Craig-Wood
c3d0f68923 vfs: fix truncation work-around on Windows
This no longer needs to deal with O_RDONLY and O_TRUNC since we
disallow this earlier.  This also fixes the code to just do it for
O_APPEND, not for everything.
2018-02-26 19:46:38 +00:00
Nick Craig-Wood
f57e92b9a5 vfs: fix creation of files when truncating #2083
As spotted by @B4dM4n
2018-02-26 19:37:58 +00:00
Nick Craig-Wood
baf9ee5cf7 vfs: update cached copy if we know it has changed before using it
Before this change we would have to wait for the --vfs-cache-max-age
to expire before getting an update.
2018-02-26 18:00:51 +00:00