This simplifies the implementation of remotes. The only required
interface is now `List` which is a simple one level directory list.
Optionally remotes may implement `ListR` if they have an efficient way
of doing a recursive list.
* Add options to Put, PutUnchecked and Update for all Fses
* Use these to create HashOption
* Implement this in local
* Pass the option in fs.Copy
This has the effect that we only calculate hashes we need to in the
local Fs which speeds up transfers significantly.
Fix issue #1315 where filenames calculated with a base distance of zero
(ie the characters add up to 0(mod 256) aren't de-obfuscated on reading.
This was due to overloading of "0" to also mean "invalid UTF8; no rotation",
so we remove that double meaning
This is a simple "rotate" of the filename, with each file having a rot
distance based on the filename. We store the distance at the beginning
of the filename. So a file called "go" would become "37.KS".
This is not a strong encryption of filenames, but it should stop automated
scanning tools from picking up on filename patterns. As such it's an
intermediate between "off" and "standard". The advantage is that it
allows for longer path segment names.
We use the nameKey as an additional input to calculate the obfuscation
distance. This should mean that two different passwords will result
in two different keys
The obfuscation rotation works by splitting the ranges up and handle cases
0-9
A-Za-z
0xA0-0xFF
and anything greater in blocks of 256
This happened when the underlying reader returned io.ErrUnexpectedEOF.
The error handling for the call to io.ReadFull failed to take this
into account.
io.ErrUnexpectedEOF is reasonably common when SSL connections go wrong
when communicating with ACD, so it manifested itself as transfers from
non-encrypted ACD to encrypted ACD getting stuck.
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.
Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects. However when one Fs object wraps another it
is very difficult for this scheme to work accurately. rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.
This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method. The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.
As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name. Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.
Fixes#616
Streams which truncated early with an EOF message would return a
"Failed to authenticate decrypted block" error. While technically
correct, this isn't a helpful error message as it masks the underlying
problem. This changes it to return "unexpected EOF" instead.
The rest of rclone knows it should retry such errors.
The corruption was caused when the file was read to the end thus
setting io.EOF and returning the buffers to the pool. Seek reset the
EOF and carried on using the buffers that had been returned to the
pool thus causing corruption when other goroutines claimed the buffers
simultaneously.
Fix by resetting the buffer pointers to nil when released and claiming
new ones when seek resets EOF. Also added locking for Read and Seek
which shouldn't be run concurrently.
This was caused by failing to reset the internal buffer on seek so old
data was read first before the new data.
The unit tests didn't detect this because they were reading to the end
of the file to check integrity and thus emptying the internal buffer.
Both code and unit tests were fixed up.
