1
mirror of https://github.com/rclone/rclone synced 2024-12-22 13:03:02 +01:00
Go to file
nielash 9b4b3033da fs/cache: fix parent not getting pinned when remote is a file
Before this change, when cache.GetFn was called on a file rather than a
directory, two cache entries would be added (the file + its parent) but only one
of them would get pinned if the caller then called Pin(f). This left the other
one exposed to expiration if the ci.FsCacheExpireDuration was reached. This was
problematic because both entries point to the same Fs, and if one entry expires
while the other is pinned, the Shutdown method gets erroneously called on an Fs
that is still in use.

An example of the problem showed up in the Hasher backend, which uses the
Shutdown method to stop the bolt db used to store hashes. If a command was run
on a Hasher file (ex. `rclone md5sum --download hasher:somelargefile.zip`) and
hashing the file took longer than the --fs-cache-expire-duration (5m by default), the
bolt db was stopped before the hashing operation completed, resulting in an
error.

This change fixes the issue by ensuring that:
1. only one entry is added to the cache (the file's parent, not the file).
2. future lookups correctly find the entry regardless of whether they are called
	with the parent name or one of its children.
3. fs.ErrorIsFile is returned when (and only when) fsString points to a file
	(preserving the fix from 8d5bc7f28b).

Note that f.Root() should always point to the parent dir as of c69eb84573
2024-09-28 13:49:56 +01:00
.github build: fix docker release build - fixes #8062 2024-09-12 17:57:53 +01:00
backend gcs: add access token auth with --gcs-access-token 2024-09-27 17:37:07 +01:00
bin Makefile: Prevent doc recipe from creating dir named '$HOME' 2024-09-24 10:38:25 +01:00
cmd build: replace "golang.org/x/exp/slices" with "slices" now go1.21 is required 2024-09-25 16:03:43 +01:00
cmdtest cmd: change exit code from 1 to 2 for syntax and usage errors 2024-09-20 18:51:08 +02:00
contrib docker volume plugin: add missing fuse3 dependency #6844 2023-03-15 15:57:53 +00:00
docs gcs: add access token auth with --gcs-access-token 2024-09-27 17:37:07 +01:00
fs fs/cache: fix parent not getting pinned when remote is a file 2024-09-28 13:49:56 +01:00
fstest test_all: add ignoretests parameter for skipping certain tests 2024-09-25 16:03:43 +01:00
graphics Version v1.66.0 2024-03-10 11:22:43 +00:00
lib fs/cache: fix parent not getting pinned when remote is a file 2024-09-28 13:49:56 +01:00
librclone docs/librclone: the newer and recommended ucrt64 subsystem of msys2 can now be used for building on windows 2024-01-01 21:56:45 +01:00
vfs rc: add vfs/queue-set-expiry to adjust expiry of items in the VFS queue 2024-09-06 17:33:35 +01:00
.gitattributes Add go mod and sum to gitattributes for consistent line endings 2024-04-13 11:16:42 +02:00
.gitignore docs: tidy .gitignore for docs 2024-06-15 13:08:20 +01:00
.golangci.yml build: enable custom linting rules with ruleguard via gocritic 2024-09-06 17:04:18 +01:00
CONTRIBUTING.md docs: add note to CONTRIBUTING that the overview needs editing in 2 places 2024-09-25 17:56:33 +01:00
COPYING Initial commit - some small parts working 2012-11-18 17:32:31 +00:00
Dockerfile build: fix docker build on arm/v6 2024-01-03 17:43:23 +00:00
go.mod build: replace "golang.org/x/exp/slices" with "slices" now go1.21 is required 2024-09-25 16:03:43 +01:00
go.sum build: enable custom linting rules with ruleguard via gocritic 2024-09-06 17:04:18 +01:00
MAINTAINERS.md chore: add childish-sambino as filescom maintainer 2024-08-15 22:00:26 +01:00
Makefile Makefile: Fail when doc recipes create dir named '$HOME' 2024-09-24 10:38:25 +01:00
MANUAL.html Version v1.68.0 2024-09-08 16:21:56 +01:00
MANUAL.md Version v1.68.0 2024-09-08 16:21:56 +01:00
MANUAL.txt Version v1.68.0 2024-09-08 16:21:56 +01:00
notes.txt docs: spelling: etc. 2020-10-28 18:16:23 +00:00
rclone.1 Version v1.68.0 2024-09-08 16:21:56 +01:00
rclone.go lib: add plugin support 2019-09-29 11:05:10 +01:00
README.md docs: add missing sftp providers to README and main docs page - fixes #8038 2024-08-28 07:25:49 +01:00
RELEASE.md docs: correct notes on docker manual build 2024-09-13 15:52:51 +01:00
VERSION Start v1.69.0-DEV development 2024-09-08 17:22:19 +01:00


rclone logo rclone logo

Website | Documentation | Download | Contributing | Changelog | Installation | Forum

Build Status Go Report Card GoDoc Docker Pulls

Rclone

Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.

Storage providers

  • 1Fichier 📄
  • Akamai Netstorage 📄
  • Alibaba Cloud (Aliyun) Object Storage System (OSS) 📄
  • Amazon S3 📄
  • ArvanCloud Object Storage (AOS) 📄
  • Backblaze B2 📄
  • Box 📄
  • Ceph 📄
  • China Mobile Ecloud Elastic Object Storage (EOS) 📄
  • Cloudflare R2 📄
  • Citrix ShareFile 📄
  • DigitalOcean Spaces 📄
  • Digi Storage 📄
  • Dreamhost 📄
  • Dropbox 📄
  • Enterprise File Fabric 📄
  • Fastmail Files 📄
  • Files.com 📄
  • FTP 📄
  • GoFile 📄
  • Google Cloud Storage 📄
  • Google Drive 📄
  • Google Photos 📄
  • HDFS (Hadoop Distributed Filesystem) 📄
  • Hetzner Storage Box 📄
  • HiDrive 📄
  • HTTP 📄
  • Huawei Cloud Object Storage Service(OBS) 📄
  • ImageKit 📄
  • Internet Archive 📄
  • Jottacloud 📄
  • IBM COS S3 📄
  • IONOS Cloud 📄
  • Koofr 📄
  • Leviia Object Storage 📄
  • Liara Object Storage 📄
  • Linkbox 📄
  • Linode Object Storage 📄
  • Magalu Object Storage 📄
  • Mail.ru Cloud 📄
  • Memset Memstore 📄
  • Mega 📄
  • Memory 📄
  • Microsoft Azure Blob Storage 📄
  • Microsoft Azure Files Storage 📄
  • Microsoft OneDrive 📄
  • Minio 📄
  • Nextcloud 📄
  • OVH 📄
  • Blomp Cloud Storage 📄
  • OpenDrive 📄
  • OpenStack Swift 📄
  • Oracle Cloud Storage 📄
  • Oracle Object Storage 📄
  • ownCloud 📄
  • pCloud 📄
  • Petabox 📄
  • PikPak 📄
  • Pixeldrain 📄
  • premiumize.me 📄
  • put.io 📄
  • Proton Drive 📄
  • QingStor 📄
  • Qiniu Cloud Object Storage (Kodo) 📄
  • Quatrix 📄
  • Rackspace Cloud Files 📄
  • RackCorp Object Storage 📄
  • rsync.net 📄
  • Scaleway 📄
  • Seafile 📄
  • SeaweedFS 📄
  • SFTP 📄
  • SMB / CIFS 📄
  • StackPath 📄
  • Storj 📄
  • SugarSync 📄
  • Synology C2 Object Storage 📄
  • Tencent Cloud Object Storage (COS) 📄
  • Uloz.to 📄
  • Wasabi 📄
  • WebDAV 📄
  • Yandex Disk 📄
  • Zoho WorkDrive 📄
  • The local filesystem 📄

Please see the full list of all storage providers and their features

Virtual storage providers

These backends adapt or modify other storage providers

  • Alias: rename existing remotes 📄
  • Cache: cache remotes (DEPRECATED) 📄
  • Chunker: split large files 📄
  • Combine: combine multiple remotes into a directory tree 📄
  • Compress: compress files 📄
  • Crypt: encrypt files 📄
  • Hasher: hash files 📄
  • Union: join multiple remotes to work together 📄

Features

  • MD5/SHA-1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Bisync (two way) to keep two directories in sync bidirectionally
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g. two different cloud accounts
  • Optional large file chunking (Chunker)
  • Optional transparent compression (Compress)
  • Optional encryption (Crypt)
  • Optional FUSE mount (rclone mount)
  • Multi-threaded downloads to local disk
  • Can serve local or remote files over HTTP/WebDAV/FTP/SFTP/DLNA

Installation & documentation

Please see the rclone website for:

Downloads

License

This is free software under the terms of the MIT license (check the COPYING file included in this package).