github/rclone
1
Fork 0
mirror of https://github.com/rclone/rclone synced 2025-03-08 09:09:21 +01:00
Code Releases Activity

Break the fs package up into smaller parts.

Browse Source 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
This commit is contained in:
Nick Craig-Wood 2018-01-12 16:30:54 +00:00
parent 92624bbbf1
commit 11da2a6c9b
183 changed files with 5749 additions and 5063 deletions
CONTRIBUTING.mdMakefile
backend
amazonclouddrive
amazonclouddrive.go
azureblob
azureblob.go
b2
api
types.go
b2.goupload.go
box
box.go
cache
cache.gocache_internal_test.goobject.goplex.go
crypt
crypt.gocrypt_config_test.go
drive
drive.goupload.go
dropbox
dropbox.go
ftp
ftp.go
googlecloudstorage
googlecloudstorage.go
http
http.gohttp_internal_test.go
hubic
hubic.go
local
local.goread_device_unix.go
onedrive
onedrive.go
pcloud
pcloud.go
qingstor
qingstor.go
s3
s3.go
sftp
sftp.go
swift
swift.go
webdav
webdav.go
yandex
yandex.go
cmd
authorize
authorize.go
cachestats
cachestats.go
cat
cat.go
check
check.go
cleanup
cleanup.go
cmd.go
cmount
fs.go
config
config.go
copy
copy.go
copyto
copyto.go
cryptcheck
cryptcheck.go
cryptdecode
cryptdecode.go
dbhashsum
dbhashsum.go
dedupe
dedupe.go
delete
delete.go
info
info.go
listremotes
listremotes.go
ls
ls.go
lsd
lsd.go
lsf
lsf.golsf_test.go
lsjson
lsjson.go
lsl
lsl.go
md5sum
md5sum.go
memtest
memtest.go
mkdir
mkdir.go
mount
dir.gofile.gofs.gohandle.go
mountlib
mount.go
mounttest
fs.go
move
move.go
moveto
moveto.go
ncdu/scan
scan.go
obscure
obscure.go
purge
purge.go
rcat
rcat.go
rmdir
rmdir.go
rmdirs
rmdirs.go
serve
http
http.gohttp_test.go
webdav
webdav.go
sha1sum
sha1sum.go
size
size.go
sync
sync.go
touch
touch.go
tree
tree.go
fs
accounting
accounting.goaccounting_other.goaccounting_unix.go
asyncreader
asyncreader.goasyncreader_test.go
bwtimetable.gobwtimetable_test.goconfig.go
config
config.goconfig_other.goconfig_read_password.goconfig_read_password_unsupported.goconfig_test.goconfig_unix.go
configflags
configflags.go
flags
flags.go
obscure.goobscure_test.go
testdata
enc-invalid.confenc-short.conf
Show More

56
CONTRIBUTING.md

@ -116,6 +116,52 @@ then run in that directory
go run test_all.go
## Code Organisation ##
Rclone code is organised into a small number of top level directories
with modules beneath.
* backend - the rclone backends for interfacing to cloud providers -
* all - import this to load all the cloud providers
* ...providers
* bin - scripts for use while building or maintaining rclone
* cmd - the rclone commands
* all - import this to load all the commands
* ...commands
* docs - the documentation and website
* content - adjust these docs only - everything else is autogenerated
* fs - main rclone definitions - minimal amount of code
* accounting - bandwidth limiting and statistics
* asyncreader - an io.Reader which reads ahead
* config - manage the config file and flags
* driveletter - detect if a name is a drive letter
* filter - implements include/exclude filtering
* fserrors - rclone specific error handling
* fshttp - http handling for rclone
* fspath - path handling for rclone
* hash - defines rclones hash types and functions
* list - list a remote
* log - logging facilities
* march - iterates directories in lock step
* object - in memory Fs objects
* operations - primitives for sync, eg Copy, Move
* sync - sync directories
* walk - walk a directory
* fstest - provides integration test framework
* fstests - integration tests for the backends
* mockdir - mocks an fs.Directory
* mockobject - mocks an fs.Object
* test_all - Runs integration tests for everything
* graphics - the images used in the website etc
* lib - libraries used by the backend
* dircache - directory ID to name caching
* oauthutil - helpers for using oauth
* pacer - retries with backoff and paces operations
* readers - a selection of useful io.Readers
* rest - a thin abstraction over net/http for REST
* vendor - 3rd party code managed by the dep tool
* vfs - Virtual FileSystem layer for implementing rclone mount and similar
## Writing Documentation ##
If you are adding a new feature then please update the documentation.
@ -240,10 +286,10 @@ Research
Getting going
* Create `remote/remote.go` (copy this from a similar remote)
* Create `backend/remote/remote.go` (copy this from a similar remote)
* box is a good one to start from if you have a directory based remote
* b2 is a good one to start from if you have a bucket based remote
* Add your remote to the imports in `fs/all/all.go`
* Add your remote to the imports in `backend/all/all.go`
* HTTP based remotes are easiest to maintain if they use rclone's rest module, but if there is a really good go SDK then use that instead.
* Try to implement as many optional methods as possible as it makes the remote more usable.
@ -251,14 +297,14 @@ Unit tests
* Create a config entry called `TestRemote` for the unit tests to use
* Add your fs to the end of `fstest/fstests/gen_tests.go`
* generate `remote/remote_test.go` unit tests `cd fstest/fstests; go generate`
* generate `backend/remote/remote_test.go` unit tests `cd fstest/fstests; go generate`
* Make sure all tests pass with `go test -v`
Integration tests
* Add your fs to `fs/test_all.go`
* Add your fs to `fstest/test_all/test_all.go`
* Make sure integration tests pass with
* `cd fs`
* `cd fs/operations`
* `go test -v -remote TestRemote:`
* If you are making a bucket based remote, then check with this also
* `go test -v -remote TestRemote: -subdir`

3
Makefile

@ -32,8 +32,9 @@ version:
# Full suite of integration tests
test: rclone
go install github.com/ncw/fstest/test_all
-go test $(BUILDTAGS) $(GO_FILES) 2>&1 | tee test.log
-cd fs && go run $(BUILDTAGS) test_all.go 2>&1 | tee test_all.log
-test_all github.com/ncw/rclone/fs/operations github.com/ncw/rclone/fs/sync 2>&1 | tee fs/test_all.log
@echo "Written logs in test.log and fs/test_all.log"
# Quick test

35
backend/amazonclouddrive/amazonclouddrive.go

@ -24,6 +24,11 @@ import (
"github.com/ncw/go-acd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/dircache"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/ncw/rclone/lib/pacer"
@ -46,7 +51,7 @@ const (
var (
// Flags
tempLinkThreshold = fs.SizeSuffix(9 << 30) // Download files bigger than this via the tempLink
uploadWaitPerGB = fs.DurationP("acd-upload-wait-per-gb", "", 180*time.Second, "Additional time per GB to wait after a failed complete upload to see if it appears.")
uploadWaitPerGB = flags.DurationP("acd-upload-wait-per-gb", "", 180*time.Second, "Additional time per GB to wait after a failed complete upload to see if it appears.")
// Description of how to auth for this app
acdConfig = &oauth2.Config{
Scopes: []string{"clouddrive:read_all", "clouddrive:write"},
@ -73,20 +78,20 @@ func init() {
}
},
Options: []fs.Option{{
Name: fs.ConfigClientID,
Name: config.ConfigClientID,
Help: "Amazon Application Client Id - required.",
}, {
Name: fs.ConfigClientSecret,
Name: config.ConfigClientSecret,
Help: "Amazon Application Client Secret - required.",
}, {
Name: fs.ConfigAuthURL,
Name: config.ConfigAuthURL,
Help: "Auth server URL - leave blank to use Amazon's.",
}, {
Name: fs.ConfigTokenURL,
Name: config.ConfigTokenURL,
Help: "Token server url - leave blank to use Amazon's.",
}},
})
fs.VarP(&tempLinkThreshold, "acd-templink-threshold", "", "Files >= this size will be downloaded via their tempLink.")
flags.VarP(&tempLinkThreshold, "acd-templink-threshold", "", "Files >= this size will be downloaded via their tempLink.")
}
// Fs represents a remote acd server
@ -171,7 +176,7 @@ func (f *Fs) shouldRetry(resp *http.Response, err error) (bool, error) {
return true, err
}
}
return fs.ShouldRetry(err) || fs.ShouldRetryHTTP(resp, retryErrorCodes), err
return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}
// If query parameters contain X-Amz-Algorithm remove Authorization header
@ -193,7 +198,7 @@ func filterRequest(req *http.Request) {
// NewFs constructs an Fs from the path, container:path
func NewFs(name, root string) (fs.Fs, error) {
root = parsePath(root)
baseClient := fs.Config.Client()
baseClient := fshttp.NewClient(fs.Config)
if do, ok := baseClient.Transport.(interface {
SetRequestFilter(f func(req *http.Request))
}); ok {
@ -212,7 +217,7 @@ func NewFs(name, root string) (fs.Fs, error) {
root: root,
c: c,
pacer: pacer.New().SetMinSleep(minSleep).SetPacer(pacer.AmazonCloudDrivePacer),
noAuthClient: fs.Config.Client(),
noAuthClient: fshttp.NewClient(fs.Config),
}
f.features = (&fs.Features{
CaseInsensitive: true,
@ -472,7 +477,7 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
if iErr != nil {
return nil, iErr
}
if fs.IsRetryError(err) {
if fserrors.IsRetryError(err) {
fs.Debugf(f, "Directory listing error for %q: %v - low level retry %d/%d", dir, err, tries, maxTries)
continue
}
@ -875,8 +880,8 @@ func (f *Fs) Precision() time.Duration {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5)
}
// Copy src to this remote using server side copy operations.
@ -932,9 +937,9 @@ func (o *Object) Remote() string {
}
// Hash returns the Md5sum of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 {
return "", hash.ErrHashUnsupported
}
if o.info.ContentProperties != nil && o.info.ContentProperties.Md5 != nil {
return *o.info.ContentProperties.Md5, nil

38
backend/azureblob/azureblob.go

@ -11,7 +11,7 @@ import (
"encoding/binary"
"encoding/hex"
"fmt"
"hash"
gohash "hash"
"io"
"net/http"
"path"
@ -23,6 +23,12 @@ import (
"github.com/Azure/azure-sdk-for-go/storage"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/walk"
"github.com/ncw/rclone/lib/pacer"
"github.com/pkg/errors"
)
@ -66,8 +72,8 @@ func init() {
},
},
})
fs.VarP(&uploadCutoff, "azureblob-upload-cutoff", "", "Cutoff for switching to chunked upload")
fs.VarP(&chunkSize, "azureblob-chunk-size", "", "Upload chunk size. Must fit in memory.")
flags.VarP(&uploadCutoff, "azureblob-upload-cutoff", "", "Cutoff for switching to chunked upload")
flags.VarP(&chunkSize, "azureblob-chunk-size", "", "Upload chunk size. Must fit in memory.")
}
// Fs represents a remote azure server
@ -165,7 +171,7 @@ func (f *Fs) shouldRetry(err error) (bool, error) {
}
}
}
return fs.ShouldRetry(err), err
return fserrors.ShouldRetry(err), err
}
// NewFs contstructs an Fs from the path, container:path
@ -180,11 +186,11 @@ func NewFs(name, root string) (fs.Fs, error) {
if err != nil {
return nil, err
}
account := fs.ConfigFileGet(name, "account")
account := config.FileGet(name, "account")
if account == "" {
return nil, errors.New("account not found")
}
key := fs.ConfigFileGet(name, "key")
key := config.FileGet(name, "key")
if key == "" {
return nil, errors.New("key not found")
}
@ -193,13 +199,13 @@ func NewFs(name, root string) (fs.Fs, error) {
return nil, errors.Errorf("malformed storage account key: %v", err)
}
endpoint := fs.ConfigFileGet(name, "endpoint", storage.DefaultBaseURL)
endpoint := config.FileGet(name, "endpoint", storage.DefaultBaseURL)
client, err := storage.NewClient(account, key, endpoint, apiVersion, true)
if err != nil {
return nil, errors.Wrap(err, "failed to make azure storage client")
}
client.HTTPClient = fs.Config.Client()
client.HTTPClient = fshttp.NewClient(fs.Config)
bc := client.GetBlobService()
f := &Fs{
@ -473,7 +479,7 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
if f.container == "" {
return fs.ErrorListBucketRequired
}
list := fs.NewListRHelper(callback)
list := walk.NewListRHelper(callback)
err = f.list(dir, true, listChunkSize, func(remote string, object *storage.Blob, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory)
if err != nil {
@ -622,8 +628,8 @@ func (f *Fs) Precision() time.Duration {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5)
}
// Purge deletes all the files and directories including the old versions.
@ -690,9 +696,9 @@ func (o *Object) Remote() string {
}
// Hash returns the MD5 of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 {
return "", hash.ErrHashUnsupported
}
// Convert base64 encoded md5 into lower case hex
if o.md5 == "" {
@ -834,7 +840,7 @@ type openFile struct {
o *Object // Object we are reading for
resp *http.Response // response of the GET
body io.Reader // reading from here
hash hash.Hash // currently accumulating MD5
hash gohash.Hash // currently accumulating MD5
bytes int64 // number of bytes read on this connection
eof bool // whether we have read end of file
}
@ -1059,7 +1065,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
size := src.Size()
blob := o.getBlobWithModTime(src.ModTime())
blob.Properties.ContentType = fs.MimeType(o)
if sourceMD5, _ := src.Hash(fs.HashMD5); sourceMD5 != "" {
if sourceMD5, _ := src.Hash(hash.HashMD5); sourceMD5 != "" {
sourceMD5bytes, err := hex.DecodeString(sourceMD5)
if err == nil {
blob.Properties.ContentMD5 = base64.StdEncoding.EncodeToString(sourceMD5bytes)

4
backend/b2/api/types.go

@ -7,7 +7,7 @@ import (
"strings"
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/fserrors"
)
// Error describes a B2 error response
@ -29,7 +29,7 @@ func (e *Error) Fatal() bool {
return e.Status == 403 // 403 errors shouldn't be retried
}
var _ fs.Fataler = (*Error)(nil)
var _ fserrors.Fataler = (*Error)(nil)
// Account describes a B2 account
type Account struct {

53
backend/b2/b2.go

@ -9,7 +9,7 @@ import (
"bytes"
"crypto/sha1"
"fmt"
"hash"
gohash "hash"
"io"
"net/http"
"path"
@ -21,6 +21,13 @@ import (
"github.com/ncw/rclone/backend/b2/api"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/accounting"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/walk"
"github.com/ncw/rclone/lib/pacer"
"github.com/ncw/rclone/lib/rest"
"github.com/pkg/errors"
@ -48,9 +55,9 @@ var (
minChunkSize = fs.SizeSuffix(5E6)
chunkSize = fs.SizeSuffix(96 * 1024 * 1024)
uploadCutoff = fs.SizeSuffix(200E6)
b2TestMode = fs.StringP("b2-test-mode", "", "", "A flag string for X-Bz-Test-Mode header.")
b2Versions = fs.BoolP("b2-versions", "", false, "Include old versions in directory listings.")
b2HardDelete = fs.BoolP("b2-hard-delete", "", false, "Permanently delete files on remote removal, otherwise hide files.")
b2TestMode = flags.StringP("b2-test-mode", "", "", "A flag string for X-Bz-Test-Mode header.")
b2Versions = flags.BoolP("b2-versions", "", false, "Include old versions in directory listings.")
b2HardDelete = flags.BoolP("b2-hard-delete", "", false, "Permanently delete files on remote removal, otherwise hide files.")
errNotWithVersions = errors.New("can't modify or delete files in --b2-versions mode")
)
@ -72,8 +79,8 @@ func init() {
},
},
})
fs.VarP(&uploadCutoff, "b2-upload-cutoff", "", "Cutoff for switching to chunked upload")
fs.VarP(&chunkSize, "b2-chunk-size", "", "Upload chunk size. Must fit in memory.")
flags.VarP(&uploadCutoff, "b2-upload-cutoff", "", "Cutoff for switching to chunked upload")
flags.VarP(&chunkSize, "b2-chunk-size", "", "Upload chunk size. Must fit in memory.")
}
// Fs represents a remote b2 server
@ -186,7 +193,7 @@ func (f *Fs) shouldRetryNoReauth(resp *http.Response, err error) (bool, error) {
}
return true, err
}
return fs.ShouldRetry(err) || fs.ShouldRetryHTTP(resp, retryErrorCodes), err
return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}
// shouldRetry returns a boolean as to whether this resp and err
@ -236,15 +243,15 @@ func NewFs(name, root string) (fs.Fs, error) {
if err != nil {
return nil, err
}
account := fs.ConfigFileGet(name, "account")
account := config.FileGet(name, "account")
if account == "" {
return nil, errors.New("account not found")
}
key := fs.ConfigFileGet(name, "key")
key := config.FileGet(name, "key")
if key == "" {
return nil, errors.New("key not found")
}
endpoint := fs.ConfigFileGet(name, "endpoint", defaultEndpoint)
endpoint := config.FileGet(name, "endpoint", defaultEndpoint)
f := &Fs{
name: name,
bucket: bucket,
@ -252,7 +259,7 @@ func NewFs(name, root string) (fs.Fs, error) {
account: account,
key: key,
endpoint: endpoint,
srv: rest.NewClient(fs.Config.Client()).SetErrorHandler(errorHandler),
srv: rest.NewClient(fshttp.NewClient(fs.Config)).SetErrorHandler(errorHandler),
pacer: pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant),
bufferTokens: make(chan []byte, fs.Config.Transfers),
}
@ -615,7 +622,7 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
if f.bucket == "" {
return fs.ErrorListBucketRequired
}
list := fs.NewListRHelper(callback)
list := walk.NewListRHelper(callback)
last := ""
err = f.list(dir, true, "", 0, *b2Versions, func(remote string, object *api.File, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory, &last)
@ -868,16 +875,16 @@ func (f *Fs) purge(oldOnly bool) error {
go func() {
defer wg.Done()
for object := range toBeDeleted {
fs.Stats.Checking(object.Name)
accounting.Stats.Checking(object.Name)
checkErr(f.deleteByID(object.ID, object.Name))
fs.Stats.DoneChecking(object.Name)
accounting.Stats.DoneChecking(object.Name)
}
}()
}
last := ""
checkErr(f.list("", true, "", 0, true, func(remote string, object *api.File, isDirectory bool) error {
if !isDirectory {
fs.Stats.Checking(remote)
accounting.Stats.Checking(remote)
if oldOnly && last != remote {
if object.Action == "hide" {
fs.Debugf(remote, "Deleting current version (id %q) as it is a hide marker", object.ID)
@ -890,7 +897,7 @@ func (f *Fs) purge(oldOnly bool) error {
toBeDeleted <- object
}
last = remote
fs.Stats.DoneChecking(remote)
accounting.Stats.DoneChecking(remote)
}
return nil
}))
@ -914,8 +921,8 @@ func (f *Fs) CleanUp() error {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashSHA1)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashSHA1)
}
// ------------------------------------------------------------
@ -939,9 +946,9 @@ func (o *Object) Remote() string {
}
// Hash returns the Sha-1 of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashSHA1 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashSHA1 {
return "", hash.ErrHashUnsupported
}
if o.sha1 == "" {
// Error is logged in readMetaData
@ -1094,7 +1101,7 @@ type openFile struct {
o *Object // Object we are reading for
resp *http.Response // response of the GET
body io.Reader // reading from here
hash hash.Hash // currently accumulating SHA1
hash gohash.Hash // currently accumulating SHA1
bytes int64 // number of bytes read on this connection
eof bool // whether we have read end of file
}
@ -1279,7 +1286,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
modTime := src.ModTime()
calculatedSha1, _ := src.Hash(fs.HashSHA1)
calculatedSha1, _ := src.Hash(hash.HashSHA1)
if calculatedSha1 == "" {
calculatedSha1 = "hex_digits_at_end"
har := newHashAppendingReader(in, sha1.New())

16
backend/b2/upload.go

@ -9,19 +9,21 @@ import (
"crypto/sha1"
"encoding/hex"
"fmt"
"hash"
gohash "hash"
"io"
"strings"
"sync"
"github.com/ncw/rclone/backend/b2/api"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/accounting"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/rest"
"github.com/pkg/errors"
)
type hashAppendingReader struct {
h hash.Hash
h gohash.Hash
in io.Reader
hexSum string
hexReader io.Reader
@ -58,7 +60,7 @@ func (har *hashAppendingReader) HexSum() string {
// newHashAppendingReader takes a Reader and a Hash and will append the hex sum
// after the original reader reaches EOF. The increased size depends on the
// given hash, which may be queried through AdditionalLength()
func newHashAppendingReader(in io.Reader, h hash.Hash) *hashAppendingReader {
func newHashAppendingReader(in io.Reader, h gohash.Hash) *hashAppendingReader {
withHash := io.TeeReader(in, h)
return &hashAppendingReader{h: h, in: withHash}
}
@ -113,7 +115,7 @@ func (f *Fs) newLargeUpload(o *Object, in io.Reader, src fs.ObjectInfo) (up *lar
},
}
// Set the SHA1 if known
if calculatedSha1, err := src.Hash(fs.HashSHA1); err == nil && calculatedSha1 != "" {
if calculatedSha1, err := src.Hash(hash.HashSHA1); err == nil && calculatedSha1 != "" {
request.Info[sha1Key] = calculatedSha1
}
var response api.StartLargeFileResponse
@ -219,7 +221,7 @@ func (up *largeUpload) transferChunk(part int64, body []byte) error {
opts := rest.Opts{
Method: "POST",
RootURL: upload.UploadURL,
Body: fs.AccountPart(up.o, in),
Body: accounting.AccountPart(up.o, in),
ExtraHeaders: map[string]string{
"Authorization": upload.AuthorizationToken,
"X-Bz-Part-Number": fmt.Sprintf("%d", part),
@ -329,7 +331,7 @@ func (up *largeUpload) Stream(initialUploadBlock []byte) (err error) {
errs := make(chan error, 1)
hasMoreParts := true
var wg sync.WaitGroup
fs.AccountByPart(up.o) // Cancel whole file accounting before reading
accounting.AccountByPart(up.o) // Cancel whole file accounting before reading
// Transfer initial chunk
up.size = int64(len(initialUploadBlock))
@ -390,7 +392,7 @@ func (up *largeUpload) Upload() error {
errs := make(chan error, 1)
var wg sync.WaitGroup
var err error
fs.AccountByPart(up.o) // Cancel whole file accounting before reading
accounting.AccountByPart(up.o) // Cancel whole file accounting before reading
outer:
for part := int64(1); part <= up.parts; part++ {
// Check any errors

26
backend/box/box.go

@ -22,9 +22,11 @@ import (
"time"
"github.com/ncw/rclone/backend/box/api"
"github.com/ncw/rclone/box/api"
"github.com/ncw/rclone/dircache"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/dircache"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/ncw/rclone/lib/pacer"
@ -56,7 +58,7 @@ var (
TokenURL: "https://app.box.com/api/oauth2/token",
},
ClientID: rcloneClientID,
ClientSecret: fs.MustReveal(rcloneEncryptedClientSecret),
ClientSecret: config.MustReveal(rcloneEncryptedClientSecret),
RedirectURL: oauthutil.RedirectURL,
}
uploadCutoff = fs.SizeSuffix(50 * 1024 * 1024)
@ -75,14 +77,14 @@ func init() {
}
},
Options: []fs.Option{{
Name: fs.ConfigClientID,
Name: config.ConfigClientID,
Help: "Box App Client Id - leave blank normally.",
}, {
Name: fs.ConfigClientSecret,
Name: config.ConfigClientSecret,
Help: "Box App Client Secret - leave blank normally.",
}},
})
fs.VarP(&uploadCutoff, "box-upload-cutoff", "", "Cutoff for switching to multipart upload")
flags.VarP(&uploadCutoff, "box-upload-cutoff", "", "Cutoff for switching to multipart upload")
}
// Fs represents a remote box
@ -160,7 +162,7 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
authRety = true
fs.Debugf(nil, "Should retry: %v", err)
}
return authRety || fs.ShouldRetry(err) || fs.ShouldRetryHTTP(resp, retryErrorCodes), err
return authRety || fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}
// substitute reserved characters for box
@ -827,8 +829,8 @@ func (f *Fs) DirCacheFlush() {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashSHA1)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashSHA1)
}
// ------------------------------------------------------------
@ -857,9 +859,9 @@ func (o *Object) srvPath() string {
}
// Hash returns the SHA-1 of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashSHA1 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashSHA1 {
return "", hash.ErrHashUnsupported
}
return o.sha1, nil
}

56
backend/cache/cache.go vendored

@ -18,6 +18,10 @@ import (
"github.com/ncw/rclone/backend/crypt"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/walk"
"github.com/pkg/errors"
"golang.org/x/net/context"
"golang.org/x/time/rate"
@ -47,18 +51,18 @@ const (
// Globals
var (
// Flags
cacheDbPath = fs.StringP("cache-db-path", "", filepath.Join(fs.CacheDir, "cache-backend"), "Directory to cache DB")
cacheChunkPath = fs.StringP("cache-chunk-path", "", filepath.Join(fs.CacheDir, "cache-backend"), "Directory to cached chunk files")
cacheDbPurge = fs.BoolP("cache-db-purge", "", false, "Purge the cache DB before")
cacheChunkSize = fs.StringP("cache-chunk-size", "", DefCacheChunkSize, "The size of a chunk")
cacheTotalChunkSize = fs.StringP("cache-total-chunk-size", "", DefCacheTotalChunkSize, "The total size which the chunks can take up from the disk")
cacheChunkCleanInterval = fs.StringP("cache-chunk-clean-interval", "", DefCacheChunkCleanInterval, "Interval at which chunk cleanup runs")
cacheInfoAge = fs.StringP("cache-info-age", "", DefCacheInfoAge, "How much time should object info be stored in cache")
cacheReadRetries = fs.IntP("cache-read-retries", "", DefCacheReadRetries, "How many times to retry a read from a cache storage")
cacheTotalWorkers = fs.IntP("cache-workers", "", DefCacheTotalWorkers, "How many workers should run in parallel to download chunks")
cacheChunkNoMemory = fs.BoolP("cache-chunk-no-memory", "", DefCacheChunkNoMemory, "Disable the in-memory cache for storing chunks during streaming")
cacheRps = fs.IntP("cache-rps", "", int(DefCacheRps), "Limits the number of requests per second to the source FS. -1 disables the rate limiter")
cacheStoreWrites = fs.BoolP("cache-writes", "", DefCacheWrites, "Will cache file data on writes through the FS")
cacheDbPath = flags.StringP("cache-db-path", "", filepath.Join(config.CacheDir, "cache-backend"), "Directory to cache DB")
cacheChunkPath = flags.StringP("cache-chunk-path", "", filepath.Join(config.CacheDir, "cache-backend"), "Directory to cached chunk files")
cacheDbPurge = flags.BoolP("cache-db-purge", "", false, "Purge the cache DB before")
cacheChunkSize = flags.StringP("cache-chunk-size", "", DefCacheChunkSize, "The size of a chunk")
cacheTotalChunkSize = flags.StringP("cache-total-chunk-size", "", DefCacheTotalChunkSize, "The total size which the chunks can take up from the disk")
cacheChunkCleanInterval = flags.StringP("cache-chunk-clean-interval", "", DefCacheChunkCleanInterval, "Interval at which chunk cleanup runs")
cacheInfoAge = flags.StringP("cache-info-age", "", DefCacheInfoAge, "How much time should object info be stored in cache")
cacheReadRetries = flags.IntP("cache-read-retries", "", DefCacheReadRetries, "How many times to retry a read from a cache storage")
cacheTotalWorkers = flags.IntP("cache-workers", "", DefCacheTotalWorkers, "How many workers should run in parallel to download chunks")
cacheChunkNoMemory = flags.BoolP("cache-chunk-no-memory", "", DefCacheChunkNoMemory, "Disable the in-memory cache for storing chunks during streaming")
cacheRps = flags.IntP("cache-rps", "", int(DefCacheRps), "Limits the number of requests per second to the source FS. -1 disables the rate limiter")
cacheStoreWrites = flags.BoolP("cache-writes", "", DefCacheWrites, "Will cache file data on writes through the FS")
)
// Register with Fs
@ -223,7 +227,7 @@ type Fs struct {
// NewFs contstructs an Fs from the path, container:path
func NewFs(name, rpath string) (fs.Fs, error) {
remote := fs.ConfigFileGet(name, "remote")
remote := config.FileGet(name, "remote")
if strings.HasPrefix(remote, name+":") {
return nil, errors.New("can't point cache remote at itself - check the value of the remote setting")
}
@ -235,10 +239,10 @@ func NewFs(name, rpath string) (fs.Fs, error) {
}
fs.Debugf(name, "wrapped %v:%v at root %v", wrappedFs.Name(), wrappedFs.Root(), rpath)
plexURL := fs.ConfigFileGet(name, "plex_url")
plexToken := fs.ConfigFileGet(name, "plex_token")
plexURL := config.FileGet(name, "plex_url")
plexToken := config.FileGet(name, "plex_token")
var chunkSize fs.SizeSuffix
chunkSizeString := fs.ConfigFileGet(name, "chunk_size", DefCacheChunkSize)
chunkSizeString := config.FileGet(name, "chunk_size", DefCacheChunkSize)
if *cacheChunkSize != DefCacheChunkSize {
chunkSizeString = *cacheChunkSize
}
@ -247,7 +251,7 @@ func NewFs(name, rpath string) (fs.Fs, error) {
return nil, errors.Wrapf(err, "failed to understand chunk size", chunkSizeString)
}
var chunkTotalSize fs.SizeSuffix
chunkTotalSizeString := fs.ConfigFileGet(name, "chunk_total_size", DefCacheTotalChunkSize)
chunkTotalSizeString := config.FileGet(name, "chunk_total_size", DefCacheTotalChunkSize)
if *cacheTotalChunkSize != DefCacheTotalChunkSize {
chunkTotalSizeString = *cacheTotalChunkSize
}
@ -260,7 +264,7 @@ func NewFs(name, rpath string) (fs.Fs, error) {
if err != nil {
return nil, errors.Wrapf(err, "failed to understand duration %v", chunkCleanIntervalStr)
}
infoAge := fs.ConfigFileGet(name, "info_age", DefCacheInfoAge)
infoAge := config.FileGet(name, "info_age", DefCacheInfoAge)
if *cacheInfoAge != DefCacheInfoAge {
infoAge = *cacheInfoAge
}
@ -301,10 +305,10 @@ func NewFs(name, rpath string) (fs.Fs, error) {
return nil, errors.Wrapf(err, "failed to connect to the Plex API %v", plexURL)
}
} else {
plexUsername := fs.ConfigFileGet(name, "plex_username")
plexPassword := fs.ConfigFileGet(name, "plex_password")
plexUsername := config.FileGet(name, "plex_username")
plexPassword := config.FileGet(name, "plex_password")
if plexPassword != "" && plexUsername != "" {
decPass, err := fs.Reveal(plexPassword)
decPass, err := config.Reveal(plexPassword)
if err != nil {
decPass = plexPassword
}
@ -319,8 +323,8 @@ func NewFs(name, rpath string) (fs.Fs, error) {
dbPath := *cacheDbPath
chunkPath := *cacheChunkPath
// if the dbPath is non default but the chunk path is default, we overwrite the last to follow the same one as dbPath
if dbPath != filepath.Join(fs.CacheDir, "cache-backend") &&
chunkPath == filepath.Join(fs.CacheDir, "cache-backend") {
if dbPath != filepath.Join(config.CacheDir, "cache-backend") &&
chunkPath == filepath.Join(config.CacheDir, "cache-backend") {
chunkPath = dbPath
}
if filepath.Ext(dbPath) != "" {
@ -506,7 +510,7 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
return cachedEntries, nil
}
func (f *Fs) recurse(dir string, list *fs.ListRHelper) error {
func (f *Fs) recurse(dir string, list *walk.ListRHelper) error {
entries, err := f.List(dir)
if err != nil {
return err
@ -558,7 +562,7 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
}
// if we're here, we're gonna do a standard recursive traversal and cache everything
list := fs.NewListRHelper(callback)
list := walk.NewListRHelper(callback)
err = f.recurse(dir, list)
if err != nil {
return err
@ -895,7 +899,7 @@ func (f *Fs) Move(src fs.Object, remote string) (fs.Object, error) {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
func (f *Fs) Hashes() hash.Set {
return f.Fs.Hashes()
}

60
backend/cache/cache_internal_test.go vendored

@ -20,6 +20,8 @@ import (
//"strings"
"github.com/ncw/rclone/backend/cache"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/object"
//"github.com/ncw/rclone/cmd/mount"
//_ "github.com/ncw/rclone/cmd/cmount"
//"github.com/ncw/rclone/cmd/mountlib"
@ -492,7 +494,7 @@ func writeObjectString(t *testing.T, f fs.Fs, remote, content string) fs.Object
func writeObjectBytes(t *testing.T, f fs.Fs, remote string, data []byte) fs.Object {
in := bytes.NewReader(data)
modTime := time.Now()
objInfo := fs.NewStaticObjectInfo(remote, modTime, int64(len(data)), true, nil, f)
objInfo := object.NewStaticObjectInfo(remote, modTime, int64(len(data)), true, nil, f)
obj, err := f.Put(in, objInfo)
require.NoError(t, err)
@ -503,8 +505,8 @@ func writeObjectBytes(t *testing.T, f fs.Fs, remote string, data []byte) fs.Obje
func updateObjectBytes(t *testing.T, f fs.Fs, remote string, data1 []byte, data2 []byte) fs.Object {
in1 := bytes.NewReader(data1)
in2 := bytes.NewReader(data2)
objInfo1 := fs.NewStaticObjectInfo(remote, time.Now(), int64(len(data1)), true, nil, f)
objInfo2 := fs.NewStaticObjectInfo(remote, time.Now(), int64(len(data2)), true, nil, f)
objInfo1 := object.NewStaticObjectInfo(remote, time.Now(), int64(len(data1)), true, nil, f)
objInfo2 := object.NewStaticObjectInfo(remote, time.Now(), int64(len(data2)), true, nil, f)
obj, err := f.Put(in1, objInfo1)
require.NoError(t, err)
@ -540,15 +542,15 @@ func cleanupFs(t *testing.T, f fs.Fs, b *cache.Persistent) {
func newLocalCacheCryptFs(t *testing.T, localRemote, cacheRemote, cryptRemote string, purge bool, cfg map[string]string) (fs.Fs, *cache.Persistent) {
fstest.Initialise()
dbPath := filepath.Join(fs.CacheDir, "cache-backend", cacheRemote+".db")
chunkPath := filepath.Join(fs.CacheDir, "cache-backend", cacheRemote)
dbPath := filepath.Join(config.CacheDir, "cache-backend", cacheRemote+".db")
chunkPath := filepath.Join(config.CacheDir, "cache-backend", cacheRemote)
boltDb, err := cache.GetPersistent(dbPath, chunkPath, &cache.Features{PurgeDb: true})
require.NoError(t, err)
localExists := false
cacheExists := false
cryptExists := false
for _, s := range fs.ConfigFileSections() {
for _, s := range config.FileSections() {
if s == localRemote {
localExists = true
}
@ -563,28 +565,28 @@ func newLocalCacheCryptFs(t *testing.T, localRemote, cacheRemote, cryptRemote st
localRemoteWrap := ""
if !localExists {
localRemoteWrap = localRemote + ":/var/tmp/" + localRemote
fs.ConfigFileSet(localRemote, "type", "local")
fs.ConfigFileSet(localRemote, "nounc", "true")
config.FileSet(localRemote, "type", "local")
config.FileSet(localRemote, "nounc", "true")
}
if !cacheExists {
fs.ConfigFileSet(cacheRemote, "type", "cache")
fs.ConfigFileSet(cacheRemote, "remote", localRemoteWrap)
config.FileSet(cacheRemote, "type", "cache")
config.FileSet(cacheRemote, "remote", localRemoteWrap)
}
if c, ok := cfg["chunk_size"]; ok {
fs.ConfigFileSet(cacheRemote, "chunk_size", c)
config.FileSet(cacheRemote, "chunk_size", c)
} else {
fs.ConfigFileSet(cacheRemote, "chunk_size", "1m")
config.FileSet(cacheRemote, "chunk_size", "1m")
}
if c, ok := cfg["chunk_total_size"]; ok {
fs.ConfigFileSet(cacheRemote, "chunk_total_size", c)
config.FileSet(cacheRemote, "chunk_total_size", c)
} else {
fs.ConfigFileSet(cacheRemote, "chunk_total_size", "2m")
config.FileSet(cacheRemote, "chunk_total_size", "2m")
}
if c, ok := cfg["info_age"]; ok {
fs.ConfigFileSet(cacheRemote, "info_age", c)
config.FileSet(cacheRemote, "info_age", c)
} else {
fs.ConfigFileSet(cacheRemote, "info_age", infoAge.String())
config.FileSet(cacheRemote, "info_age", infoAge.String())
}
if !cryptExists {
@ -627,14 +629,14 @@ func newLocalCacheCryptFs(t *testing.T, localRemote, cacheRemote, cryptRemote st
func newLocalCacheFs(t *testing.T, localRemote, cacheRemote string, cfg map[string]string) (fs.Fs, *cache.Persistent) {
fstest.Initialise()
dbPath := filepath.Join(fs.CacheDir, "cache-backend", cacheRemote+".db")
chunkPath := filepath.Join(fs.CacheDir, "cache-backend", cacheRemote)
dbPath := filepath.Join(config.CacheDir, "cache-backend", cacheRemote+".db")
chunkPath := filepath.Join(config.CacheDir, "cache-backend", cacheRemote)
boltDb, err := cache.GetPersistent(dbPath, chunkPath, &cache.Features{PurgeDb: true})
require.NoError(t, err)
localExists := false
cacheExists := false
for _, s := range fs.ConfigFileSections() {
for _, s := range config.FileSections() {
if s == localRemote {
localExists = true
}
@ -646,28 +648,28 @@ func newLocalCacheFs(t *testing.T, localRemote, cacheRemote string, cfg map[stri
localRemoteWrap := ""
if !localExists {
localRemoteWrap = localRemote + ":/var/tmp/" + localRemote
fs.ConfigFileSet(localRemote, "type", "local")
fs.ConfigFileSet(localRemote, "nounc", "true")
config.FileSet(localRemote, "type", "local")
config.FileSet(localRemote, "nounc", "true")
}
if !cacheExists {
fs.ConfigFileSet(cacheRemote, "type", "cache")
fs.ConfigFileSet(cacheRemote, "remote", localRemoteWrap)
config.FileSet(cacheRemote, "type", "cache")
config.FileSet(cacheRemote, "remote", localRemoteWrap)
}
if c, ok := cfg["chunk_size"]; ok {
fs.ConfigFileSet(cacheRemote, "chunk_size", c)
config.FileSet(cacheRemote, "chunk_size", c)
} else {
fs.ConfigFileSet(cacheRemote, "chunk_size", "1m")
config.FileSet(cacheRemote, "chunk_size", "1m")
}
if c, ok := cfg["chunk_total_size"]; ok {
fs.ConfigFileSet(cacheRemote, "chunk_total_size", c)
config.FileSet(cacheRemote, "chunk_total_size", c)
} else {
fs.ConfigFileSet(cacheRemote, "chunk_total_size", "2m")
config.FileSet(cacheRemote, "chunk_total_size", "2m")
}
if c, ok := cfg["info_age"]; ok {
fs.ConfigFileSet(cacheRemote, "info_age", c)
config.FileSet(cacheRemote, "info_age", c)
} else {
fs.ConfigFileSet(cacheRemote, "info_age", infoAge.String())
config.FileSet(cacheRemote, "info_age", infoAge.String())
}
if c, ok := cfg["cache-chunk-no-memory"]; ok {

15
backend/cache/object.go vendored

@ -13,6 +13,7 @@ import (
"strconv"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/hash"
)
// Object is a generic file like object that stores basic information about it
@ -27,7 +28,7 @@ type Object struct {
CacheStorable bool `json:"storable"` // says whether this object can be stored
CacheType string `json:"cacheType"`
CacheTs time.Time `json:"cacheTs"`
cacheHashes map[fs.HashType]string // all supported hashes cached
cacheHashes map[hash.Type]string // all supported hashes cached
refreshMutex sync.Mutex
}
@ -80,10 +81,10 @@ func (o *Object) UnmarshalJSON(b []byte) error {
return err
}
o.cacheHashes = make(map[fs.HashType]string)
o.cacheHashes = make(map[hash.Type]string)
for k, v := range aux.Hashes {
ht, _ := strconv.Atoi(k)
o.cacheHashes[fs.HashType(ht)] = v
o.cacheHashes[hash.Type(ht)] = v
}
return nil
@ -112,7 +113,7 @@ func (o *Object) updateData(source fs.Object) {
o.CacheSize = source.Size()
o.CacheStorable = source.Storable()
o.CacheTs = time.Now()
o.cacheHashes = make(map[fs.HashType]string)
o.cacheHashes = make(map[hash.Type]string)
}
// Fs returns its FS info
@ -251,7 +252,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
o.CacheModTime = src.ModTime().UnixNano()
o.CacheSize = src.Size()
o.cacheHashes = make(map[fs.HashType]string)
o.cacheHashes = make(map[hash.Type]string)
o.persist()
return nil
@ -274,9 +275,9 @@ func (o *Object) Remove() error {
// Hash requests a hash of the object and stores in the cache
// since it might or might not be called, this is lazy loaded
func (o *Object) Hash(ht fs.HashType) (string, error) {
func (o *Object) Hash(ht hash.Type) (string, error) {
if o.cacheHashes == nil {
o.cacheHashes = make(map[fs.HashType]string)
o.cacheHashes = make(map[hash.Type]string)
}
cachedHash, found := o.cacheHashes[ht]

5
backend/cache/plex.go vendored

@ -13,6 +13,7 @@ import (
"sync"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
)
const (
@ -107,8 +108,8 @@ func (p *plexConnector) authenticate() error {
}
p.token = token
if p.token != "" {
fs.ConfigFileSet(p.f.Name(), "plex_token", p.token)
fs.SaveConfig()
config.FileSet(p.f.Name(), "plex_token", p.token)
config.SaveConfig()
fs.Infof(p.f.Name(), "Connected to Plex server: %v", p.url.String())
}

33
backend/crypt/crypt.go

@ -10,13 +10,16 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/hash"
"github.com/pkg/errors"
)
// Globals
var (
// Flags
cryptShowMapping = fs.BoolP("crypt-show-mapping", "", false, "For all files listed show how the names encrypt.")
cryptShowMapping = flags.BoolP("crypt-show-mapping", "", false, "For all files listed show how the names encrypt.")
)
// Register with Fs
@ -71,25 +74,25 @@ func init() {
// NewFs contstructs an Fs from the path, container:path
func NewFs(name, rpath string) (fs.Fs, error) {
mode, err := NewNameEncryptionMode(fs.ConfigFileGet(name, "filename_encryption", "standard"))
mode, err := NewNameEncryptionMode(config.FileGet(name, "filename_encryption", "standard"))
if err != nil {
return nil, err
}
dirNameEncrypt, err := strconv.ParseBool(fs.ConfigFileGet(name, "directory_name_encryption", "true"))
dirNameEncrypt, err := strconv.ParseBool(config.FileGet(name, "directory_name_encryption", "true"))
if err != nil {
return nil, err
}
password := fs.ConfigFileGet(name, "password", "")
password := config.FileGet(name, "password", "")
if password == "" {
return nil, errors.New("password not set in config file")
}
password, err = fs.Reveal(password)
password, err = config.Reveal(password)
if err != nil {
return nil, errors.Wrap(err, "failed to decrypt password")
}
salt := fs.ConfigFileGet(name, "password2", "")
salt := config.FileGet(name, "password2", "")
if salt != "" {
salt, err = fs.Reveal(salt)
salt, err = config.Reveal(salt)
if err != nil {
return nil, errors.Wrap(err, "failed to decrypt password2")
}
@ -98,7 +101,7 @@ func NewFs(name, rpath string) (fs.Fs, error) {
if err != nil {
return nil, errors.Wrap(err, "failed to make cipher")
}
remote := fs.ConfigFileGet(name, "remote")
remote := config.FileGet(name, "remote")
if strings.HasPrefix(remote, name+":") {
return nil, errors.New("can't point crypt remote at itself - check the value of the remote setting")
}
@ -305,8 +308,8 @@ func (f *Fs) PutStream(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashNone)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashNone)
}
// Mkdir makes the directory (container, bucket)
@ -459,7 +462,7 @@ func (f *Fs) DecryptFileName(encryptedFileName string) (string, error) {
// src with it, and calcuates the hash given by HashType on the fly
//
// Note that we break lots of encapsulation in this function.
func (f *Fs) ComputeHash(o *Object, src fs.Object, hashType fs.HashType) (hash string, err error) {
func (f *Fs) ComputeHash(o *Object, src fs.Object, hashType hash.Type) (hashStr string, err error) {
// Read the nonce - opening the file is sufficient to read the nonce in
in, err := o.Open()
if err != nil {
@ -499,7 +502,7 @@ func (f *Fs) ComputeHash(o *Object, src fs.Object, hashType fs.HashType) (hash s
}
// pipe into hash
m := fs.NewMultiHasher()
m := hash.NewMultiHasher()
_, err = io.Copy(m, out)
if err != nil {
return "", errors.Wrap(err, "failed to hash data")
@ -558,8 +561,8 @@ func (o *Object) Size() int64 {
// Hash returns the selected checksum of the file
// If no checksum is available it returns ""
func (o *Object) Hash(hash fs.HashType) (string, error) {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(ht hash.Type) (string, error) {
return "", hash.ErrHashUnsupported
}
// UnWrap returns the wrapped Object
@ -652,7 +655,7 @@ func (o *ObjectInfo) Size() int64 {
// Hash returns the selected checksum of the file
// If no checksum is available it returns ""
func (o *ObjectInfo) Hash(hash fs.HashType) (string, error) {
func (o *ObjectInfo) Hash(hash hash.Type) (string, error) {
return "", nil
}

8
backend/crypt/crypt_config_test.go

@ -4,7 +4,7 @@ import (
"os"
"path/filepath"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fstest/fstests"
)
@ -19,15 +19,15 @@ func init() {
fstests.ExtraConfig = []fstests.ExtraConfigItem{
{Name: name, Key: "type", Value: "crypt"},
{Name: name, Key: "remote", Value: tempdir},
{Name: name, Key: "password", Value: fs.MustObscure("potato")},
{Name: name, Key: "password", Value: config.MustObscure("potato")},
{Name: name, Key: "filename_encryption", Value: "standard"},
{Name: name2, Key: "type", Value: "crypt"},
{Name: name2, Key: "remote", Value: tempdir2},
{Name: name2, Key: "password", Value: fs.MustObscure("potato2")},
{Name: name2, Key: "password", Value: config.MustObscure("potato2")},
{Name: name2, Key: "filename_encryption", Value: "off"},
{Name: name3, Key: "type", Value: "crypt"},
{Name: name3, Key: "remote", Value: tempdir3},
{Name: name3, Key: "password", Value: fs.MustObscure("potato2")},
{Name: name3, Key: "password", Value: config.MustObscure("potato2")},
{Name: name3, Key: "filename_encryption", Value: "obfuscate"},
}
fstests.SkipBadWindowsCharacters[name3+":"] = true

58
backend/drive/drive.go

@ -21,11 +21,15 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/dircache"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/ncw/rclone/lib/pacer"
"github.com/pkg/errors"
"github.com/spf13/pflag"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"google.golang.org/api/drive/v2"
@ -46,13 +50,13 @@ const (
// Globals
var (
// Flags
driveAuthOwnerOnly = fs.BoolP("drive-auth-owner-only", "", false, "Only consider files owned by the authenticated user.")
driveUseTrash = fs.BoolP("drive-use-trash", "", true, "Send files to the trash instead of deleting permanently.")
driveSkipGdocs = fs.BoolP("drive-skip-gdocs", "", false, "Skip google documents in all listings.")
driveSharedWithMe = fs.BoolP("drive-shared-with-me", "", false, "Only show files that are shared with me")
driveTrashedOnly = fs.BoolP("drive-trashed-only", "", false, "Only show files that are in the trash")
driveExtensions = fs.StringP("drive-formats", "", defaultExtensions, "Comma separated list of preferred formats for downloading Google docs.")
driveListChunk = pflag.Int64P("drive-list-chunk", "", 1000, "Size of listing chunk 100-1000. 0 to disable.")
driveAuthOwnerOnly = flags.BoolP("drive-auth-owner-only", "", false, "Only consider files owned by the authenticated user.")
driveUseTrash = flags.BoolP("drive-use-trash", "", true, "Send files to the trash instead of deleting permanently.")
driveSkipGdocs = flags.BoolP("drive-skip-gdocs", "", false, "Skip google documents in all listings.")
driveSharedWithMe = flags.BoolP("drive-shared-with-me", "", false, "Only show files that are shared with me")
driveTrashedOnly = flags.BoolP("drive-trashed-only", "", false, "Only show files that are in the trash")
driveExtensions = flags.StringP("drive-formats", "", defaultExtensions, "Comma separated list of preferred formats for downloading Google docs.")
driveListChunk = flags.Int64P("drive-list-chunk", "", 1000, "Size of listing chunk 100-1000. 0 to disable.")
// chunkSize is the size of the chunks created during a resumable upload and should be a power of two.
// 1<<18 is the minimum size supported by the Google uploader, and there is no maximum.
chunkSize = fs.SizeSuffix(8 * 1024 * 1024)
@ -62,7 +66,7 @@ var (
Scopes: []string{"https://www.googleapis.com/auth/drive"},
Endpoint: google.Endpoint,
ClientID: rcloneClientID,
ClientSecret: fs.MustReveal(rcloneEncryptedClientSecret),
ClientSecret: config.MustReveal(rcloneEncryptedClientSecret),
RedirectURL: oauthutil.TitleBarRedirectURL,
}
mimeTypeToExtension = map[string]string{
@ -99,7 +103,7 @@ func init() {
NewFs: NewFs,
Config: func(name string) {
var err error
if fs.ConfigFileGet(name, "service_account_file") == "" {
if config.FileGet(name, "service_account_file") == "" {
err = oauthutil.Config("drive", name, driveConfig)
if err != nil {
log.Fatalf("Failed to configure token: %v", err)
@ -111,18 +115,18 @@ func init() {
}
},
Options: []fs.Option{{
Name: fs.ConfigClientID,
Name: config.ConfigClientID,
Help: "Google Application Client Id - leave blank normally.",
}, {
Name: fs.ConfigClientSecret,
Name: config.ConfigClientSecret,
Help: "Google Application Client Secret - leave blank normally.",
}, {
Name: "service_account_file",
Help: "Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login.",
}},
})
fs.VarP(&driveUploadCutoff, "drive-upload-cutoff", "", "Cutoff for switching to chunked upload")
fs.VarP(&chunkSize, "drive-chunk-size", "", "Upload chunk size. Must a power of 2 >= 256k.")
flags.VarP(&driveUploadCutoff, "drive-upload-cutoff", "", "Cutoff for switching to chunked upload")
flags.VarP(&chunkSize, "drive-chunk-size", "", "Upload chunk size. Must a power of 2 >= 256k.")
// Invert mimeTypeToExtension
extensionToMimeType = make(map[string]string, len(mimeTypeToExtension))
@ -185,7 +189,7 @@ func (f *Fs) Features() *fs.Features {
func shouldRetry(err error) (again bool, errOut error) {
again = false
if err != nil {
if fs.ShouldRetry(err) {
if fserrors.ShouldRetry(err) {
again = true
} else {
switch gerr := err.(type) {
@ -337,13 +341,13 @@ func (f *Fs) parseExtensions(extensions string) error {
// Figure out if the user wants to use a team drive
func configTeamDrive(name string) error {
teamDrive := fs.ConfigFileGet(name, "team_drive")
teamDrive := config.FileGet(name, "team_drive")
if teamDrive == "" {
fmt.Printf("Configure this as a team drive?\n")
} else {
fmt.Printf("Change current team drive ID %q?\n", teamDrive)
}
if !fs.Confirm() {
if !config.Confirm() {
return nil
}
client, err := authenticate(name)
@ -379,9 +383,9 @@ func configTeamDrive(name string) error {
if len(driveIDs) == 0 {
fmt.Printf("No team drives found in your account")
} else {
driveID = fs.Choose("Enter a Team Drive ID", driveIDs, driveNames, true)
driveID = config.Choose("Enter a Team Drive ID", driveIDs, driveNames, true)
}
fs.ConfigFileSet(name, "team_drive", driveID)
config.FileSet(name, "team_drive", driveID)
return nil
}
@ -399,7 +403,7 @@ func getServiceAccountClient(keyJsonfilePath string) (*http.Client, error) {
if err != nil {
return nil, errors.Wrap(err, "error processing credentials")
}
ctxWithSpecialClient := oauthutil.Context(fs.Config.Client())
ctxWithSpecialClient := oauthutil.Context(fshttp.NewClient(fs.Config))
return oauth2.NewClient(ctxWithSpecialClient, conf.TokenSource(ctxWithSpecialClient)), nil
}
@ -407,7 +411,7 @@ func authenticate(name string) (*http.Client, error) {
var oAuthClient *http.Client
var err error
serviceAccountPath := fs.ConfigFileGet(name, "service_account_file")
serviceAccountPath := config.FileGet(name, "service_account_file")
if serviceAccountPath != "" {
oAuthClient, err = getServiceAccountClient(serviceAccountPath)
if err != nil {
@ -444,7 +448,7 @@ func NewFs(name, path string) (fs.Fs, error) {
root: root,
pacer: newPacer(),
}
f.teamDriveID = fs.ConfigFileGet(name, "team_drive")
f.teamDriveID = config.FileGet(name, "team_drive")
f.isTeamDrive = f.teamDriveID != ""
f.features = (&fs.Features{
DuplicateFiles: true,
@ -1188,8 +1192,8 @@ func (f *Fs) DirCacheFlush() {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5)
}
// ------------------------------------------------------------
@ -1213,9 +1217,9 @@ func (o *Object) Remote() string {
}
// Hash returns the Md5sum of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 {
return "", hash.ErrHashUnsupported
}
return o.md5sum, nil
}

6
backend/drive/upload.go

@ -20,6 +20,8 @@ import (
"strconv"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/lib/readers"
"github.com/pkg/errors"
"google.golang.org/api/drive/v2"
"google.golang.org/api/googleapi"
@ -201,7 +203,7 @@ func (rx *resumableUpload) Upload() (*drive.File, error) {
if reqSize >= int64(chunkSize) {
reqSize = int64(chunkSize)
}
chunk := fs.NewRepeatableLimitReaderBuffer(rx.Media, buf, reqSize)
chunk := readers.NewRepeatableLimitReaderBuffer(rx.Media, buf, reqSize)
// Transfer the chunk
err = rx.f.pacer.Call(func() (bool, error) {
@ -241,7 +243,7 @@ func (rx *resumableUpload) Upload() (*drive.File, error) {
// Handle 404 Not Found errors when doing resumable uploads by starting
// the entire upload over from the beginning.
if rx.ret == nil {
return nil, fs.RetryErrorf("Incomplete upload - retry, last error %d", StatusCode)
return nil, fserrors.RetryErrorf("Incomplete upload - retry, last error %d", StatusCode)
}
return rx.ret, nil
}

35
backend/dropbox/dropbox.go

@ -34,8 +34,13 @@ import (
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox"
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/files"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/ncw/rclone/lib/pacer"
"github.com/ncw/rclone/lib/readers"
"github.com/pkg/errors"
"golang.org/x/oauth2"
)
@ -59,7 +64,7 @@ var (
// },
Endpoint: dropbox.OAuthEndpoint(""),
ClientID: rcloneClientID,
ClientSecret: fs.MustReveal(rcloneEncryptedClientSecret),
ClientSecret: config.MustReveal(rcloneEncryptedClientSecret),
RedirectURL: oauthutil.RedirectLocalhostURL,
}
// A regexp matching path names for files Dropbox ignores
@ -112,7 +117,7 @@ func init() {
Help: "Dropbox App Secret - leave blank normally.",
}},
})
fs.VarP(&uploadChunkSize, "dropbox-chunk-size", "", fmt.Sprintf("Upload chunk size. Max %v.", maxUploadChunkSize))
flags.VarP(&uploadChunkSize, "dropbox-chunk-size", "", fmt.Sprintf("Upload chunk size. Max %v.", maxUploadChunkSize))
}
// Fs represents a remote dropbox server
@ -170,7 +175,7 @@ func shouldRetry(err error) (bool, error) {
if strings.Contains(baseErrString, "too_many_write_operations") || strings.Contains(baseErrString, "too_many_requests") {
return true, err
}
return fs.ShouldRetry(err), err
return fserrors.ShouldRetry(err), err
}
// NewFs contstructs an Fs from the path, container:path
@ -181,11 +186,11 @@ func NewFs(name, root string) (fs.Fs, error) {
// Convert the old token if it exists. The old token was just
// just a string, the new one is a JSON blob
oldToken := strings.TrimSpace(fs.ConfigFileGet(name, fs.ConfigToken))
oldToken := strings.TrimSpace(config.FileGet(name, config.ConfigToken))
if oldToken != "" && oldToken[0] != '{' {
fs.Infof(name, "Converting token to new format")
newToken := fmt.Sprintf(`{"access_token":"%s","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}`, oldToken)
err := fs.ConfigSetValueAndSave(name, fs.ConfigToken, newToken)
err := config.SetValueAndSave(name, config.ConfigToken, newToken)
if err != nil {
return nil, errors.Wrap(err, "NewFS convert token")
}
@ -675,8 +680,8 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashDropbox)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashDropbox)
}
// ------------------------------------------------------------
@ -700,9 +705,9 @@ func (o *Object) Remote() string {
}
// Hash returns the dropbox special hash
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashDropbox {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashDropbox {
return "", hash.ErrHashUnsupported
}
err := o.readMetaData()
if err != nil {
@ -813,7 +818,7 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
case files.DownloadAPIError:
// Don't attempt to retry copyright violation errors
if e.EndpointError.Path.Tag == files.LookupErrorRestrictedContent {
return nil, fs.NoRetryError(err)
return nil, fserrors.NoRetryError(err)
}
}
@ -831,7 +836,7 @@ func (o *Object) uploadChunked(in0 io.Reader, commitInfo *files.CommitInfo, size
if size != -1 {
chunks = int(size/chunkSize) + 1
}
in := fs.NewCountingReader(in0)
in := readers.NewCountingReader(in0)
buf := make([]byte, int(chunkSize))
fmtChunk := func(cur int, last bool) {
@ -847,7 +852,7 @@ func (o *Object) uploadChunked(in0 io.Reader, commitInfo *files.CommitInfo, size
// write the first chunk
fmtChunk(1, false)
var res *files.UploadSessionStartResult
chunk := fs.NewRepeatableLimitReaderBuffer(in, buf, chunkSize)
chunk := readers.NewRepeatableLimitReaderBuffer(in, buf, chunkSize)
err = o.fs.pacer.Call(func() (bool, error) {
// seek to the start in case this is a retry
if _, err = chunk.Seek(0, 0); err != nil {
@ -883,7 +888,7 @@ func (o *Object) uploadChunked(in0 io.Reader, commitInfo *files.CommitInfo, size
}
cursor.Offset = in.BytesRead()
fmtChunk(currentChunk, false)
chunk = fs.NewRepeatableLimitReaderBuffer(in, buf, chunkSize)
chunk = readers.NewRepeatableLimitReaderBuffer(in, buf, chunkSize)
err = o.fs.pacer.Call(func() (bool, error) {
// seek to the start in case this is a retry
if _, err = chunk.Seek(0, 0); err != nil {
@ -906,7 +911,7 @@ func (o *Object) uploadChunked(in0 io.Reader, commitInfo *files.CommitInfo, size
Commit: commitInfo,
}
fmtChunk(currentChunk, true)
chunk = fs.NewRepeatableReaderBuffer(in, buf)
chunk = readers.NewRepeatableReaderBuffer(in, buf)
err = o.fs.pacer.Call(func() (bool, error) {
// seek to the start in case this is a retry
if _, err = chunk.Seek(0, 0); err != nil {

38
backend/ftp/ftp.go

@ -13,6 +13,8 @@ import (
"github.com/jlaffaye/ftp"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/hash"
"github.com/pkg/errors"
)
@ -160,33 +162,33 @@ func (f *Fs) putFtpConnection(pc **ftp.ServerConn, err error) {
func NewFs(name, root string) (ff fs.Fs, err error) {
// defer fs.Trace(nil, "name=%q, root=%q", name, root)("fs=%v, err=%v", &ff, &err)
// FIXME Convert the old scheme used for the first beta - remove after release
if ftpURL := fs.ConfigFileGet(name, "url"); ftpURL != "" {
if ftpURL := config.FileGet(name, "url"); ftpURL != "" {
fs.Infof(name, "Converting old configuration")
u, err := url.Parse(ftpURL)
if err != nil {
return nil, errors.Wrapf(err, "Failed to parse old url %q", ftpURL)
}
parts := strings.Split(u.Host, ":")
fs.ConfigFileSet(name, "host", parts[0])
config.FileSet(name, "host", parts[0])
if len(parts) > 1 {
fs.ConfigFileSet(name, "port", parts[1])
config.FileSet(name, "port", parts[1])
}
fs.ConfigFileSet(name, "host", u.Host)
fs.ConfigFileSet(name, "user", fs.ConfigFileGet(name, "username"))
fs.ConfigFileSet(name, "pass", fs.ConfigFileGet(name, "password"))
fs.ConfigFileDeleteKey(name, "username")
fs.ConfigFileDeleteKey(name, "password")
fs.ConfigFileDeleteKey(name, "url")
fs.SaveConfig()
config.FileSet(name, "host", u.Host)
config.FileSet(name, "user", config.FileGet(name, "username"))
config.FileSet(name, "pass", config.FileGet(name, "password"))
config.FileDeleteKey(name, "username")
config.FileDeleteKey(name, "password")
config.FileDeleteKey(name, "url")
config.SaveConfig()
if u.Path != "" && u.Path != "/" {
fs.Errorf(name, "Path %q in FTP URL no longer supported - put it on the end of the remote %s:%s", u.Path, name, u.Path)
}
}
host := fs.ConfigFileGet(name, "host")
user := fs.ConfigFileGet(name, "user")
pass := fs.ConfigFileGet(name, "pass")
port := fs.ConfigFileGet(name, "port")
pass, err = fs.Reveal(pass)
host := config.FileGet(name, "host")
user := config.FileGet(name, "user")
pass := config.FileGet(name, "pass")
port := config.FileGet(name, "port")
pass, err = config.Reveal(pass)
if err != nil {
return nil, errors.Wrap(err, "NewFS decrypt password")
}
@ -346,7 +348,7 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
}
// Hashes are not supported
func (f *Fs) Hashes() fs.HashSet {
func (f *Fs) Hashes() hash.Set {
return 0
}
@ -565,8 +567,8 @@ func (o *Object) Remote() string {
}
// Hash returns the hash of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
return "", hash.ErrHashUnsupported
}
// Size returns the size of an object in bytes

43
backend/googlecloudstorage/googlecloudstorage.go

@ -28,6 +28,11 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/walk"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/pkg/errors"
"golang.org/x/oauth2"
@ -46,14 +51,14 @@ const (
)
var (
gcsLocation = fs.StringP("gcs-location", "", "", "Default location for buckets (us|eu|asia|us-central1|us-east1|us-east4|us-west1|asia-east1|asia-noetheast1|asia-southeast1|australia-southeast1|europe-west1|europe-west2).")
gcsStorageClass = fs.StringP("gcs-storage-class", "", "", "Default storage class for buckets (MULTI_REGIONAL|REGIONAL|STANDARD|NEARLINE|COLDLINE|DURABLE_REDUCED_AVAILABILITY).")
gcsLocation = flags.StringP("gcs-location", "", "", "Default location for buckets (us|eu|asia|us-central1|us-east1|us-east4|us-west1|asia-east1|asia-noetheast1|asia-southeast1|australia-southeast1|europe-west1|europe-west2).")
gcsStorageClass = flags.StringP("gcs-storage-class", "", "", "Default storage class for buckets (MULTI_REGIONAL|REGIONAL|STANDARD|NEARLINE|COLDLINE|DURABLE_REDUCED_AVAILABILITY).")
// Description of how to auth for this app
storageConfig = &oauth2.Config{
Scopes: []string{storage.DevstorageFullControlScope},
Endpoint: google.Endpoint,
ClientID: rcloneClientID,
ClientSecret: fs.MustReveal(rcloneEncryptedClientSecret),
ClientSecret: config.MustReveal(rcloneEncryptedClientSecret),
RedirectURL: oauthutil.TitleBarRedirectURL,
}
)
@ -65,7 +70,7 @@ func init() {
Description: "Google Cloud Storage (this is not Google Drive)",
NewFs: NewFs,
Config: func(name string) {
if fs.ConfigFileGet(name, "service_account_file") != "" {
if config.FileGet(name, "service_account_file") != "" {
return
}
err := oauthutil.Config("google cloud storage", name, storageConfig)
@ -74,10 +79,10 @@ func init() {
}
},
Options: []fs.Option{{
Name: fs.ConfigClientID,
Name: config.ConfigClientID,
Help: "Google Application Client Id - leave blank normally.",
}, {
Name: fs.ConfigClientSecret,
Name: config.ConfigClientSecret,
Help: "Google Application Client Secret - leave blank normally.",
}, {
Name: "project_number",
@ -280,7 +285,7 @@ func getServiceAccountClient(keyJsonfilePath string) (*http.Client, error) {
if err != nil {
return nil, errors.Wrap(err, "error processing credentials")
}
ctxWithSpecialClient := oauthutil.Context(fs.Config.Client())
ctxWithSpecialClient := oauthutil.Context(fshttp.NewClient(fs.Config))
return oauth2.NewClient(ctxWithSpecialClient, conf.TokenSource(ctxWithSpecialClient)), nil
}
@ -289,7 +294,7 @@ func NewFs(name, root string) (fs.Fs, error) {
var oAuthClient *http.Client
var err error
serviceAccountPath := fs.ConfigFileGet(name, "service_account_file")
serviceAccountPath := config.FileGet(name, "service_account_file")
if serviceAccountPath != "" {
oAuthClient, err = getServiceAccountClient(serviceAccountPath)
if err != nil {
@ -311,11 +316,11 @@ func NewFs(name, root string) (fs.Fs, error) {
name: name,
bucket: bucket,
root: directory,
projectNumber: fs.ConfigFileGet(name, "project_number"),
objectACL: fs.ConfigFileGet(name, "object_acl"),
bucketACL: fs.ConfigFileGet(name, "bucket_acl"),
location: fs.ConfigFileGet(name, "location"),
storageClass: fs.ConfigFileGet(name, "storage_class"),
projectNumber: config.FileGet(name, "project_number"),
objectACL: config.FileGet(name, "object_acl"),
bucketACL: config.FileGet(name, "bucket_acl"),
location: config.FileGet(name, "location"),
storageClass: config.FileGet(name, "storage_class"),
}
f.features = (&fs.Features{
ReadMimeType: true,
@ -538,7 +543,7 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
if f.bucket == "" {
return fs.ErrorListBucketRequired
}
list := fs.NewListRHelper(callback)
list := walk.NewListRHelper(callback)
err = f.list(dir, true, func(remote string, object *storage.Object, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory)
if err != nil {
@ -669,8 +674,8 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5)
}
// ------------------------------------------------------------
@ -694,9 +699,9 @@ func (o *Object) Remote() string {
}
// Hash returns the Md5sum of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 {
return "", hash.ErrHashUnsupported
}
return o.md5sum, nil
}

17
backend/http/http.go

@ -17,6 +17,9 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/rest"
"github.com/pkg/errors"
"golang.org/x/net/html"
@ -79,7 +82,7 @@ func statusError(res *http.Response, err error) error {
// NewFs creates a new Fs object from the name and root. It connects to
// the host specified in the config file.
func NewFs(name, root string) (fs.Fs, error) {
endpoint := fs.ConfigFileGet(name, "url")
endpoint := config.FileGet(name, "url")
if !strings.HasSuffix(endpoint, "/") {
endpoint += "/"
}
@ -94,7 +97,7 @@ func NewFs(name, root string) (fs.Fs, error) {
return nil, err
}
client := fs.Config.Client()
client := fshttp.NewClient(fs.Config)
var isFile = false
if !strings.HasSuffix(u.String(), "/") {
@ -363,8 +366,8 @@ func (o *Object) Remote() string {
}
// Hash returns "" since HTTP (in Go or OpenSSH) doesn't support remote calculation of hashes
func (o *Object) Hash(r fs.HashType) (string, error) {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(r hash.Type) (string, error) {
return "", hash.ErrHashUnsupported
}
// Size returns the size in bytes of the remote http file
@ -434,9 +437,9 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
return res.Body, nil
}
// Hashes returns fs.HashNone to indicate remote hashing is unavailable
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashNone)
// Hashes returns hash.HashNone to indicate remote hashing is unavailable
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashNone)
}
// Mkdir makes the root directory of the Fs object

7
backend/http/http_internal_test.go

@ -15,6 +15,7 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fstest"
"github.com/ncw/rclone/lib/rest"
"github.com/stretchr/testify/assert"
@ -36,12 +37,12 @@ func prepareServer(t *testing.T) func() {
ts := httptest.NewServer(fileServer)
// Configure the remote
fs.LoadConfig()
config.LoadConfig()
// fs.Config.LogLevel = fs.LogLevelDebug
// fs.Config.DumpHeaders = true
// fs.Config.DumpBodies = true
fs.ConfigFileSet(remoteName, "type", "http")
fs.ConfigFileSet(remoteName, "url", ts.URL)
config.FileSet(remoteName, "type", "http")
config.FileSet(remoteName, "url", ts.URL)
// return a function to tidy up
return ts.Close

12
backend/hubic/hubic.go

@ -15,9 +15,9 @@ import (
"github.com/ncw/rclone/backend/swift"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/ncw/rclone/oauthutil"
"github.com/ncw/rclone/swift"
swiftLib "github.com/ncw/swift"
"github.com/pkg/errors"
"golang.org/x/oauth2"
@ -40,7 +40,7 @@ var (
TokenURL: "https://api.hubic.com/oauth/token/",
},
ClientID: rcloneClientID,
ClientSecret: fs.MustReveal(rcloneEncryptedClientSecret),
ClientSecret: config.MustReveal(rcloneEncryptedClientSecret),
RedirectURL: oauthutil.RedirectLocalhostURL,
}
)
@ -58,10 +58,10 @@ func init() {
}
},
Options: []fs.Option{{
Name: fs.ConfigClientID,
Name: config.ConfigClientID,
Help: "Hubic Client Id - leave blank normally.",
}, {
Name: fs.ConfigClientSecret,
Name: config.ConfigClientSecret,
Help: "Hubic Client Secret - leave blank normally.",
}},
})
@ -159,7 +159,7 @@ func NewFs(name, root string) (fs.Fs, error) {
Auth: newAuth(f),
ConnectTimeout: 10 * fs.Config.ConnectTimeout, // Use the timeouts in the transport
Timeout: 10 * fs.Config.Timeout, // Use the timeouts in the transport
Transport: fs.Config.Transport(),
Transport: fshttp.NewTransport(fs.Config),
}
err = c.Authenticate()
if err != nil {

33
backend/local/local.go

@ -16,14 +16,17 @@ import (
"unicode/utf8"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/hash"
"github.com/pkg/errors"
"google.golang.org/appengine/log"
)
var (
followSymlinks = fs.BoolP("copy-links", "L", false, "Follow symlinks and copy the pointed to item.")
skipSymlinks = fs.BoolP("skip-links", "", false, "Don't warn about skipped symlinks.")
noUTFNorm = fs.BoolP("local-no-unicode-normalization", "", false, "Don't apply unicode normalization to paths and filenames")
followSymlinks = flags.BoolP("copy-links", "L", false, "Follow symlinks and copy the pointed to item.")
skipSymlinks = flags.BoolP("skip-links", "", false, "Don't warn about skipped symlinks.")
noUTFNorm = flags.BoolP("local-no-unicode-normalization", "", false, "Don't apply unicode normalization to paths and filenames")
)
// Constants
@ -72,7 +75,7 @@ type Object struct {
size int64 // file metadata - always present
mode os.FileMode
modTime time.Time
hashes map[fs.HashType]string // Hashes
hashes map[hash.Type]string // Hashes
}
// ------------------------------------------------------------
@ -85,7 +88,7 @@ func NewFs(name, root string) (fs.Fs, error) {
log.Errorf(nil, "The --local-no-unicode-normalization flag is deprecated and will be removed")
}
nounc := fs.ConfigFileGet(name, "nounc")
nounc := config.FileGet(name, "nounc")
f := &Fs{
name: name,
warned: make(map[string]struct{}),
@ -532,8 +535,8 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.SupportedHashes
func (f *Fs) Hashes() hash.Set {
return hash.SupportedHashes
}
// ------------------------------------------------------------
@ -557,7 +560,7 @@ func (o *Object) Remote() string {
}
// Hash returns the requested hash of a file as a lowercase hex string
func (o *Object) Hash(r fs.HashType) (string, error) {
func (o *Object) Hash(r hash.Type) (string, error) {
// Check that the underlying file hasn't changed
oldtime := o.modTime
oldsize := o.size
@ -571,12 +574,12 @@ func (o *Object) Hash(r fs.HashType) (string, error) {
}
if o.hashes == nil {
o.hashes = make(map[fs.HashType]string)
o.hashes = make(map[hash.Type]string)
in, err := os.Open(o.path)
if err != nil {
return "", errors.Wrap(err, "hash: failed to open")
}
o.hashes, err = fs.HashStream(in)
o.hashes, err = hash.Stream(in)
closeErr := in.Close()
if err != nil {
return "", errors.Wrap(err, "hash: failed to read")
@ -643,7 +646,7 @@ func (o *Object) Storable() bool {
type localOpenFile struct {
o *Object // object that is open
in io.ReadCloser // handle we are wrapping
hash *fs.MultiHasher // currently accumulating hashes
hash *hash.MultiHasher // currently accumulating hashes
}
// Read bytes from the object - see io.Reader
@ -670,7 +673,7 @@ func (file *localOpenFile) Close() (err error) {
// Open an object for read
func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
var offset int64
hashes := fs.SupportedHashes
hashes := hash.SupportedHashes
for _, option := range options {
switch x := option.(type) {
case *fs.SeekOption:
@ -694,7 +697,7 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
// don't attempt to make checksums
return fd, err
}
hash, err := fs.NewMultiHasherTypes(hashes)
hash, err := hash.NewMultiHasherTypes(hashes)
if err != nil {
return nil, err
}
@ -715,7 +718,7 @@ func (o *Object) mkdirAll() error {
// Update the object from in with modTime and size
func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
hashes := fs.SupportedHashes
hashes := hash.SupportedHashes
for _, option := range options {
switch x := option.(type) {
case *fs.HashesOption:
@ -734,7 +737,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
}
// Calculate the hash of the object we are reading as we go along
hash, err := fs.NewMultiHasherTypes(hashes)
hash, err := hash.NewMultiHasherTypes(hashes)
if err != nil {
return err
}

3
backend/local/read_device_unix.go

@ -9,10 +9,11 @@ import (
"syscall"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config/flags"
)
var (
oneFileSystem = fs.BoolP("one-file-system", "x", false, "Don't cross filesystem boundaries.")
oneFileSystem = flags.BoolP("one-file-system", "x", false, "Don't cross filesystem boundaries.")
)
// readDevice turns a valid os.FileInfo into a device number,

50
backend/onedrive/onedrive.go

@ -15,16 +15,16 @@ import (
"time"
"github.com/ncw/rclone/backend/onedrive/api"
"github.com/ncw/rclone/dircache"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/dircache"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/ncw/rclone/lib/pacer"
"github.com/ncw/rclone/lib/readers"
"github.com/ncw/rclone/lib/rest"
"github.com/ncw/rclone/oauthutil"
"github.com/ncw/rclone/onedrive/api"
"github.com/ncw/rclone/pacer"
"github.com/ncw/rclone/rest"
"github.com/pkg/errors"
"golang.org/x/oauth2"
)
@ -56,7 +56,7 @@ var (
TokenURL: "https://login.live.com/oauth20_token.srf",
},
ClientID: rclonePersonalClientID,
ClientSecret: fs.MustReveal(rclonePersonalEncryptedClientSecret),
ClientSecret: config.MustReveal(rclonePersonalEncryptedClientSecret),
RedirectURL: oauthutil.RedirectLocalhostURL,
}
@ -67,7 +67,7 @@ var (
TokenURL: "https://login.microsoftonline.com/common/oauth2/token",
},
ClientID: rcloneBusinessClientID,
ClientSecret: fs.MustReveal(rcloneBusinessEncryptedClientSecret),
ClientSecret: config.MustReveal(rcloneBusinessEncryptedClientSecret),
RedirectURL: oauthutil.RedirectLocalhostURL,
}
oauthBusinessResource = oauth2.SetAuthURLParam("resource", discoveryServiceURL)
@ -87,7 +87,7 @@ func init() {
fmt.Printf("Choose OneDrive account type?\n")
fmt.Printf(" * Say b for a OneDrive business account\n")
fmt.Printf(" * Say p for a personal OneDrive account\n")
isPersonal := fs.Command([]string{"bBusiness", "pPersonal"}) == 'p'
isPersonal := config.Command([]string{"bBusiness", "pPersonal"}) == 'p'
if isPersonal {
// for personal accounts we don't safe a field about the account
@ -103,7 +103,7 @@ func init() {
}
// Are we running headless?
if fs.ConfigFileGet(name, fs.ConfigAutomatic) != "" {
if config.FileGet(name, config.ConfigAutomatic) != "" {
// Yes, okay we are done
return
}
@ -159,10 +159,10 @@ func init() {
} else if len(resourcesID) == 1 {
foundService = resourcesID[0]
} else {
foundService = fs.Choose("Choose resource URL", resourcesID, resourcesURL, false)
foundService = config.Choose("Choose resource URL", resourcesID, resourcesURL, false)
}
fs.ConfigFileSet(name, configResourceURL, foundService)
config.FileSet(name, configResourceURL, foundService)
oauthBusinessResource = oauth2.SetAuthURLParam("resource", foundService)
// get the token from the inital config
@ -218,16 +218,16 @@ func init() {
}
},
Options: []fs.Option{{
Name: fs.ConfigClientID,
Name: config.ConfigClientID,
Help: "Microsoft App Client Id - leave blank normally.",
}, {
Name: fs.ConfigClientSecret,
Name: config.ConfigClientSecret,
Help: "Microsoft App Client Secret - leave blank normally.",
}},
})
fs.VarP(&chunkSize, "onedrive-chunk-size", "", "Above this size files will be chunked - must be multiple of 320k.")
fs.VarP(&uploadCutoff, "onedrive-upload-cutoff", "", "Cutoff for switching to chunked upload - must be <= 100MB")
flags.VarP(&chunkSize, "onedrive-chunk-size", "", "Above this size files will be chunked - must be multiple of 320k.")
flags.VarP(&uploadCutoff, "onedrive-upload-cutoff", "", "Cutoff for switching to chunked upload - must be <= 100MB")
}
// Fs represents a remote one drive
@ -306,7 +306,7 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
authRety = true
fs.Debugf(nil, "Should retry: %v", err)
}
return authRety || fs.ShouldRetry(err) || fs.ShouldRetryHTTP(resp, retryErrorCodes), err
return authRety || fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}
// readMetaDataForPath reads the metadata from the path
@ -339,7 +339,7 @@ func errorHandler(resp *http.Response) error {
// NewFs constructs an Fs from the path, container:path
func NewFs(name, root string) (fs.Fs, error) {
// get the resource URL from the config file0
resourceURL := fs.ConfigFileGet(name, configResourceURL, "")
resourceURL := config.FileGet(name, configResourceURL, "")
// if we have a resource URL it's a business account otherwise a personal one
var rootURL string
var oauthConfig *oauth2.Config
@ -743,10 +743,10 @@ func (f *Fs) waitForJob(location string, o *Object) error {
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
if err != nil {
return fs.ShouldRetry(err), err
return fserrors.ShouldRetry(err), err
}
body, err = rest.ReadBody(resp)
return fs.ShouldRetry(err), err
return fserrors.ShouldRetry(err), err
})
if err != nil {
return err
@ -915,8 +915,8 @@ func (f *Fs) DirCacheFlush() {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashSHA1)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashSHA1)
}
// ------------------------------------------------------------
@ -945,9 +945,9 @@ func (o *Object) srvPath() string {
}
// Hash returns the SHA-1 of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashSHA1 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashSHA1 {
return "", hash.ErrHashUnsupported
}
return o.sha1, nil
}
@ -1161,7 +1161,7 @@ func (o *Object) uploadMultipart(in io.Reader, size int64) (err error) {
if remaining < n {
n = remaining
}
seg := fs.NewRepeatableReader(io.LimitReader(in, n))
seg := readers.NewRepeatableReader(io.LimitReader(in, n))
fs.Debugf(o, "Uploading segment %d/%d size %d", position, size, n)
err = o.uploadFragment(uploadURL, position, size, seg, n)
if err != nil {

31
backend/pcloud/pcloud.go

@ -22,16 +22,15 @@ import (
"time"
"github.com/ncw/rclone/backend/pcloud/api"
"github.com/ncw/rclone/dircache"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/dircache"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/ncw/rclone/lib/pacer"
"github.com/ncw/rclone/lib/rest"
"github.com/ncw/rclone/oauthutil"
"github.com/ncw/rclone/pacer"
"github.com/ncw/rclone/pcloud/api"
"github.com/ncw/rclone/rest"
"github.com/pkg/errors"
"golang.org/x/oauth2"
)
@ -56,7 +55,7 @@ var (
TokenURL: "https://api.pcloud.com/oauth2_token",
},
ClientID: rcloneClientID,
ClientSecret: fs.MustReveal(rcloneEncryptedClientSecret),
ClientSecret: config.MustReveal(rcloneEncryptedClientSecret),
RedirectURL: oauthutil.RedirectLocalhostURL,
}
uploadCutoff = fs.SizeSuffix(50 * 1024 * 1024)
@ -75,14 +74,14 @@ func init() {
}
},
Options: []fs.Option{{
Name: fs.ConfigClientID,
Name: config.ConfigClientID,
Help: "Pcloud App Client Id - leave blank normally.",
}, {
Name: fs.ConfigClientSecret,
Name: config.ConfigClientSecret,
Help: "Pcloud App Client Secret - leave blank normally.",
}},
})
fs.VarP(&uploadCutoff, "pcloud-upload-cutoff", "", "Cutoff for switching to multipart upload")
flags.VarP(&uploadCutoff, "pcloud-upload-cutoff", "", "Cutoff for switching to multipart upload")
}
// Fs represents a remote pcloud
@ -174,7 +173,7 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
doRetry = true
fs.Debugf(nil, "Should retry: %v", err)
}
return doRetry || fs.ShouldRetry(err) || fs.ShouldRetryHTTP(resp, retryErrorCodes), err
return doRetry || fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}
// substitute reserved characters for pcloud
@ -812,8 +811,8 @@ func (f *Fs) DirCacheFlush() {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5 | fs.HashSHA1)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5 | hash.HashSHA1)
}
// ------------------------------------------------------------
@ -859,9 +858,9 @@ func (o *Object) getHashes() (err error) {
}
// Hash returns the SHA-1 of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 && t != fs.HashSHA1 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 && t != hash.HashSHA1 {
return "", hash.ErrHashUnsupported
}
if o.md5 == "" && o.sha1 == "" {
err := o.getHashes()
@ -869,7 +868,7 @@ func (o *Object) Hash(t fs.HashType) (string, error) {
return "", errors.Wrap(err, "failed to get hash")
}
}
if t == fs.HashMD5 {
if t == hash.HashMD5 {
return o.md5, nil
}
return o.sha1, nil

36
backend/qingstor/qingstor.go

@ -17,8 +17,12 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/walk"
"github.com/pkg/errors"
"github.com/yunify/qingstor-sdk-go/config"
qsConfig "github.com/yunify/qingstor-sdk-go/config"
qsErr "github.com/yunify/qingstor-sdk-go/request/errors"
qs "github.com/yunify/qingstor-sdk-go/service"
)
@ -162,11 +166,11 @@ func qsParseEndpoint(endpoint string) (protocol, host, port string, err error) {
// qsConnection makes a connection to qingstor
func qsServiceConnection(name string) (*qs.Service, error) {
accessKeyID := fs.ConfigFileGet(name, "access_key_id")
secretAccessKey := fs.ConfigFileGet(name, "secret_access_key")
accessKeyID := config.FileGet(name, "access_key_id")
secretAccessKey := config.FileGet(name, "secret_access_key")
switch {
case fs.ConfigFileGetBool(name, "env_auth", false):
case config.FileGetBool(name, "env_auth", false):
// No need for empty checks if "env_auth" is true
case accessKeyID == "" && secretAccessKey == "":
// if no access key/secret and iam is explicitly disabled then fall back to anon interaction
@ -180,7 +184,7 @@ func qsServiceConnection(name string) (*qs.Service, error) {
host := "qingstor.com"
port := 443
endpoint := fs.ConfigFileGet(name, "endpoint", "")
endpoint := config.FileGet(name, "endpoint", "")
if endpoint != "" {
_protocol, _host, _port, err := qsParseEndpoint(endpoint)
@ -201,19 +205,19 @@ func qsServiceConnection(name string) (*qs.Service, error) {
}
connectionRetries := 3
retries := fs.ConfigFileGet(name, "connection_retries", "")
retries := config.FileGet(name, "connection_retries", "")
if retries != "" {
connectionRetries, _ = strconv.Atoi(retries)
}
cf, err := config.NewDefault()
cf, err := qsConfig.NewDefault()
cf.AccessKeyID = accessKeyID
cf.SecretAccessKey = secretAccessKey
cf.Protocol = protocol
cf.Host = host
cf.Port = port
cf.ConnectionRetries = connectionRetries
cf.Connection = fs.Config.Client()
cf.Connection = fshttp.NewClient(fs.Config)
svc, _ := qs.Init(cf)
@ -231,7 +235,7 @@ func NewFs(name, root string) (fs.Fs, error) {
return nil, err
}
zone := fs.ConfigFileGet(name, "zone")
zone := config.FileGet(name, "zone")
if zone == "" {
zone = "pek3a"
}
@ -302,9 +306,9 @@ func (f *Fs) Precision() time.Duration {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5)
//return fs.HashSet(fs.HashNone)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5)
//return hash.HashSet(hash.HashNone)
}
// Features returns the optional features of this Fs
@ -591,7 +595,7 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
if f.bucket == "" {
return fs.ErrorListBucketRequired
}
list := fs.NewListRHelper(callback)
list := walk.NewListRHelper(callback)
err = f.list(dir, true, func(remote string, object *qs.KeyType, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory)
if err != nil {
@ -925,9 +929,9 @@ var matchMd5 = regexp.MustCompile(`^[0-9a-f]{32}$`)
// Hash returns the selected checksum of the file
// If no checksum is available it returns ""
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 {
return "", hash.ErrHashUnsupported
}
etag := strings.Trim(strings.ToLower(o.etag), `"`)
// Check the etag is a valid md5sum

45
backend/s3/s3.go

@ -37,6 +37,11 @@ import (
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/walk"
"github.com/ncw/rclone/lib/rest"
"github.com/ncw/swift"
"github.com/pkg/errors"
@ -233,8 +238,8 @@ const (
// Globals
var (
// Flags
s3ACL = fs.StringP("s3-acl", "", "", "Canned ACL used when creating buckets and/or storing objects in S3")
s3StorageClass = fs.StringP("s3-storage-class", "", "", "Storage class to use when uploading S3 objects (STANDARD|REDUCED_REDUNDANCY|STANDARD_IA)")
s3ACL = flags.StringP("s3-acl", "", "", "Canned ACL used when creating buckets and/or storing objects in S3")
s3StorageClass = flags.StringP("s3-storage-class", "", "", "Storage class to use when uploading S3 objects (STANDARD|REDUCED_REDUNDANCY|STANDARD_IA)")
)
// Fs represents a remote s3 server
@ -316,9 +321,9 @@ func s3ParsePath(path string) (bucket, directory string, err error) {
func s3Connection(name string) (*s3.S3, *session.Session, error) {
// Make the auth
v := credentials.Value{
AccessKeyID: fs.ConfigFileGet(name, "access_key_id"),
SecretAccessKey: fs.ConfigFileGet(name, "secret_access_key"),
SessionToken: fs.ConfigFileGet(name, "session_token"),
AccessKeyID: config.FileGet(name, "access_key_id"),
SecretAccessKey: config.FileGet(name, "secret_access_key"),
SessionToken: config.FileGet(name, "session_token"),
}
lowTimeoutClient := &http.Client{Timeout: 1 * time.Second} // low timeout to ec2 metadata service
@ -348,7 +353,7 @@ func s3Connection(name string) (*s3.S3, *session.Session, error) {
cred := credentials.NewChainCredentials(providers)
switch {
case fs.ConfigFileGetBool(name, "env_auth", false):
case config.FileGetBool(name, "env_auth", false):
// No need for empty checks if "env_auth" is true
case v.AccessKeyID == "" && v.SecretAccessKey == "":
// if no access key/secret and iam is explicitly disabled then fall back to anon interaction
@ -359,8 +364,8 @@ func s3Connection(name string) (*s3.S3, *session.Session, error) {
return nil, nil, errors.New("secret_access_key not found")
}
endpoint := fs.ConfigFileGet(name, "endpoint")
region := fs.ConfigFileGet(name, "region")
endpoint := config.FileGet(name, "endpoint")
region := config.FileGet(name, "region")
if region == "" && endpoint == "" {
endpoint = "https://s3.amazonaws.com/"
}
@ -372,7 +377,7 @@ func s3Connection(name string) (*s3.S3, *session.Session, error) {
WithMaxRetries(maxRetries).
WithCredentials(cred).
WithEndpoint(endpoint).
WithHTTPClient(fs.Config.Client()).
WithHTTPClient(fshttp.NewClient(fs.Config)).
WithS3ForcePathStyle(true)
// awsConfig.WithLogLevel(aws.LogDebugWithSigning)
ses := session.New()
@ -408,11 +413,11 @@ func NewFs(name, root string) (fs.Fs, error) {
c: c,
bucket: bucket,
ses: ses,
acl: fs.ConfigFileGet(name, "acl"),
acl: config.FileGet(name, "acl"),
root: directory,
locationConstraint: fs.ConfigFileGet(name, "location_constraint"),
sse: fs.ConfigFileGet(name, "server_side_encryption"),
storageClass: fs.ConfigFileGet(name, "storage_class"),
locationConstraint: config.FileGet(name, "location_constraint"),
sse: config.FileGet(name, "server_side_encryption"),
storageClass: config.FileGet(name, "storage_class"),
}
f.features = (&fs.Features{
ReadMimeType: true,
@ -657,7 +662,7 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
if f.bucket == "" {
return fs.ErrorListBucketRequired
}
list := fs.NewListRHelper(callback)
list := walk.NewListRHelper(callback)
err = f.list(dir, true, func(remote string, object *s3.Object, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory)
if err != nil {
@ -804,8 +809,8 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5)
}
// ------------------------------------------------------------
@ -831,9 +836,9 @@ func (o *Object) Remote() string {
var matchMd5 = regexp.MustCompile(`^[0-9a-f]{32}$`)
// Hash returns the Md5sum of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 {
return "", hash.ErrHashUnsupported
}
hash := strings.Trim(strings.ToLower(o.etag), `"`)
// Check the etag is a valid md5sum
@ -1027,7 +1032,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
}
if size > uploader.PartSize {
hash, err := src.Hash(fs.HashMD5)
hash, err := src.Hash(hash.HashMD5)
if err == nil && matchMd5.MatchString(hash) {
hashBytes, err := hex.DecodeString(hash)

81
backend/sftp/sftp.go

@ -16,6 +16,9 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/pkg/errors"
"github.com/pkg/sftp"
"github.com/xanzy/ssh-agent"
@ -94,7 +97,7 @@ type Fs struct {
port string
url string
mkdirLock *stringLock
cachedHashes *fs.HashSet
cachedHashes *hash.Set
poolMu sync.Mutex
pool []*conn
connLimit *rate.Limiter // for limiting number of connections per second
@ -134,13 +137,13 @@ func readCurrentUser() (userName string) {
// Dial starts a client connection to the given SSH server. It is a
// convenience function that connects to the given network address,
// initiates the SSH handshake, and then sets up a Client.
func Dial(network, addr string, config *ssh.ClientConfig) (*ssh.Client, error) {
dialer := fs.Config.NewDialer()
func Dial(network, addr string, sshConfig *ssh.ClientConfig) (*ssh.Client, error) {
dialer := fshttp.NewDialer(fs.Config)
conn, err := dialer.Dial(network, addr)
if err != nil {
return nil, err
}
c, chans, reqs, err := ssh.NewClientConn(conn, addr, config)
c, chans, reqs, err := ssh.NewClientConn(conn, addr, sshConfig)
if err != nil {
return nil, err
}
@ -263,19 +266,19 @@ func (f *Fs) putSftpConnection(pc **conn, err error) {
// NewFs creates a new Fs object from the name and root. It connects to
// the host specified in the config file.
func NewFs(name, root string) (fs.Fs, error) {
user := fs.ConfigFileGet(name, "user")
host := fs.ConfigFileGet(name, "host")
port := fs.ConfigFileGet(name, "port")
pass := fs.ConfigFileGet(name, "pass")
keyFile := fs.ConfigFileGet(name, "key_file")
insecureCipher := fs.ConfigFileGetBool(name, "use_insecure_cipher")
user := config.FileGet(name, "user")
host := config.FileGet(name, "host")
port := config.FileGet(name, "port")
pass := config.FileGet(name, "pass")
keyFile := config.FileGet(name, "key_file")
insecureCipher := config.FileGetBool(name, "use_insecure_cipher")
if user == "" {
user = currentUser
}
if port == "" {
port = "22"
}
config := &ssh.ClientConfig{
sshConfig := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
@ -283,8 +286,8 @@ func NewFs(name, root string) (fs.Fs, error) {
}
if insecureCipher {
config.Config.SetDefaults()
config.Config.Ciphers = append(config.Config.Ciphers, "aes128-cbc")
sshConfig.Config.SetDefaults()
sshConfig.Config.Ciphers = append(sshConfig.Config.Ciphers, "aes128-cbc")
}
// Add ssh agent-auth if no password or file specified
@ -297,7 +300,7 @@ func NewFs(name, root string) (fs.Fs, error) {
if err != nil {
return nil, errors.Wrap(err, "couldn't read ssh agent signers")
}
config.Auth = append(config.Auth, ssh.PublicKeys(signers...))
sshConfig.Auth = append(sshConfig.Auth, ssh.PublicKeys(signers...))
}
// Load key file if specified
@ -310,22 +313,22 @@ func NewFs(name, root string) (fs.Fs, error) {
if err != nil {
return nil, errors.Wrap(err, "failed to parse private key file")
}
config.Auth = append(config.Auth, ssh.PublicKeys(signer))
sshConfig.Auth = append(sshConfig.Auth, ssh.PublicKeys(signer))
}
// Auth from password if specified
if pass != "" {
clearpass, err := fs.Reveal(pass)
clearpass, err := config.Reveal(pass)
if err != nil {
return nil, err
}
config.Auth = append(config.Auth, ssh.Password(clearpass))
sshConfig.Auth = append(sshConfig.Auth, ssh.Password(clearpass))
}
f := &Fs{
name: name,
root: root,
config: config,
config: sshConfig,
host: host,
port: port,
url: "sftp://" + user + "@" + host + ":" + port + "/" + root,
@ -631,25 +634,25 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
}
// Hashes returns the supported hash types of the filesystem
func (f *Fs) Hashes() fs.HashSet {
func (f *Fs) Hashes() hash.Set {
if f.cachedHashes != nil {
return *f.cachedHashes
}
hashcheckDisabled := fs.ConfigFileGetBool(f.name, "disable_hashcheck")
hashcheckDisabled := config.FileGetBool(f.name, "disable_hashcheck")
if hashcheckDisabled {
return fs.HashSet(fs.HashNone)
return hash.Set(hash.HashNone)
}
c, err := f.getSftpConnection()
if err != nil {
fs.Errorf(f, "Couldn't get SSH connection to figure out Hashes: %v", err)
return fs.HashSet(fs.HashNone)
return hash.Set(hash.HashNone)
}
defer f.putSftpConnection(&c, err)
session, err := c.sshClient.NewSession()
if err != nil {
return fs.HashSet(fs.HashNone)
return hash.Set(hash.HashNone)
}
sha1Output, _ := session.Output("echo 'abc' | sha1sum")
expectedSha1 := "03cfd743661f07975fa2f1220c5194cbaff48451"
@ -657,7 +660,7 @@ func (f *Fs) Hashes() fs.HashSet {
session, err = c.sshClient.NewSession()
if err != nil {
return fs.HashSet(fs.HashNone)
return hash.Set(hash.HashNone)
}
md5Output, _ := session.Output("echo 'abc' | md5sum")
expectedMd5 := "0bee89b07a248e27c83fc3d5951213c1"
@ -666,15 +669,15 @@ func (f *Fs) Hashes() fs.HashSet {
sha1Works := parseHash(sha1Output) == expectedSha1
md5Works := parseHash(md5Output) == expectedMd5
set := fs.NewHashSet()
set := hash.NewHashSet()
if !sha1Works && !md5Works {
set.Add(fs.HashNone)
set.Add(hash.HashNone)
}
if sha1Works {
set.Add(fs.HashSHA1)
set.Add(hash.HashSHA1)
}
if md5Works {
set.Add(fs.HashMD5)
set.Add(hash.HashMD5)
}
_ = session.Close()
@ -702,10 +705,10 @@ func (o *Object) Remote() string {
// Hash returns the selected checksum of the file
// If no checksum is available it returns ""
func (o *Object) Hash(r fs.HashType) (string, error) {
if r == fs.HashMD5 && o.md5sum != nil {
func (o *Object) Hash(r hash.Type) (string, error) {
if r == hash.HashMD5 && o.md5sum != nil {
return *o.md5sum, nil
} else if r == fs.HashSHA1 && o.sha1sum != nil {
} else if r == hash.HashSHA1 && o.sha1sum != nil {
return *o.sha1sum, nil
}
@ -717,29 +720,29 @@ func (o *Object) Hash(r fs.HashType) (string, error) {
o.fs.putSftpConnection(&c, err)
if err != nil {
o.fs.cachedHashes = nil // Something has changed on the remote system
return "", fs.ErrHashUnsupported
return "", hash.ErrHashUnsupported
}
err = fs.ErrHashUnsupported
err = hash.ErrHashUnsupported
var outputBytes []byte
escapedPath := shellEscape(o.path())
if r == fs.HashMD5 {
if r == hash.HashMD5 {
outputBytes, err = session.Output("md5sum " + escapedPath)
} else if r == fs.HashSHA1 {
} else if r == hash.HashSHA1 {
outputBytes, err = session.Output("sha1sum " + escapedPath)
}
if err != nil {
o.fs.cachedHashes = nil // Something has changed on the remote system
_ = session.Close()
return "", fs.ErrHashUnsupported
return "", hash.ErrHashUnsupported
}
_ = session.Close()
str := parseHash(outputBytes)
if r == fs.HashMD5 {
if r == hash.HashMD5 {
o.md5sum = &str
} else if r == fs.HashSHA1 {
} else if r == hash.HashSHA1 {
o.sha1sum = &str
}
return str, nil
@ -812,7 +815,7 @@ func (o *Object) SetModTime(modTime time.Time) error {
if err != nil {
return errors.Wrap(err, "SetModTime")
}
if fs.ConfigFileGetBool(o.fs.name, "set_modtime", true) {
if config.FileGetBool(o.fs.name, "set_modtime", true) {
err = c.sftpClient.Chtimes(o.path(), modTime, modTime)
o.fs.putSftpConnection(&c, err)
if err != nil {

55
backend/swift/swift.go

@ -14,6 +14,13 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/operations"
"github.com/ncw/rclone/fs/walk"
"github.com/ncw/swift"
"github.com/pkg/errors"
)
@ -118,7 +125,7 @@ func init() {
},
},
})
fs.VarP(&chunkSize, "swift-chunk-size", "", "Above this size files will be chunked into a _segments container.")
flags.VarP(&chunkSize, "swift-chunk-size", "", "Above this size files will be chunked into a _segments container.")
}
// Fs represents a remote swift server
@ -191,24 +198,24 @@ func parsePath(path string) (container, directory string, err error) {
func swiftConnection(name string) (*swift.Connection, error) {
c := &swift.Connection{
// Keep these in the same order as the Config for ease of checking
UserName: fs.ConfigFileGet(name, "user"),
ApiKey: fs.ConfigFileGet(name, "key"),
AuthUrl: fs.ConfigFileGet(name, "auth"),
UserId: fs.ConfigFileGet(name, "user_id"),
Domain: fs.ConfigFileGet(name, "domain"),
Tenant: fs.ConfigFileGet(name, "tenant"),
TenantId: fs.ConfigFileGet(name, "tenant_id"),
TenantDomain: fs.ConfigFileGet(name, "tenant_domain"),
Region: fs.ConfigFileGet(name, "region"),
StorageUrl: fs.ConfigFileGet(name, "storage_url"),
AuthToken: fs.ConfigFileGet(name, "auth_token"),
AuthVersion: fs.ConfigFileGetInt(name, "auth_version", 0),
EndpointType: swift.EndpointType(fs.ConfigFileGet(name, "endpoint_type", "public")),
UserName: config.FileGet(name, "user"),
ApiKey: config.FileGet(name, "key"),
AuthUrl: config.FileGet(name, "auth"),
UserId: config.FileGet(name, "user_id"),
Domain: config.FileGet(name, "domain"),
Tenant: config.FileGet(name, "tenant"),
TenantId: config.FileGet(name, "tenant_id"),
TenantDomain: config.FileGet(name, "tenant_domain"),
Region: config.FileGet(name, "region"),
StorageUrl: config.FileGet(name, "storage_url"),
AuthToken: config.FileGet(name, "auth_token"),
AuthVersion: config.FileGetInt(name, "auth_version", 0),
EndpointType: swift.EndpointType(config.FileGet(name, "endpoint_type", "public")),
ConnectTimeout: 10 * fs.Config.ConnectTimeout, // Use the timeouts in the transport
Timeout: 10 * fs.Config.Timeout, // Use the timeouts in the transport
Transport: fs.Config.Transport(),
Transport: fshttp.NewTransport(fs.Config),
}
if fs.ConfigFileGetBool(name, "env_auth", false) {
if config.FileGetBool(name, "env_auth", false) {
err := c.ApplyEnvironment()
if err != nil {
return nil, errors.Wrap(err, "failed to read environment variables")
@ -466,7 +473,7 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
if f.container == "" {
return errors.New("container needed for recursive list")
}
list := fs.NewListRHelper(callback)
list := walk.NewListRHelper(callback)
err = f.list(dir, true, func(entry fs.DirEntry) error {
return list.Add(entry)
})
@ -549,7 +556,7 @@ func (f *Fs) Purge() error {
toBeDeleted := make(chan fs.Object, fs.Config.Transfers)
delErr := make(chan error, 1)
go func() {
delErr <- fs.DeleteFiles(toBeDeleted)
delErr <- operations.DeleteFiles(toBeDeleted)
}()
err := f.list("", true, func(entry fs.DirEntry) error {
if o, ok := entry.(*Object); ok {
@ -596,8 +603,8 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5)
}
// ------------------------------------------------------------
@ -621,9 +628,9 @@ func (o *Object) Remote() string {
}
// Hash returns the Md5sum of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 {
return "", hash.ErrHashUnsupported
}
isDynamicLargeObject, err := o.isDynamicLargeObject()
if err != nil {
@ -855,7 +862,7 @@ func (o *Object) updateChunks(in0 io.Reader, headers swift.Headers, size int64,
// The new object may have been created if an error is returned
func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
if o.fs.container == "" {
return fs.FatalError(errors.New("container name needed in remote"))
return fserrors.FatalError(errors.New("container name needed in remote"))
}
err := o.fs.Mkdir("")
if err != nil {

31
backend/webdav/webdav.go

@ -30,11 +30,12 @@ import (
"github.com/ncw/rclone/backend/webdav/api"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/pacer"
"github.com/ncw/rclone/lib/rest"
"github.com/ncw/rclone/pacer"
"github.com/ncw/rclone/rest"
"github.com/ncw/rclone/webdav/api"
"github.com/pkg/errors"
)
@ -159,7 +160,7 @@ var retryErrorCodes = []int{
// shouldRetry returns a boolean as to whether this resp and err
// deserve to be retried. It returns the err as a convenience
func shouldRetry(resp *http.Response, err error) (bool, error) {
return fs.ShouldRetry(err) || fs.ShouldRetryHTTP(resp, retryErrorCodes), err
return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}
// itemIsDir returns true if the item is a directory
@ -250,21 +251,21 @@ func (o *Object) filePath() string {
// NewFs constructs an Fs from the path, container:path
func NewFs(name, root string) (fs.Fs, error) {
endpoint := fs.ConfigFileGet(name, "url")
endpoint := config.FileGet(name, "url")
if !strings.HasSuffix(endpoint, "/") {
endpoint += "/"
}
user := fs.ConfigFileGet(name, "user")
pass := fs.ConfigFileGet(name, "pass")
user := config.FileGet(name, "user")
pass := config.FileGet(name, "pass")
if pass != "" {
var err error
pass, err = fs.Reveal(pass)
pass, err = config.Reveal(pass)
if err != nil {
return nil, errors.Wrap(err, "couldn't decrypt password")
}
}
vendor := fs.ConfigFileGet(name, "vendor")
vendor := config.FileGet(name, "vendor")
// Parse the endpoint
u, err := url.Parse(endpoint)
@ -277,7 +278,7 @@ func NewFs(name, root string) (fs.Fs, error) {
root: root,
endpoint: u,
endpointURL: u.String(),
srv: rest.NewClient(fs.Config.Client()).SetRoot(u.String()).SetUserPass(user, pass),
srv: rest.NewClient(fshttp.NewClient(fs.Config)).SetRoot(u.String()).SetUserPass(user, pass),
pacer: pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant),
user: user,
pass: pass,
@ -765,8 +766,8 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashNone)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashNone)
}
// ------------------------------------------------------------
@ -790,9 +791,9 @@ func (o *Object) Remote() string {
}
// Hash returns the SHA-1 of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashSHA1 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashSHA1 {
return "", hash.ErrHashUnsupported
}
return o.sha1, nil
}

28
backend/yandex/yandex.go

@ -15,9 +15,11 @@ import (
yandex "github.com/ncw/rclone/backend/yandex/api"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/lib/oauthutil"
"github.com/ncw/rclone/oauthutil"
yandex "github.com/ncw/rclone/yandex/api"
"github.com/ncw/rclone/lib/readers"
"github.com/pkg/errors"
"golang.org/x/oauth2"
)
@ -37,7 +39,7 @@ var (
TokenURL: "https://oauth.yandex.com/token", //same as https://oauth.yandex.ru/token
},
ClientID: rcloneClientID,
ClientSecret: fs.MustReveal(rcloneEncryptedClientSecret),
ClientSecret: config.MustReveal(rcloneEncryptedClientSecret),
RedirectURL: oauthutil.RedirectURL,
}
)
@ -55,10 +57,10 @@ func init() {
}
},
Options: []fs.Option{{
Name: fs.ConfigClientID,
Name: config.ConfigClientID,
Help: "Yandex Client Id - leave blank normally.",
}, {
Name: fs.ConfigClientSecret,
Name: config.ConfigClientSecret,
Help: "Yandex Client Secret - leave blank normally.",
}},
})
@ -109,7 +111,7 @@ func (f *Fs) Features() *fs.Features {
// read access token from ConfigFile string
func getAccessToken(name string) (*oauth2.Token, error) {
// Read the token from the config file
tokenConfig := fs.ConfigFileGet(name, "token")
tokenConfig := config.FileGet(name, "token")
//Get access token from config string
decoder := json.NewDecoder(strings.NewReader(tokenConfig))
var result *oauth2.Token
@ -129,7 +131,7 @@ func NewFs(name, root string) (fs.Fs, error) {
}
//create new client
yandexDisk := yandex.NewClient(token.AccessToken, fs.Config.Client())
yandexDisk := yandex.NewClient(token.AccessToken, fshttp.NewClient(fs.Config))
f := &Fs{
name: name,
@ -487,8 +489,8 @@ func (f *Fs) CleanUp() error {
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() fs.HashSet {
return fs.HashSet(fs.HashMD5)
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.HashMD5)
}
// ------------------------------------------------------------
@ -512,9 +514,9 @@ func (o *Object) Remote() string {
}
// Hash returns the Md5sum of an object returning a lowercase hex string
func (o *Object) Hash(t fs.HashType) (string, error) {
if t != fs.HashMD5 {
return "", fs.ErrHashUnsupported
func (o *Object) Hash(t hash.Type) (string, error) {
if t != hash.HashMD5 {
return "", hash.ErrHashUnsupported
}
return o.md5sum, nil
}
@ -578,7 +580,7 @@ func (o *Object) remotePath() string {
//
// The new object may have been created if an error is returned
func (o *Object) Update(in0 io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
in := fs.NewCountingReader(in0)
in := readers.NewCountingReader(in0)
modTime := src.ModTime()
remote := o.remotePath()

4
cmd/authorize/authorize.go

@ -2,7 +2,7 @@ package authorize
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/spf13/cobra"
)
@ -19,6 +19,6 @@ rclone from a machine with a browser - use as instructed by
rclone config.`,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(1, 3, command, args)
fs.Authorize(args)
config.Authorize(args)
},
}

7
cmd/cachestats/cachestats.go

@ -9,6 +9,7 @@ import (
"github.com/ncw/rclone/backend/cache"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
@ -32,9 +33,9 @@ Print cache stats for a remote in JSON format
return
}
if !fs.ConfigFileGetBool(configName, "read_only", false) {
fs.ConfigFileSet(configName, "read_only", "true")
defer fs.ConfigFileDeleteKey(configName, "read_only")
if !config.FileGetBool(configName, "read_only", false) {
config.FileSet(configName, "read_only", "true")
defer config.FileDeleteKey(configName, "read_only")
}
fsrc := cmd.NewFsSrc(args)

4
cmd/cat/cat.go

@ -7,7 +7,7 @@ import (
"os"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -74,7 +74,7 @@ Note that if offset is negative it will count from the end, so
w = ioutil.Discard
}
cmd.Run(false, false, command, func() error {
return fs.Cat(fsrc, w, offset, count)
return operations.Cat(fsrc, w, offset, count)
})
},
}

6
cmd/check/check.go

@ -2,7 +2,7 @@ package check
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -37,9 +37,9 @@ to check all the data.
fsrc, fdst := cmd.NewFsSrcDst(args)
cmd.Run(false, false, command, func() error {
if download {
return fs.CheckDownload(fdst, fsrc)
return operations.CheckDownload(fdst, fsrc)
}
return fs.Check(fdst, fsrc)
return operations.Check(fdst, fsrc)
})
},
}

4
cmd/cleanup/cleanup.go

@ -2,7 +2,7 @@ package cleanup
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -21,7 +21,7 @@ versions. Not supported by all remotes.
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(true, false, command, func() error {
return fs.CleanUp(fsrc)
return operations.CleanUp(fsrc)
})
},
}

97
cmd/cmd.go

@ -21,17 +21,26 @@ import (
"github.com/spf13/pflag"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/accounting"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/configflags"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/fs/filter"
"github.com/ncw/rclone/fs/filter/filterflags"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/fspath"
fslog "github.com/ncw/rclone/fs/log"
)
// Globals
var (
// Flags
cpuProfile = fs.StringP("cpuprofile", "", "", "Write cpu profile to file")
memProfile = fs.StringP("memprofile", "", "", "Write memory profile to file")
statsInterval = fs.DurationP("stats", "", time.Minute*1, "Interval between printing stats, e.g 500ms, 60s, 5m. (0 to disable)")
dataRateUnit = fs.StringP("stats-unit", "", "bytes", "Show data rate in stats as either 'bits' or 'bytes'/s")
cpuProfile = flags.StringP("cpuprofile", "", "", "Write cpu profile to file")
memProfile = flags.StringP("memprofile", "", "", "Write memory profile to file")
statsInterval = flags.DurationP("stats", "", time.Minute*1, "Interval between printing stats, e.g 500ms, 60s, 5m. (0 to disable)")
dataRateUnit = flags.StringP("stats-unit", "", "bytes", "Show data rate in stats as either 'bits' or 'bytes'/s")
version bool
retries = fs.IntP("retries", "", 3, "Retry operations this many times if they fail")
retries = flags.IntP("retries", "", 3, "Retry operations this many times if they fail")
// Errors
errorCommandNotFound = errors.New("command not found")
errorUncategorized = errors.New("uncategorized error")
@ -113,6 +122,10 @@ func runRoot(cmd *cobra.Command, args []string) {
}
func init() {
// Add global flags
configflags.AddFlags(pflag.CommandLine)
filterflags.AddFlags(pflag.CommandLine)
Root.Run = runRoot
Root.Flags().BoolVarP(&version, "version", "V", false, "Print the version number")
cobra.OnInitialize(initConfig)
@ -131,7 +144,7 @@ func ShowVersion() {
func newFsFile(remote string) (fs.Fs, string) {
fsInfo, configName, fsPath, err := fs.ParseRemote(remote)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
log.Fatalf("Failed to create file system for %q: %v", remote, err)
}
f, err := fsInfo.NewFs(configName, fsPath)
@ -141,7 +154,7 @@ func newFsFile(remote string) (fs.Fs, string) {
case nil:
return f, ""
default:
fs.Stats.Error(err)
fs.CountError(err)
log.Fatalf("Failed to create file system for %q: %v", remote, err)
}
return nil, ""
@ -155,15 +168,15 @@ func newFsFile(remote string) (fs.Fs, string) {
func newFsSrc(remote string) (fs.Fs, string) {
f, fileName := newFsFile(remote)
if fileName != "" {
if !fs.Config.Filter.InActive() {
if !filter.Active.InActive() {
err := errors.Errorf("Can't limit to single files when using filters: %v", remote)
fs.Stats.Error(err)
fs.CountError(err)
log.Fatalf(err.Error())
}
// Limit transfers to this file
err := fs.Config.Filter.AddFile(fileName)
err := filter.Active.AddFile(fileName)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
log.Fatalf("Failed to limit to single file %q: %v", remote, err)
}
// Set --no-traverse as only one file
@ -178,7 +191,7 @@ func newFsSrc(remote string) (fs.Fs, string) {
func newFsDst(remote string) fs.Fs {
f, err := fs.NewFs(remote)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
log.Fatalf("Failed to create file system for %q: %v", remote, err)
}
return f
@ -201,7 +214,7 @@ func NewFsSrcDstFiles(args []string) (fsrc fs.Fs, srcFileName string, fdst fs.Fs
// If file exists then srcFileName != "", however if the file
// doesn't exist then we assume it is a directory...
if srcFileName != "" {
dstRemote, dstFileName = fs.RemoteSplit(dstRemote)
dstRemote, dstFileName = fspath.RemoteSplit(dstRemote)
if dstRemote == "" {
dstRemote = "."
}
@ -212,11 +225,11 @@ func NewFsSrcDstFiles(args []string) (fsrc fs.Fs, srcFileName string, fdst fs.Fs
fdst, err := fs.NewFs(dstRemote)
switch err {
case fs.ErrorIsFile:
fs.Stats.Error(err)
fs.CountError(err)
log.Fatalf("Source doesn't exist or is a directory and destination is a file")
case nil:
default:
fs.Stats.Error(err)
fs.CountError(err)
log.Fatalf("Failed to create file system for destination %q: %v", dstRemote, err)
}
fs.CalculateModifyWindow(fdst, fsrc)
@ -241,7 +254,7 @@ func NewFsDst(args []string) fs.Fs {
// NewFsDstFile creates a new dst fs with a destination file name from the arguments
func NewFsDstFile(args []string) (fdst fs.Fs, dstFileName string) {
dstRemote, dstFileName := fs.RemoteSplit(args[0])
dstRemote, dstFileName := fspath.RemoteSplit(args[0])
if dstRemote == "" {
dstRemote = "."
}
@ -274,27 +287,27 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
}
for try := 1; try <= *retries; try++ {
err = f()
if !Retry || (err == nil && !fs.Stats.Errored()) {
if !Retry || (err == nil && !accounting.Stats.Errored()) {
if try > 1 {
fs.Errorf(nil, "Attempt %d/%d succeeded", try, *retries)
}
break
}
if fs.IsFatalError(err) {
if fserrors.IsFatalError(err) {
fs.Errorf(nil, "Fatal error received - not attempting retries")
break
}
if fs.IsNoRetryError(err) {
if fserrors.IsNoRetryError(err) {
fs.Errorf(nil, "Can't retry this error - not attempting retries")
break
}
if err != nil {
fs.Errorf(nil, "Attempt %d/%d failed with %d errors and: %v", try, *retries, fs.Stats.GetErrors(), err)
fs.Errorf(nil, "Attempt %d/%d failed with %d errors and: %v", try, *retries, accounting.Stats.GetErrors(), err)
} else {
fs.Errorf(nil, "Attempt %d/%d failed with %d errors", try, *retries, fs.Stats.GetErrors())
fs.Errorf(nil, "Attempt %d/%d failed with %d errors", try, *retries, accounting.Stats.GetErrors())
}
if try < *retries {
fs.Stats.ResetErrors()
accounting.Stats.ResetErrors()
}
}
if showStats {
@ -304,12 +317,12 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
log.Printf("Failed to %s: %v", cmd.Name(), err)
resolveExitCode(err)
}
if showStats && (fs.Stats.Errored() || *statsInterval > 0) {
fs.Stats.Log()
if showStats && (accounting.Stats.Errored() || *statsInterval > 0) {
accounting.Stats.Log()
}
fs.Debugf(nil, "Go routines at exit %d\n", runtime.NumGoroutine())
if fs.Stats.Errored() {
resolveExitCode(fs.Stats.GetLastError())
if accounting.Stats.Errored() {
resolveExitCode(accounting.Stats.GetLastError())
}
}
@ -339,7 +352,7 @@ func StartStats() chan struct{} {
for {
select {
case <-ticker.C:
fs.Stats.Log()
accounting.Stats.Log()
case <-stopStats:
ticker.Stop()
return
@ -353,10 +366,20 @@ func StartStats() chan struct{} {
// initConfig is run by cobra after initialising the flags
func initConfig() {
// Start the logger
fs.InitLogging()
fslog.InitLogging()
// Finish parsing any command line flags
configflags.SetFlags()
// Load the rest of the config now we have started the logger
fs.LoadConfig()
config.LoadConfig()
// Load filters
var err error
filter.Active, err = filter.NewFilter(&filterflags.Opt)
if err != nil {
log.Fatalf("Failed to load filters: %v", err)
}
// Write the args for debug purposes
fs.Debugf("rclone", "Version %q starting with parameters %q", fs.Version, os.Args)
@ -366,12 +389,12 @@ func initConfig() {
fs.Infof(nil, "Creating CPU profile %q\n", *cpuProfile)
f, err := os.Create(*cpuProfile)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
log.Fatal(err)
}
err = pprof.StartCPUProfile(f)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
log.Fatal(err)
}
AtExit(func() {
@ -385,17 +408,17 @@ func initConfig() {
fs.Infof(nil, "Saving Memory profile %q\n", *memProfile)
f, err := os.Create(*memProfile)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
log.Fatal(err)
}
err = pprof.WriteHeapProfile(f)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
log.Fatal(err)
}
err = f.Close()
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
log.Fatal(err)
}
})
@ -423,11 +446,11 @@ func resolveExitCode(err error) {
os.Exit(exitCodeFileNotFound)
case err == errorUncategorized:
os.Exit(exitCodeUncategorizedError)
case fs.ShouldRetry(err):
case fserrors.ShouldRetry(err):
os.Exit(exitCodeRetryError)
case fs.IsNoRetryError(err):
case fserrors.IsNoRetryError(err):
os.Exit(exitCodeNoRetryError)
case fs.IsFatalError(err):
case fserrors.IsFatalError(err):
os.Exit(exitCodeFatalError)
default:
os.Exit(exitCodeUsageError)

57
cmd/cmount/fs.go

@ -14,6 +14,7 @@ import (
"github.com/billziss-gh/cgofuse/fuse"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/log"
"github.com/ncw/rclone/vfs"
"github.com/ncw/rclone/vfs/vfsflags"
"github.com/pkg/errors"
@ -178,7 +179,7 @@ func (fsys *FS) stat(node vfs.Node, stat *fuse.Stat_t) (errc int) {
// Init is called after the filesystem is ready
func (fsys *FS) Init() {
defer fs.Trace(fsys.f, "")("")
defer log.Trace(fsys.f, "")("")
close(fsys.ready)
}
@ -186,12 +187,12 @@ func (fsys *FS) Init() {
// the file system is terminated the file system may not receive the
// Destroy call).
func (fsys *FS) Destroy() {
defer fs.Trace(fsys.f, "")("")
defer log.Trace(fsys.f, "")("")
}
// Getattr reads the attributes for path
func (fsys *FS) Getattr(path string, stat *fuse.Stat_t, fh uint64) (errc int) {
defer fs.Trace(path, "fh=0x%X", fh)("errc=%v", &errc)
defer log.Trace(path, "fh=0x%X", fh)("errc=%v", &errc)
node, _, errc := fsys.getNode(path, fh)
if errc == 0 {
errc = fsys.stat(node, stat)
@ -201,7 +202,7 @@ func (fsys *FS) Getattr(path string, stat *fuse.Stat_t, fh uint64) (errc int) {
// Opendir opens path as a directory
func (fsys *FS) Opendir(path string) (errc int, fh uint64) {
defer fs.Trace(path, "")("errc=%d, fh=0x%X", &errc, &fh)
defer log.Trace(path, "")("errc=%d, fh=0x%X", &errc, &fh)
handle, err := fsys.VFS.OpenFile(path, os.O_RDONLY, 0777)
if errc != 0 {
return translateError(err), fhUnset
@ -215,7 +216,7 @@ func (fsys *FS) Readdir(dirPath string,
ofst int64,
fh uint64) (errc int) {
itemsRead := -1
defer fs.Trace(dirPath, "ofst=%d, fh=0x%X", ofst, fh)("items=%d, errc=%d", &itemsRead, &errc)
defer log.Trace(dirPath, "ofst=%d, fh=0x%X", ofst, fh)("items=%d, errc=%d", &itemsRead, &errc)
node, errc := fsys.getHandle(fh)
if errc != 0 {
@ -254,13 +255,13 @@ func (fsys *FS) Readdir(dirPath string,
// Releasedir finished reading the directory
func (fsys *FS) Releasedir(path string, fh uint64) (errc int) {
defer fs.Trace(path, "fh=0x%X", fh)("errc=%d", &errc)
defer log.Trace(path, "fh=0x%X", fh)("errc=%d", &errc)
return fsys.closeHandle(fh)
}
// Statfs reads overall stats on the filessystem
func (fsys *FS) Statfs(path string, stat *fuse.Statfs_t) (errc int) {
defer fs.Trace(path, "")("stat=%+v, errc=%d", stat, &errc)
defer log.Trace(path, "")("stat=%+v, errc=%d", stat, &errc)
const blockSize = 4096
fsBlocks := uint64(1 << 50)
if runtime.GOOS == "windows" {
@ -279,7 +280,7 @@ func (fsys *FS) Statfs(path string, stat *fuse.Statfs_t) (errc int) {
// Open opens a file
func (fsys *FS) Open(path string, flags int) (errc int, fh uint64) {
defer fs.Trace(path, "flags=0x%X", flags)("errc=%d, fh=0x%X", &errc, &fh)
defer log.Trace(path, "flags=0x%X", flags)("errc=%d, fh=0x%X", &errc, &fh)
// translate the fuse flags to os flags
flags = translateOpenFlags(flags) | os.O_CREATE
@ -293,7 +294,7 @@ func (fsys *FS) Open(path string, flags int) (errc int, fh uint64) {
// Create creates and opens a file.
func (fsys *FS) Create(filePath string, flags int, mode uint32) (errc int, fh uint64) {
defer fs.Trace(filePath, "flags=0x%X, mode=0%o", flags, mode)("errc=%d, fh=0x%X", &errc, &fh)
defer log.Trace(filePath, "flags=0x%X, mode=0%o", flags, mode)("errc=%d, fh=0x%X", &errc, &fh)
leaf, parentDir, errc := fsys.lookupParentDir(filePath)
if errc != 0 {
return errc, fhUnset
@ -313,7 +314,7 @@ func (fsys *FS) Create(filePath string, flags int, mode uint32) (errc int, fh ui
// Truncate truncates a file to size
func (fsys *FS) Truncate(path string, size int64, fh uint64) (errc int) {
defer fs.Trace(path, "size=%d, fh=0x%X", size, fh)("errc=%d", &errc)
defer log.Trace(path, "size=%d, fh=0x%X", size, fh)("errc=%d", &errc)
node, handle, errc := fsys.getNode(path, fh)
if errc != 0 {
return errc
@ -332,7 +333,7 @@ func (fsys *FS) Truncate(path string, size int64, fh uint64) (errc int) {
// Read data from file handle
func (fsys *FS) Read(path string, buff []byte, ofst int64, fh uint64) (n int) {
defer fs.Trace(path, "ofst=%d, fh=0x%X", ofst, fh)("n=%d", &n)
defer log.Trace(path, "ofst=%d, fh=0x%X", ofst, fh)("n=%d", &n)
handle, errc := fsys.getHandle(fh)
if errc != 0 {
return errc
@ -348,7 +349,7 @@ func (fsys *FS) Read(path string, buff []byte, ofst int64, fh uint64) (n int) {
// Write data to file handle
func (fsys *FS) Write(path string, buff []byte, ofst int64, fh uint64) (n int) {
defer fs.Trace(path, "ofst=%d, fh=0x%X", ofst, fh)("n=%d", &n)
defer log.Trace(path, "ofst=%d, fh=0x%X", ofst, fh)("n=%d", &n)
handle, errc := fsys.getHandle(fh)
if errc != 0 {
return errc
@ -362,7 +363,7 @@ func (fsys *FS) Write(path string, buff []byte, ofst int64, fh uint64) (n int) {
// Flush flushes an open file descriptor or path
func (fsys *FS) Flush(path string, fh uint64) (errc int) {
defer fs.Trace(path, "fh=0x%X", fh)("errc=%d", &errc)
defer log.Trace(path, "fh=0x%X", fh)("errc=%d", &errc)
handle, errc := fsys.getHandle(fh)
if errc != 0 {
return errc
@ -372,7 +373,7 @@ func (fsys *FS) Flush(path string, fh uint64) (errc int) {
// Release closes the file if still open
func (fsys *FS) Release(path string, fh uint64) (errc int) {
defer fs.Trace(path, "fh=0x%X", fh)("errc=%d", &errc)
defer log.Trace(path, "fh=0x%X", fh)("errc=%d", &errc)
handle, errc := fsys.getHandle(fh)
if errc != 0 {
return errc
@ -383,7 +384,7 @@ func (fsys *FS) Release(path string, fh uint64) (errc int) {
// Unlink removes a file.
func (fsys *FS) Unlink(filePath string) (errc int) {
defer fs.Trace(filePath, "")("errc=%d", &errc)
defer log.Trace(filePath, "")("errc=%d", &errc)
leaf, parentDir, errc := fsys.lookupParentDir(filePath)
if errc != 0 {
return errc
@ -393,7 +394,7 @@ func (fsys *FS) Unlink(filePath string) (errc int) {
// Mkdir creates a directory.
func (fsys *FS) Mkdir(dirPath string, mode uint32) (errc int) {
defer fs.Trace(dirPath, "mode=0%o", mode)("errc=%d", &errc)
defer log.Trace(dirPath, "mode=0%o", mode)("errc=%d", &errc)
leaf, parentDir, errc := fsys.lookupParentDir(dirPath)
if errc != 0 {
return errc
@ -404,7 +405,7 @@ func (fsys *FS) Mkdir(dirPath string, mode uint32) (errc int) {
// Rmdir removes a directory
func (fsys *FS) Rmdir(dirPath string) (errc int) {
defer fs.Trace(dirPath, "")("errc=%d", &errc)
defer log.Trace(dirPath, "")("errc=%d", &errc)
leaf, parentDir, errc := fsys.lookupParentDir(dirPath)
if errc != 0 {
return errc
@ -414,13 +415,13 @@ func (fsys *FS) Rmdir(dirPath string) (errc int) {
// Rename renames a file.
func (fsys *FS) Rename(oldPath string, newPath string) (errc int) {
defer fs.Trace(oldPath, "newPath=%q", newPath)("errc=%d", &errc)
defer log.Trace(oldPath, "newPath=%q", newPath)("errc=%d", &errc)
return translateError(fsys.VFS.Rename(oldPath, newPath))
}
// Utimens changes the access and modification times of a file.
func (fsys *FS) Utimens(path string, tmsp []fuse.Timespec) (errc int) {
defer fs.Trace(path, "tmsp=%+v", tmsp)("errc=%d", &errc)
defer log.Trace(path, "tmsp=%+v", tmsp)("errc=%d", &errc)
node, errc := fsys.lookupNode(path)
if errc != 0 {
return errc
@ -436,59 +437,59 @@ func (fsys *FS) Utimens(path string, tmsp []fuse.Timespec) (errc int) {
// Mknod creates a file node.
func (fsys *FS) Mknod(path string, mode uint32, dev uint64) (errc int) {
defer fs.Trace(path, "mode=0x%X, dev=0x%X", mode, dev)("errc=%d", &errc)
defer log.Trace(path, "mode=0x%X, dev=0x%X", mode, dev)("errc=%d", &errc)
return -fuse.ENOSYS
}
// Fsync synchronizes file contents.
func (fsys *FS) Fsync(path string, datasync bool, fh uint64) (errc int) {
defer fs.Trace(path, "datasync=%v, fh=0x%X", datasync, fh)("errc=%d", &errc)
defer log.Trace(path, "datasync=%v, fh=0x%X", datasync, fh)("errc=%d", &errc)
// This is a no-op for rclone
return 0
}
// Link creates a hard link to a file.
func (fsys *FS) Link(oldpath string, newpath string) (errc int) {
defer fs.Trace(oldpath, "newpath=%q", newpath)("errc=%d", &errc)
defer log.Trace(oldpath, "newpath=%q", newpath)("errc=%d", &errc)
return -fuse.ENOSYS
}
// Symlink creates a symbolic link.
func (fsys *FS) Symlink(target string, newpath string) (errc int) {
defer fs.Trace(target, "newpath=%q", newpath)("errc=%d", &errc)
defer log.Trace(target, "newpath=%q", newpath)("errc=%d", &errc)
return -fuse.ENOSYS
}
// Readlink reads the target of a symbolic link.
func (fsys *FS) Readlink(path string) (errc int, linkPath string) {
defer fs.Trace(path, "")("linkPath=%q, errc=%d", &linkPath, &errc)
defer log.Trace(path, "")("linkPath=%q, errc=%d", &linkPath, &errc)
return -fuse.ENOSYS, ""
}
// Chmod changes the permission bits of a file.
func (fsys *FS) Chmod(path string, mode uint32) (errc int) {
defer fs.Trace(path, "mode=0%o", mode)("errc=%d", &errc)
defer log.Trace(path, "mode=0%o", mode)("errc=%d", &errc)
// This is a no-op for rclone
return 0
}
// Chown changes the owner and group of a file.
func (fsys *FS) Chown(path string, uid uint32, gid uint32) (errc int) {
defer fs.Trace(path, "uid=%d, gid=%d", uid, gid)("errc=%d", &errc)
defer log.Trace(path, "uid=%d, gid=%d", uid, gid)("errc=%d", &errc)
// This is a no-op for rclone
return 0
}
// Access checks file access permissions.
func (fsys *FS) Access(path string, mask uint32) (errc int) {
defer fs.Trace(path, "mask=0%o", mask)("errc=%d", &errc)
defer log.Trace(path, "mask=0%o", mask)("errc=%d", &errc)
// This is a no-op for rclone
return 0
}
// Fsyncdir synchronizes directory contents.
func (fsys *FS) Fsyncdir(path string, datasync bool, fh uint64) (errc int) {
defer fs.Trace(path, "datasync=%v, fh=0x%X", datasync, fh)("errc=%d", &errc)
defer log.Trace(path, "datasync=%v, fh=0x%X", datasync, fh)("errc=%d", &errc)
// This is a no-op for rclone
return 0
}

22
cmd/config/config.go

@ -2,7 +2,7 @@ package config
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/spf13/cobra"
)
@ -28,7 +28,7 @@ password to protect your configuration.
`,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(0, 0, command, args)
fs.EditConfig()
config.EditConfig()
},
}
@ -44,7 +44,7 @@ var configFileCommand = &cobra.Command{
Short: `Show path of configuration file in use.`,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(0, 0, command, args)
fs.ShowConfigLocation()
config.ShowConfigLocation()
},
}
@ -54,9 +54,9 @@ var configShowCommand = &cobra.Command{
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(0, 1, command, args)
if len(args) == 0 {
fs.ShowConfig()
config.ShowConfig()
} else {
fs.ShowRemote(args[0])
config.ShowRemote(args[0])
}
},
}
@ -66,7 +66,7 @@ var configDumpCommand = &cobra.Command{
Short: `Dump the config file as JSON.`,
RunE: func(command *cobra.Command, args []string) error {
cmd.CheckArgs(0, 0, command, args)
return fs.ConfigDump()
return config.Dump()
},
}
@ -75,7 +75,7 @@ var configProvidersCommand = &cobra.Command{
Short: `List in JSON format all the providers and options.`,
RunE: func(command *cobra.Command, args []string) error {
cmd.CheckArgs(0, 0, command, args)
return fs.JSONListProviders()
return config.JSONListProviders()
},
}
@ -93,7 +93,7 @@ you would do:
`,
RunE: func(command *cobra.Command, args []string) error {
cmd.CheckArgs(2, 256, command, args)
return fs.CreateRemote(args[0], args[1], args[2:])
return config.CreateRemote(args[0], args[1], args[2:])
},
}
@ -110,7 +110,7 @@ For example to update the env_auth field of a remote of name myremote you would
`,
RunE: func(command *cobra.Command, args []string) error {
cmd.CheckArgs(3, 256, command, args)
return fs.UpdateRemote(args[0], args[1:])
return config.UpdateRemote(args[0], args[1:])
},
}
@ -119,7 +119,7 @@ var configDeleteCommand = &cobra.Command{
Short: `Delete an existing remote <name>.`,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(1, 1, command, args)
fs.DeleteRemote(args[0])
config.DeleteRemote(args[0])
},
}
@ -136,6 +136,6 @@ For example to set password of a remote of name myremote you would do:
`,
RunE: func(command *cobra.Command, args []string) error {
cmd.CheckArgs(3, 256, command, args)
return fs.PasswordRemote(args[0], args[1:])
return config.PasswordRemote(args[0], args[1:])
},
}

4
cmd/copy/copy.go

@ -2,7 +2,7 @@ package copy
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/sync"
"github.com/spf13/cobra"
)
@ -57,7 +57,7 @@ the destination directory or not.
cmd.CheckArgs(2, 2, command, args)
fsrc, fdst := cmd.NewFsSrcDst(args)
cmd.Run(true, true, command, func() error {
return fs.CopyDir(fdst, fsrc)
return sync.CopyDir(fdst, fsrc)
})
},
}

7
cmd/copyto/copyto.go

@ -2,7 +2,8 @@ package copyto
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/ncw/rclone/fs/sync"
"github.com/spf13/cobra"
)
@ -45,9 +46,9 @@ destination.
fsrc, srcFileName, fdst, dstFileName := cmd.NewFsSrcDstFiles(args)
cmd.Run(true, true, command, func() error {
if srcFileName == "" {
return fs.CopyDir(fdst, fsrc)
return sync.CopyDir(fdst, fsrc)
}
return fs.CopyFile(fdst, fsrc, dstFileName, srcFileName)
return operations.CopyFile(fdst, fsrc, dstFileName, srcFileName)
})
},
}

12
cmd/cryptcheck/cryptcheck.go

@ -4,6 +4,8 @@ import (
"github.com/ncw/rclone/backend/crypt"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/operations"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
@ -58,7 +60,7 @@ func cryptCheck(fdst, fsrc fs.Fs) error {
// Find a hash to use
funderlying := fcrypt.UnWrap()
hashType := funderlying.Hashes().GetOne()
if hashType == fs.HashNone {
if hashType == hash.HashNone {
return errors.Errorf("%s:%s does not support any hashes", funderlying.Name(), funderlying.Root())
}
fs.Infof(nil, "Using %v for hash comparisons", hashType)
@ -72,7 +74,7 @@ func cryptCheck(fdst, fsrc fs.Fs) error {
underlyingDst := cryptDst.UnWrap()
underlyingHash, err := underlyingDst.Hash(hashType)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
fs.Errorf(dst, "Error reading hash from underlying %v: %v", underlyingDst, err)
return true, false
}
@ -81,7 +83,7 @@ func cryptCheck(fdst, fsrc fs.Fs) error {
}
cryptHash, err := fcrypt.ComputeHash(cryptDst, src, hashType)
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
fs.Errorf(dst, "Error computing hash: %v", err)
return true, false
}
@ -90,7 +92,7 @@ func cryptCheck(fdst, fsrc fs.Fs) error {
}
if cryptHash != underlyingHash {
err = errors.Errorf("hashes differ (%s:%s) %q vs (%s:%s) %q", fdst.Name(), fdst.Root(), cryptHash, fsrc.Name(), fsrc.Root(), underlyingHash)
fs.Stats.Error(err)
fs.CountError(err)
fs.Errorf(src, err.Error())
return true, false
}
@ -98,5 +100,5 @@ func cryptCheck(fdst, fsrc fs.Fs) error {
return false, false
}
return fs.CheckFn(fcrypt, fsrc, checkIdentical)
return operations.CheckFn(fcrypt, fsrc, checkIdentical)
}

5
cmd/cryptdecode/cryptdecode.go

@ -6,6 +6,7 @@ import (
"github.com/ncw/rclone/backend/crypt"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config/flags"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
@ -17,8 +18,8 @@ var (
func init() {
cmd.Root.AddCommand(commandDefinition)
flags := commandDefinition.Flags()
fs.BoolVarP(flags, &Reverse, "reverse", "", Reverse, "Reverse cryptdecode, encrypts filenames")
flagSet := commandDefinition.Flags()
flags.BoolVarP(flagSet, &Reverse, "reverse", "", Reverse, "Reverse cryptdecode, encrypts filenames")
}
var commandDefinition = &cobra.Command{

4
cmd/dbhashsum/dbhashsum.go

@ -4,7 +4,7 @@ import (
"os"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -25,7 +25,7 @@ The output is in the same format as md5sum and sha1sum.
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
return fs.DropboxHashSum(fsrc, os.Stdout)
return operations.DropboxHashSum(fsrc, os.Stdout)
})
},
}

6
cmd/dedupe/dedupe.go

@ -4,12 +4,12 @@ import (
"log"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
var (
dedupeMode = fs.DeduplicateInteractive
dedupeMode = operations.DeduplicateInteractive
)
func init() {
@ -111,7 +111,7 @@ Or
}
fdst := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
return fs.Deduplicate(fdst, dedupeMode)
return operations.Deduplicate(fdst, dedupeMode)
})
},
}

4
cmd/delete/delete.go

@ -2,7 +2,7 @@ package delete
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -35,7 +35,7 @@ delete all files bigger than 100MBytes.
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(true, false, command, func() error {
return fs.Delete(fsrc)
return operations.Delete(fsrc)
})
},
}

18
cmd/info/info.go

@ -14,6 +14,8 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/object"
"github.com/ncw/rclone/fstest"
"github.com/pkg/errors"
"github.com/spf13/cobra"
@ -103,7 +105,7 @@ func (r *results) Print() {
// writeFile writes a file with some random contents
func (r *results) writeFile(path string) (fs.Object, error) {
contents := fstest.RandomString(50)
src := fs.NewStaticObjectInfo(path, time.Now(), int64(len(contents)), true, nil, r.f)
src := object.NewStaticObjectInfo(path, time.Now(), int64(len(contents)), true, nil, r.f)
return r.f.Put(bytes.NewBufferString(contents), src)
}
@ -210,10 +212,10 @@ func (r *results) checkStreaming() {
contents := "thinking of test strings is hard"
buf := bytes.NewBufferString(contents)
hashIn := fs.NewMultiHasher()
hashIn := hash.NewMultiHasher()
in := io.TeeReader(buf, hashIn)
objIn := fs.NewStaticObjectInfo("checkStreamingTest", time.Now(), -1, true, nil, r.f)
objIn := object.NewStaticObjectInfo("checkStreamingTest", time.Now(), -1, true, nil, r.f)
objR, err := putter(in, objIn)
if err != nil {
fs.Infof(r.f, "Streamed file failed to upload (%v)", err)
@ -223,15 +225,15 @@ func (r *results) checkStreaming() {
hashes := hashIn.Sums()
types := objR.Fs().Hashes().Array()
for _, hash := range types {
sum, err := objR.Hash(hash)
for _, Hash := range types {
sum, err := objR.Hash(Hash)
if err != nil {
fs.Infof(r.f, "Streamed file failed when getting hash %v (%v)", hash, err)
fs.Infof(r.f, "Streamed file failed when getting hash %v (%v)", Hash, err)
r.canStream = false
return
}
if !fs.HashEquals(hashes[hash], sum) {
fs.Infof(r.f, "Streamed file has incorrect hash %v: expecting %q got %q", hash, hashes[hash], sum)
if !hash.Equals(hashes[Hash], sum) {
fs.Infof(r.f, "Streamed file has incorrect hash %v: expecting %q got %q", Hash, hashes[Hash], sum)
r.canStream = false
return
}

3
cmd/listremotes/listremotes.go

@ -6,6 +6,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/spf13/cobra"
)
@ -29,7 +30,7 @@ When uses with the -l flag it lists the types too.
`,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(0, 0, command, args)
remotes := fs.ConfigFileSections()
remotes := config.FileSections()
sort.Strings(remotes)
maxlen := 1
for _, remote := range remotes {

4
cmd/ls/ls.go

@ -5,7 +5,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/cmd/ls/lshelp"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -24,7 +24,7 @@ readable format with size and path. Recurses by default.
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
return fs.List(fsrc, os.Stdout)
return operations.List(fsrc, os.Stdout)
})
},
}

4
cmd/lsd/lsd.go

@ -5,7 +5,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/cmd/ls/lshelp"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -24,7 +24,7 @@ by default.
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
return fs.ListDir(fsrc, os.Stdout)
return operations.ListDir(fsrc, os.Stdout)
})
},
}

13
cmd/lsf/lsf.go

@ -8,6 +8,9 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/cmd/ls/lshelp"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/operations"
"github.com/ncw/rclone/fs/walk"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
@ -17,7 +20,7 @@ var (
separator string
dirSlash bool
recurse bool
hashType = fs.HashMD5
hashType = hash.HashMD5
filesOnly bool
dirsOnly bool
)
@ -84,7 +87,7 @@ putting it last is a good strategy.
// Lsf lists all the objects in the path with modification time, size
// and path in specific format.
func Lsf(fsrc fs.Fs, out io.Writer) error {
var list fs.ListFormat
var list operations.ListFormat
list.SetSeparator(separator)
list.SetDirSlash(dirSlash)
@ -103,9 +106,9 @@ func Lsf(fsrc fs.Fs, out io.Writer) error {
}
}
return fs.Walk(fsrc, "", false, fs.ConfigMaxDepth(recurse), func(path string, entries fs.DirEntries, err error) error {
return walk.Walk(fsrc, "", false, operations.ConfigMaxDepth(recurse), func(path string, entries fs.DirEntries, err error) error {
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
fs.Errorf(path, "error listing: %v", err)
return nil
}
@ -120,7 +123,7 @@ func Lsf(fsrc fs.Fs, out io.Writer) error {
continue
}
}
fmt.Fprintln(out, fs.ListFormatted(&entry, &list))
fmt.Fprintln(out, operations.ListFormatted(&entry, &list))
}
return nil
})

7
cmd/lsf/lsf_test.go

@ -5,6 +5,7 @@ import (
"testing"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/list"
"github.com/ncw/rclone/fstest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@ -137,7 +138,7 @@ file3
err = Lsf(f, buf)
require.NoError(t, err)
items, _ := fs.ListDirSorted(f, true, "")
items, _ := list.DirSorted(f, true, "")
var expectedOutput string
for _, item := range items {
expectedOutput += item.ModTime().Format("2006-01-02 15:04:05") + "\n"
@ -198,8 +199,8 @@ func TestWholeLsf(t *testing.T) {
err = Lsf(f, buf)
require.NoError(t, err)
items, _ := fs.ListDirSorted(f, true, "")
itemsInSubdir, _ := fs.ListDirSorted(f, true, "subdir")
items, _ := list.DirSorted(f, true, "")
itemsInSubdir, _ := list.DirSorted(f, true, "subdir")
var expectedOutput []string
for _, item := range items {
expectedOutput = append(expectedOutput, item.ModTime().Format("2006-01-02 15:04:05"))

6
cmd/lsjson/lsjson.go

@ -10,6 +10,8 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/cmd/ls/lshelp"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/ncw/rclone/fs/walk"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
@ -84,9 +86,9 @@ can be processed line by line as each item is written one to a line.
cmd.Run(false, false, command, func() error {
fmt.Println("[")
first := true
err := fs.Walk(fsrc, "", false, fs.ConfigMaxDepth(recurse), func(dirPath string, entries fs.DirEntries, err error) error {
err := walk.Walk(fsrc, "", false, operations.ConfigMaxDepth(recurse), func(dirPath string, entries fs.DirEntries, err error) error {
if err != nil {
fs.Stats.Error(err)
fs.CountError(err)
fs.Errorf(dirPath, "error listing: %v", err)
return nil
}

4
cmd/lsl/lsl.go

@ -5,7 +5,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/cmd/ls/lshelp"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -24,7 +24,7 @@ readable format with modification time, size and path. Recurses by default.
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
return fs.ListLong(fsrc, os.Stdout)
return operations.ListLong(fsrc, os.Stdout)
})
},
}

4
cmd/md5sum/md5sum.go

@ -4,7 +4,7 @@ import (
"os"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -23,7 +23,7 @@ is in the same format as the standard md5sum tool produces.
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
return fs.Md5sum(fsrc, os.Stdout)
return operations.Md5sum(fsrc, os.Stdout)
})
},
}

5
cmd/memtest/memtest.go

@ -6,6 +6,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -21,7 +22,7 @@ var commandDefintion = &cobra.Command{
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
objects, _, err := fs.Count(fsrc)
objects, _, err := operations.Count(fsrc)
if err != nil {
return err
}
@ -30,7 +31,7 @@ var commandDefintion = &cobra.Command{
runtime.GC()
runtime.ReadMemStats(&before)
var mu sync.Mutex
err = fs.ListFn(fsrc, func(o fs.Object) {
err = operations.ListFn(fsrc, func(o fs.Object) {
mu.Lock()
objs = append(objs, o)
mu.Unlock()

4
cmd/mkdir/mkdir.go

@ -2,7 +2,7 @@ package mkdir
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -17,7 +17,7 @@ var commandDefintion = &cobra.Command{
cmd.CheckArgs(1, 1, command, args)
fdst := cmd.NewFsDst(args)
cmd.Run(true, false, command, func() error {
return fs.Mkdir(fdst, "")
return operations.Mkdir(fdst, "")
})
},
}

20
cmd/mount/dir.go

@ -8,7 +8,7 @@ import (
"bazil.org/fuse"
fusefs "bazil.org/fuse/fs"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/log"
"github.com/ncw/rclone/vfs"
"github.com/pkg/errors"
"golang.org/x/net/context"
@ -24,7 +24,7 @@ var _ fusefs.Node = (*Dir)(nil)
// Attr updates the attributes of a directory
func (d *Dir) Attr(ctx context.Context, a *fuse.Attr) (err error) {
defer fs.Trace(d, "")("attr=%+v, err=%v", a, &err)
defer log.Trace(d, "")("attr=%+v, err=%v", a, &err)
a.Gid = d.VFS().Opt.GID
a.Uid = d.VFS().Opt.UID
a.Mode = os.ModeDir | d.VFS().Opt.DirPerms
@ -43,7 +43,7 @@ var _ fusefs.NodeSetattrer = (*Dir)(nil)
// Setattr handles attribute changes from FUSE. Currently supports ModTime only.
func (d *Dir) Setattr(ctx context.Context, req *fuse.SetattrRequest, resp *fuse.SetattrResponse) (err error) {
defer fs.Trace(d, "stat=%+v", req)("err=%v", &err)
defer log.Trace(d, "stat=%+v", req)("err=%v", &err)
if d.VFS().Opt.NoModTime {
return nil
}
@ -67,7 +67,7 @@ var _ fusefs.NodeRequestLookuper = (*Dir)(nil)
//
// Lookup need not to handle the names "." and "..".
func (d *Dir) Lookup(ctx context.Context, req *fuse.LookupRequest, resp *fuse.LookupResponse) (node fusefs.Node, err error) {
defer fs.Trace(d, "name=%q", req.Name)("node=%+v, err=%v", &node, &err)
defer log.Trace(d, "name=%q", req.Name)("node=%+v, err=%v", &node, &err)
mnode, err := d.Dir.Stat(req.Name)
if err != nil {
return nil, translateError(err)
@ -87,7 +87,7 @@ var _ fusefs.HandleReadDirAller = (*Dir)(nil)
// ReadDirAll reads the contents of the directory
func (d *Dir) ReadDirAll(ctx context.Context) (dirents []fuse.Dirent, err error) {
itemsRead := -1
defer fs.Trace(d, "")("item=%d, err=%v", &itemsRead, &err)
defer log.Trace(d, "")("item=%d, err=%v", &itemsRead, &err)
items, err := d.Dir.ReadDirAll()
if err != nil {
return nil, translateError(err)
@ -111,7 +111,7 @@ var _ fusefs.NodeCreater = (*Dir)(nil)
// Create makes a new file
func (d *Dir) Create(ctx context.Context, req *fuse.CreateRequest, resp *fuse.CreateResponse) (node fusefs.Node, handle fusefs.Handle, err error) {
defer fs.Trace(d, "name=%q", req.Name)("node=%v, handle=%v, err=%v", &node, &handle, &err)
defer log.Trace(d, "name=%q", req.Name)("node=%v, handle=%v, err=%v", &node, &handle, &err)
file, err := d.Dir.Create(req.Name)
if err != nil {
return nil, nil, translateError(err)
@ -127,7 +127,7 @@ var _ fusefs.NodeMkdirer = (*Dir)(nil)
// Mkdir creates a new directory
func (d *Dir) Mkdir(ctx context.Context, req *fuse.MkdirRequest) (node fusefs.Node, err error) {
defer fs.Trace(d, "name=%q", req.Name)("node=%+v, err=%v", &node, &err)
defer log.Trace(d, "name=%q", req.Name)("node=%+v, err=%v", &node, &err)
dir, err := d.Dir.Mkdir(req.Name)
if err != nil {
return nil, translateError(err)
@ -141,7 +141,7 @@ var _ fusefs.NodeRemover = (*Dir)(nil)
// the receiver, which must be a directory. The entry to be removed
// may correspond to a file (unlink) or to a directory (rmdir).
func (d *Dir) Remove(ctx context.Context, req *fuse.RemoveRequest) (err error) {
defer fs.Trace(d, "name=%q", req.Name)("err=%v", &err)
defer log.Trace(d, "name=%q", req.Name)("err=%v", &err)
err = d.Dir.RemoveName(req.Name)
if err != nil {
return translateError(err)
@ -154,7 +154,7 @@ var _ fusefs.NodeRenamer = (*Dir)(nil)
// Rename the file
func (d *Dir) Rename(ctx context.Context, req *fuse.RenameRequest, newDir fusefs.Node) (err error) {
defer fs.Trace(d, "oldName=%q, newName=%q, newDir=%+v", req.OldName, req.NewName, newDir)("err=%v", &err)
defer log.Trace(d, "oldName=%q, newName=%q, newDir=%+v", req.OldName, req.NewName, newDir)("err=%v", &err)
destDir, ok := newDir.(*Dir)
if !ok {
return errors.Errorf("Unknown Dir type %T", newDir)
@ -173,7 +173,7 @@ var _ fusefs.NodeFsyncer = (*Dir)(nil)
// Fsync the directory
func (d *Dir) Fsync(ctx context.Context, req *fuse.FsyncRequest) (err error) {
defer fs.Trace(d, "")("err=%v", &err)
defer log.Trace(d, "")("err=%v", &err)
err = d.Dir.Sync()
if err != nil {
return translateError(err)

10
cmd/mount/file.go

@ -8,7 +8,7 @@ import (
"bazil.org/fuse"
fusefs "bazil.org/fuse/fs"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/log"
"github.com/ncw/rclone/vfs"
"golang.org/x/net/context"
)
@ -23,7 +23,7 @@ var _ fusefs.Node = (*File)(nil)
// Attr fills out the attributes for the file
func (f *File) Attr(ctx context.Context, a *fuse.Attr) (err error) {
defer fs.Trace(f, "")("a=%+v, err=%v", a, &err)
defer log.Trace(f, "")("a=%+v, err=%v", a, &err)
modTime := f.File.ModTime()
Size := uint64(f.File.Size())
Blocks := (Size + 511) / 512
@ -44,7 +44,7 @@ var _ fusefs.NodeSetattrer = (*File)(nil)
// Setattr handles attribute changes from FUSE. Currently supports ModTime and Size only
func (f *File) Setattr(ctx context.Context, req *fuse.SetattrRequest, resp *fuse.SetattrResponse) (err error) {
defer fs.Trace(f, "a=%+v", req)("err=%v", &err)
defer log.Trace(f, "a=%+v", req)("err=%v", &err)
if !f.VFS().Opt.NoModTime {
if req.Valid.MtimeNow() {
err = f.File.SetModTime(time.Now())
@ -64,7 +64,7 @@ var _ fusefs.NodeOpener = (*File)(nil)
// Open the file for read or write
func (f *File) Open(ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenResponse) (fh fusefs.Handle, err error) {
defer fs.Trace(f, "flags=%v", req.Flags)("fh=%v, err=%v", &fh, &err)
defer log.Trace(f, "flags=%v", req.Flags)("fh=%v, err=%v", &fh, &err)
// fuse flags are based off syscall flags as are os flags, so
// should be compatible
@ -91,6 +91,6 @@ var _ fusefs.NodeFsyncer = (*File)(nil)
//
// Note that we don't do anything except return OK
func (f *File) Fsync(ctx context.Context, req *fuse.FsyncRequest) (err error) {
defer fs.Trace(f, "")("err=%v", &err)
defer log.Trace(f, "")("err=%v", &err)
return nil
}

5
cmd/mount/fs.go

@ -10,6 +10,7 @@ import (
"bazil.org/fuse"
fusefs "bazil.org/fuse/fs"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/log"
"github.com/ncw/rclone/vfs"
"github.com/ncw/rclone/vfs/vfsflags"
"github.com/pkg/errors"
@ -36,7 +37,7 @@ func NewFS(f fs.Fs) *FS {
// Root returns the root node
func (f *FS) Root() (node fusefs.Node, err error) {
defer fs.Trace("", "")("node=%+v, err=%v", &node, &err)
defer log.Trace("", "")("node=%+v, err=%v", &node, &err)
root, err := f.VFS.Root()
if err != nil {
return nil, translateError(err)
@ -50,7 +51,7 @@ var _ fusefs.FSStatfser = (*FS)(nil)
// Statfs is called to obtain file system metadata.
// It should write that data to resp.
func (f *FS) Statfs(ctx context.Context, req *fuse.StatfsRequest, resp *fuse.StatfsResponse) (err error) {
defer fs.Trace("", "")("stat=%+v, err=%v", resp, &err)
defer log.Trace("", "")("stat=%+v, err=%v", resp, &err)
const blockSize = 4096
const fsBlocks = (1 << 50) / blockSize
resp.Blocks = fsBlocks // Total data blocks in file system.

10
cmd/mount/handle.go

@ -7,7 +7,7 @@ import (
"bazil.org/fuse"
fusefs "bazil.org/fuse/fs"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/log"
"github.com/ncw/rclone/vfs"
"golang.org/x/net/context"
)
@ -23,7 +23,7 @@ var _ fusefs.HandleReader = (*FileHandle)(nil)
// Read from the file handle
func (fh *FileHandle) Read(ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse) (err error) {
var n int
defer fs.Trace(fh, "len=%d, offset=%d", req.Size, req.Offset)("read=%d, err=%v", &n, &err)
defer log.Trace(fh, "len=%d, offset=%d", req.Size, req.Offset)("read=%d, err=%v", &n, &err)
data := make([]byte, req.Size)
n, err = fh.Handle.ReadAt(data, req.Offset)
if err == io.EOF {
@ -40,7 +40,7 @@ var _ fusefs.HandleWriter = (*FileHandle)(nil)
// Write data to the file handle
func (fh *FileHandle) Write(ctx context.Context, req *fuse.WriteRequest, resp *fuse.WriteResponse) (err error) {
defer fs.Trace(fh, "len=%d, offset=%d", len(req.Data), req.Offset)("written=%d, err=%v", &resp.Size, &err)
defer log.Trace(fh, "len=%d, offset=%d", len(req.Data), req.Offset)("written=%d, err=%v", &resp.Size, &err)
n, err := fh.Handle.WriteAt(req.Data, req.Offset)
if err != nil {
return translateError(err)
@ -68,7 +68,7 @@ var _ fusefs.HandleFlusher = (*FileHandle)(nil)
// Filesystems shouldn't assume that flush will always be called after
// some writes, or that if will be called at all.
func (fh *FileHandle) Flush(ctx context.Context, req *fuse.FlushRequest) (err error) {
defer fs.Trace(fh, "")("err=%v", &err)
defer log.Trace(fh, "")("err=%v", &err)
return translateError(fh.Handle.Flush())
}
@ -79,6 +79,6 @@ var _ fusefs.HandleReleaser = (*FileHandle)(nil)
// It isn't called directly from userspace so the error is ignored by
// the kernel
func (fh *FileHandle) Release(ctx context.Context, req *fuse.ReleaseRequest) (err error) {
defer fs.Trace(fh, "")("err=%v", &err)
defer log.Trace(fh, "")("err=%v", &err)
return translateError(fh.Handle.Release())
}

25
cmd/mountlib/mount.go

@ -8,6 +8,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config/flags"
"github.com/ncw/rclone/vfs"
"github.com/ncw/rclone/vfs/vfsflags"
"github.com/pkg/errors"
@ -181,21 +182,21 @@ will see all files and folders immediately in this mode.
cmd.Root.AddCommand(commandDefintion)
// Add flags
flags := commandDefintion.Flags()
fs.BoolVarP(flags, &DebugFUSE, "debug-fuse", "", DebugFUSE, "Debug the FUSE internals - needs -v.")
flagSet := commandDefintion.Flags()
flags.BoolVarP(flagSet, &DebugFUSE, "debug-fuse", "", DebugFUSE, "Debug the FUSE internals - needs -v.")
// mount options
fs.BoolVarP(flags, &AllowNonEmpty, "allow-non-empty", "", AllowNonEmpty, "Allow mounting over a non-empty directory.")
fs.BoolVarP(flags, &AllowRoot, "allow-root", "", AllowRoot, "Allow access to root user.")
fs.BoolVarP(flags, &AllowOther, "allow-other", "", AllowOther, "Allow access to other users.")
fs.BoolVarP(flags, &DefaultPermissions, "default-permissions", "", DefaultPermissions, "Makes kernel enforce access control based on the file mode.")
fs.BoolVarP(flags, &WritebackCache, "write-back-cache", "", WritebackCache, "Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.")
fs.FlagsVarP(flags, &MaxReadAhead, "max-read-ahead", "", "The number of bytes that can be prefetched for sequential reads.")
fs.StringArrayVarP(flags, &ExtraOptions, "option", "o", []string{}, "Option for libfuse/WinFsp. Repeat if required.")
fs.StringArrayVarP(flags, &ExtraFlags, "fuse-flag", "", []string{}, "Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required.")
//fs.BoolVarP(flags, &foreground, "foreground", "", foreground, "Do not detach.")
flags.BoolVarP(flagSet, &AllowNonEmpty, "allow-non-empty", "", AllowNonEmpty, "Allow mounting over a non-empty directory.")
flags.BoolVarP(flagSet, &AllowRoot, "allow-root", "", AllowRoot, "Allow access to root user.")
flags.BoolVarP(flagSet, &AllowOther, "allow-other", "", AllowOther, "Allow access to other users.")
flags.BoolVarP(flagSet, &DefaultPermissions, "default-permissions", "", DefaultPermissions, "Makes kernel enforce access control based on the file mode.")
flags.BoolVarP(flagSet, &WritebackCache, "write-back-cache", "", WritebackCache, "Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.")
flags.FVarP(flagSet, &MaxReadAhead, "max-read-ahead", "", "The number of bytes that can be prefetched for sequential reads.")
flags.StringArrayVarP(flagSet, &ExtraOptions, "option", "o", []string{}, "Option for libfuse/WinFsp. Repeat if required.")
flags.StringArrayVarP(flagSet, &ExtraFlags, "fuse-flag", "", []string{}, "Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required.")
//flags.BoolVarP(flagSet, &foreground, "foreground", "", foreground, "Do not detach.")
// Add in the generic flags
vfsflags.AddFlags(flags)
vfsflags.AddFlags(flagSet)
return commandDefintion
}

3
cmd/mountlib/mounttest/fs.go

@ -19,6 +19,7 @@ import (
_ "github.com/ncw/rclone/backend/all" // import all the backends
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/walk"
"github.com/ncw/rclone/fstest"
"github.com/ncw/rclone/vfs"
"github.com/stretchr/testify/assert"
@ -268,7 +269,7 @@ func (r *Run) readLocal(t *testing.T, dir dirMap, filePath string) {
// reads the remote tree into dir
func (r *Run) readRemote(t *testing.T, dir dirMap, filepath string) {
objs, dirs, err := fs.WalkGetAll(r.fremote, filepath, true, 1)
objs, dirs, err := walk.GetAll(r.fremote, filepath, true, 1)
if err == fs.ErrorDirNotFound {
return
}

4
cmd/move/move.go

@ -2,7 +2,7 @@ package move
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/sync"
"github.com/spf13/cobra"
)
@ -44,7 +44,7 @@ If you want to delete empty source directories after move, use the --delete-empt
fsrc, fdst := cmd.NewFsSrcDst(args)
cmd.Run(true, true, command, func() error {
return fs.MoveDir(fdst, fsrc, deleteEmptySrcDirs)
return sync.MoveDir(fdst, fsrc, deleteEmptySrcDirs)
})
},
}

7
cmd/moveto/moveto.go

@ -2,7 +2,8 @@ package moveto
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/ncw/rclone/fs/sync"
"github.com/spf13/cobra"
)
@ -49,9 +50,9 @@ transfer.
cmd.Run(true, true, command, func() error {
if srcFileName == "" {
return fs.MoveDir(fdst, fsrc, false)
return sync.MoveDir(fdst, fsrc, false)
}
return fs.MoveFile(fdst, fsrc, dstFileName, srcFileName)
return operations.MoveFile(fdst, fsrc, dstFileName, srcFileName)
})
},
}

3
cmd/ncdu/scan/scan.go

@ -6,6 +6,7 @@ import (
"sync"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/walk"
"github.com/pkg/errors"
)
@ -129,7 +130,7 @@ func Scan(f fs.Fs) (chan *Dir, chan error, chan struct{}) {
updated := make(chan struct{}, 1)
go func() {
parents := map[string]*Dir{}
err := fs.Walk(f, "", false, fs.Config.MaxDepth, func(dirPath string, entries fs.DirEntries, err error) error {
err := walk.Walk(f, "", false, fs.Config.MaxDepth, func(dirPath string, entries fs.DirEntries, err error) error {
if err != nil {
return err // FIXME mark directory as errored instead of aborting
}

4
cmd/obscure/obscure.go

@ -4,7 +4,7 @@ import (
"fmt"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/spf13/cobra"
)
@ -18,7 +18,7 @@ var commandDefintion = &cobra.Command{
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(1, 1, command, args)
cmd.Run(false, false, command, func() error {
obscure := fs.MustObscure(args[0])
obscure := config.MustObscure(args[0])
fmt.Println(obscure)
return nil
})

4
cmd/purge/purge.go

@ -2,7 +2,7 @@ package purge
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -22,7 +22,7 @@ you want to selectively delete files.
cmd.CheckArgs(1, 1, command, args)
fdst := cmd.NewFsDst(args)
cmd.Run(true, false, command, func() error {
return fs.Purge(fdst)
return operations.Purge(fdst)
})
},
}

4
cmd/rcat/rcat.go

@ -6,7 +6,7 @@ import (
"time"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -50,7 +50,7 @@ a lot of data, you're better off caching locally and then
fdst, dstFileName := cmd.NewFsDstFile(args)
cmd.Run(false, false, command, func() error {
_, err := fs.Rcat(fdst, dstFileName, os.Stdin, time.Now())
_, err := operations.Rcat(fdst, dstFileName, os.Stdin, time.Now())
return err
})
},

4
cmd/rmdir/rmdir.go

@ -2,7 +2,7 @@ package rmdir
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -20,7 +20,7 @@ objects in it, use purge for that.`,
cmd.CheckArgs(1, 1, command, args)
fdst := cmd.NewFsDst(args)
cmd.Run(true, false, command, func() error {
return fs.Rmdir(fdst, "")
return operations.Rmdir(fdst, "")
})
},
}

4
cmd/rmdirs/rmdirs.go

@ -2,7 +2,7 @@ package rmdir
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -32,7 +32,7 @@ empty directories in.
cmd.CheckArgs(1, 1, command, args)
fdst := cmd.NewFsDst(args)
cmd.Run(true, false, command, func() error {
return fs.Rmdirs(fdst, "", leaveRoot)
return operations.Rmdirs(fdst, "", leaveRoot)
})
},
}

11
cmd/serve/http/http.go

@ -11,6 +11,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/accounting"
"github.com/ncw/rclone/lib/rest"
"github.com/ncw/rclone/vfs"
"github.com/ncw/rclone/vfs/vfsflags"
@ -159,7 +160,7 @@ type indexData struct {
// error returns an http.StatusInternalServerError and logs the error
func internalError(what interface{}, w http.ResponseWriter, text string, err error) {
fs.Stats.Error(err)
fs.CountError(err)
fs.Errorf(what, "%s: %v", text, err)
http.Error(w, text+".", http.StatusInternalServerError)
}
@ -192,8 +193,8 @@ func (s *server) serveDir(w http.ResponseWriter, r *http.Request, dirRemote stri
}
// Account the transfer
fs.Stats.Transferring(dirRemote)
defer fs.Stats.DoneTransferring(dirRemote, true)
accounting.Stats.Transferring(dirRemote)
defer accounting.Stats.DoneTransferring(dirRemote, true)
fs.Infof(dirRemote, "%s: Serving directory", r.RemoteAddr)
err = indexTemplate.Execute(w, indexData{
@ -259,8 +260,8 @@ func (s *server) serveFile(w http.ResponseWriter, r *http.Request, remote string
}()
// Account the transfer
fs.Stats.Transferring(remote)
defer fs.Stats.DoneTransferring(remote, true)
accounting.Stats.Transferring(remote)
defer accounting.Stats.DoneTransferring(remote, true)
// FIXME in = fs.NewAccount(in, obj).WithBuffer() // account the transfer
// Serve the file

8
cmd/serve/http/http_test.go

@ -14,6 +14,8 @@ import (
_ "github.com/ncw/rclone/backend/local"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/filter"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@ -47,14 +49,14 @@ func startServer(t *testing.T, f fs.Fs) {
func TestInit(t *testing.T) {
// Configure the remote
fs.LoadConfig()
config.LoadConfig()
// fs.Config.LogLevel = fs.LogLevelDebug
// fs.Config.DumpHeaders = true
// fs.Config.DumpBodies = true
// exclude files called hidden.txt and directories called hidden
require.NoError(t, fs.Config.Filter.AddRule("- hidden.txt"))
require.NoError(t, fs.Config.Filter.AddRule("- hidden/**"))
require.NoError(t, filter.Active.AddRule("- hidden.txt"))
require.NoError(t, filter.Active.AddRule("- hidden/**"))
// Create a test Fs
f, err := fs.NewFs("testdata/files")

11
cmd/serve/webdav/webdav.go

@ -9,6 +9,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/log"
"github.com/ncw/rclone/vfs"
"github.com/ncw/rclone/vfs/vfsflags"
"github.com/spf13/cobra"
@ -96,7 +97,7 @@ func (w *WebDAV) logRequest(r *http.Request, err error) {
// Mkdir creates a directory
func (w *WebDAV) Mkdir(ctx context.Context, name string, perm os.FileMode) (err error) {
defer fs.Trace(name, "perm=%v", perm)("err = %v", &err)
defer log.Trace(name, "perm=%v", perm)("err = %v", &err)
dir, leaf, err := w.vfs.StatParent(name)
if err != nil {
return err
@ -107,13 +108,13 @@ func (w *WebDAV) Mkdir(ctx context.Context, name string, perm os.FileMode) (err
// OpenFile opens a file or a directory
func (w *WebDAV) OpenFile(ctx context.Context, name string, flags int, perm os.FileMode) (file webdav.File, err error) {
defer fs.Trace(name, "flags=%v, perm=%v", flags, perm)("err = %v", &err)
defer log.Trace(name, "flags=%v, perm=%v", flags, perm)("err = %v", &err)
return w.vfs.OpenFile(name, flags, perm)
}
// RemoveAll removes a file or a directory and its contents
func (w *WebDAV) RemoveAll(ctx context.Context, name string) (err error) {
defer fs.Trace(name, "")("err = %v", &err)
defer log.Trace(name, "")("err = %v", &err)
node, err := w.vfs.Stat(name)
if err != nil {
return err
@ -127,13 +128,13 @@ func (w *WebDAV) RemoveAll(ctx context.Context, name string) (err error) {
// Rename a file or a directory
func (w *WebDAV) Rename(ctx context.Context, oldName, newName string) (err error) {
defer fs.Trace(oldName, "newName=%q", newName)("err = %v", &err)
defer log.Trace(oldName, "newName=%q", newName)("err = %v", &err)
return w.vfs.Rename(oldName, newName)
}
// Stat returns info about the file or directory
func (w *WebDAV) Stat(ctx context.Context, name string) (fi os.FileInfo, err error) {
defer fs.Trace(name, "")("fi=%+v, err = %v", &fi, &err)
defer log.Trace(name, "")("fi=%+v, err = %v", &fi, &err)
return w.vfs.Stat(name)
}

4
cmd/sha1sum/sha1sum.go

@ -4,7 +4,7 @@ import (
"os"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -23,7 +23,7 @@ is in the same format as the standard sha1sum tool produces.
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
return fs.Sha1sum(fsrc, os.Stdout)
return operations.Sha1sum(fsrc, os.Stdout)
})
},
}

3
cmd/size/size.go

@ -5,6 +5,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/operations"
"github.com/spf13/cobra"
)
@ -19,7 +20,7 @@ var commandDefintion = &cobra.Command{
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
cmd.Run(false, false, command, func() error {
objects, size, err := fs.Count(fsrc)
objects, size, err := operations.Count(fsrc)
if err != nil {
return err
}

4
cmd/sync/sync.go

@ -2,7 +2,7 @@ package sync
import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/sync"
"github.com/spf13/cobra"
)
@ -37,7 +37,7 @@ go there.
cmd.CheckArgs(2, 2, command, args)
fsrc, fdst := cmd.NewFsSrcDst(args)
cmd.Run(true, true, command, func() error {
return fs.Sync(fdst, fsrc)
return sync.Sync(fdst, fsrc)
})
},
}

3
cmd/touch/touch.go

@ -6,6 +6,7 @@ import (
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/object"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
@ -55,7 +56,7 @@ func Touch(fsrc fs.Fs, srcFileName string) error {
if err != nil {
if !notCreateNewFile {
var buffer []byte
src := fs.NewStaticObjectInfo(srcFileName, timeAtr, int64(len(buffer)), true, nil, fsrc)
src := object.NewStaticObjectInfo(srcFileName, timeAtr, int64(len(buffer)), true, nil, fsrc)
_, err = fsrc.Put(bytes.NewBuffer(buffer), src)
if err != nil {
return err

20
cmd/tree/tree.go

@ -3,7 +3,6 @@ package tree
import (
"fmt"
"io"
"log"
"os"
"path"
"path/filepath"
@ -13,6 +12,8 @@ import (
"github.com/a8m/tree"
"github.com/ncw/rclone/cmd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/log"
"github.com/ncw/rclone/fs/walk"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
@ -88,7 +89,7 @@ The tree command has many options for controlling the listing which
are compatible with the tree command. Note that not all of them have
short options as they conflict with rclone's short options.
`,
Run: func(command *cobra.Command, args []string) {
RunE: func(command *cobra.Command, args []string) error {
cmd.CheckArgs(1, 1, command, args)
fsrc := cmd.NewFsSrc(args)
outFile := os.Stdout
@ -96,7 +97,7 @@ short options as they conflict with rclone's short options.
var err error
outFile, err = os.Create(outFileName)
if err != nil {
log.Fatalf("Failed to create output file: %v", err)
return errors.Errorf("failed to create output file: %v", err)
}
}
opts.VerSort = opts.VerSort || sort == "version"
@ -110,12 +111,13 @@ short options as they conflict with rclone's short options.
cmd.Run(false, false, command, func() error {
return Tree(fsrc, outFile, &opts)
})
return nil
},
}
// Tree lists fsrc to outFile using the Options passed in
func Tree(fsrc fs.Fs, outFile io.Writer, opts *tree.Options) error {
dirs, err := fs.NewDirTree(fsrc, "", false, opts.DeepLevel)
dirs, err := walk.NewDirTree(fsrc, "", false, opts.DeepLevel)
if err != nil {
return err
}
@ -183,22 +185,22 @@ func (to *FileInfo) String() string {
}
// Fs maps an fs.Fs into a tree.Fs
type Fs fs.DirTree
type Fs walk.DirTree
// NewFs creates a new tree
func NewFs(dirs fs.DirTree) Fs {
func NewFs(dirs walk.DirTree) Fs {
return Fs(dirs)
}
// Stat returns info about the file
func (dirs Fs) Stat(filePath string) (fi os.FileInfo, err error) {
defer fs.Trace(nil, "filePath=%q", filePath)("fi=%+v, err=%v", &fi, &err)
defer log.Trace(nil, "filePath=%q", filePath)("fi=%+v, err=%v", &fi, &err)
filePath = filepath.ToSlash(filePath)
filePath = strings.TrimLeft(filePath, "/")
if filePath == "" {
return &FileInfo{fs.NewDir("", time.Now())}, nil
}
_, entry := fs.DirTree(dirs).Find(filePath)
_, entry := walk.DirTree(dirs).Find(filePath)
if entry == nil {
return nil, errors.Errorf("Couldn't find %q in directory cache", filePath)
}
@ -207,7 +209,7 @@ func (dirs Fs) Stat(filePath string) (fi os.FileInfo, err error) {
// ReadDir returns info about the directory and fills up the directory cache
func (dirs Fs) ReadDir(dir string) (names []string, err error) {
defer fs.Trace(nil, "dir=%s", dir)("names=%+v, err=%v", &names, &err)
defer log.Trace(nil, "dir=%s", dir)("names=%+v, err=%v", &names, &err)
dir = filepath.ToSlash(dir)
dir = strings.TrimLeft(dir, "/")
entries, ok := dirs[dir]

98
fs/accounting.go → fs/accounting/accounting.go

@ -1,6 +1,5 @@
// Accounting and limiting reader
package fs
// Package accounting providers an accounting and limiting reader
package accounting
import (
"bytes"
@ -12,6 +11,8 @@ import (
"time"
"github.com/VividCortex/ewma"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/asyncreader"
"golang.org/x/net/context" // switch to "context" when we stop supporting go1.6
"golang.org/x/time/rate"
)
@ -24,31 +25,36 @@ var (
prevTokenBucket = tokenBucket
bwLimitToggledOff = false
currLimitMu sync.Mutex // protects changes to the timeslot
currLimit BwTimeSlot
currLimit fs.BwTimeSlot
)
func init() {
// Set the function pointer up in fs
fs.CountError = Stats.Error
}
const maxBurstSize = 1 * 1024 * 1024 // must be bigger than the biggest request
// make a new empty token bucket with the bandwidth given
func newTokenBucket(bandwidth SizeSuffix) *rate.Limiter {
func newTokenBucket(bandwidth fs.SizeSuffix) *rate.Limiter {
newTokenBucket := rate.NewLimiter(rate.Limit(bandwidth), maxBurstSize)
// empty the bucket
err := newTokenBucket.WaitN(context.Background(), maxBurstSize)
if err != nil {
Errorf(nil, "Failed to empty token bucket: %v", err)
fs.Errorf(nil, "Failed to empty token bucket: %v", err)
}
return newTokenBucket
}
// Start the token bucket if necessary
func startTokenBucket() {
// StartTokenBucket starts the token bucket if necessary
func StartTokenBucket() {
currLimitMu.Lock()
currLimit := bwLimit.LimitAt(time.Now())
currLimit := fs.Config.BwLimit.LimitAt(time.Now())
currLimitMu.Unlock()
if currLimit.bandwidth > 0 {
tokenBucket = newTokenBucket(currLimit.bandwidth)
Infof(nil, "Starting bandwidth limiter at %vBytes/s", &currLimit.bandwidth)
if currLimit.Bandwidth > 0 {
tokenBucket = newTokenBucket(currLimit.Bandwidth)
fs.Infof(nil, "Starting bandwidth limiter at %vBytes/s", &currLimit.Bandwidth)
// Start the SIGUSR2 signal handler to toggle bandwidth.
// This function does nothing in windows systems.
@ -56,21 +62,21 @@ func startTokenBucket() {
}
}
// startTokenTicker creates a ticker to update the bandwidth limiter every minute.
func startTokenTicker() {
// StartTokenTicker creates a ticker to update the bandwidth limiter every minute.
func StartTokenTicker() {
// If the timetable has a single entry or was not specified, we don't need
// a ticker to update the bandwidth.
if len(bwLimit) <= 1 {
if len(fs.Config.BwLimit) <= 1 {
return
}
ticker := time.NewTicker(time.Minute)
go func() {
for range ticker.C {
limitNow := bwLimit.LimitAt(time.Now())
limitNow := fs.Config.BwLimit.LimitAt(time.Now())
currLimitMu.Lock()
if currLimit.bandwidth != limitNow.bandwidth {
if currLimit.Bandwidth != limitNow.Bandwidth {
tokenBucketMu.Lock()
// If bwlimit is toggled off, the change should only
@ -84,17 +90,17 @@ func startTokenTicker() {
}
// Set new bandwidth. If unlimited, set tokenbucket to nil.
if limitNow.bandwidth > 0 {
*targetBucket = newTokenBucket(limitNow.bandwidth)
if limitNow.Bandwidth > 0 {
*targetBucket = newTokenBucket(limitNow.Bandwidth)
if bwLimitToggledOff {
Logf(nil, "Scheduled bandwidth change. "+
"Limit will be set to %vBytes/s when toggled on again.", &limitNow.bandwidth)
fs.Logf(nil, "Scheduled bandwidth change. "+
"Limit will be set to %vBytes/s when toggled on again.", &limitNow.Bandwidth)
} else {
Logf(nil, "Scheduled bandwidth change. Limit set to %vBytes/s", &limitNow.bandwidth)
fs.Logf(nil, "Scheduled bandwidth change. Limit set to %vBytes/s", &limitNow.Bandwidth)
}
} else {
*targetBucket = nil
Logf(nil, "Scheduled bandwidth change. Bandwidth limits disabled")
fs.Logf(nil, "Scheduled bandwidth change. Bandwidth limits disabled")
}
currLimit = limitNow
@ -117,7 +123,7 @@ type inProgress struct {
// newInProgress makes a new inProgress object
func newInProgress() *inProgress {
return &inProgress{
m: make(map[string]*Account, Config.Transfers),
m: make(map[string]*Account, fs.Config.Transfers),
}
}
@ -181,8 +187,8 @@ type StatsInfo struct {
// NewStats cretates an initialised StatsInfo
func NewStats() *StatsInfo {
return &StatsInfo{
checking: make(stringSet, Config.Checkers),
transferring: make(stringSet, Config.Transfers),
checking: make(stringSet, fs.Config.Checkers),
transferring: make(stringSet, fs.Config.Transfers),
start: time.Now(),
inProgress: newInProgress(),
}
@ -201,7 +207,7 @@ func (s *StatsInfo) String() string {
dtRounded := dt - (dt % (time.Second / 10))
buf := &bytes.Buffer{}
if Config.DataRateUnit == "bits" {
if fs.Config.DataRateUnit == "bits" {
speed = speed * 8
}
@ -212,7 +218,7 @@ Checks: %10d
Transferred: %10d
Elapsed time: %10v
`,
SizeSuffix(s.bytes).Unit("Bytes"), SizeSuffix(speed).Unit(strings.Title(Config.DataRateUnit)+"/s"),
fs.SizeSuffix(s.bytes).Unit("Bytes"), fs.SizeSuffix(speed).Unit(strings.Title(fs.Config.DataRateUnit)+"/s"),
s.errors,
s.checks,
s.transfers,
@ -228,7 +234,7 @@ Elapsed time: %10v
// Log outputs the StatsInfo to the log
func (s *StatsInfo) Log() {
LogLevelPrintf(Config.StatsLogLevel, nil, "%v\n", s)
fs.LogLevelPrintf(fs.Config.StatsLogLevel, nil, "%v\n", s)
}
// Bytes updates the stats for bytes bytes
@ -375,7 +381,7 @@ func NewAccountSizeName(in io.ReadCloser, size int64, name string) *Account {
}
// NewAccount makes a Account reader for an object
func NewAccount(in io.ReadCloser, obj Object) *Account {
func NewAccount(in io.ReadCloser, obj fs.Object) *Account {
return NewAccountSizeName(in, obj.Size(), obj.Remote())
}
@ -383,16 +389,16 @@ func NewAccount(in io.ReadCloser, obj Object) *Account {
func (acc *Account) WithBuffer() *Account {
acc.withBuf = true
var buffers int
if acc.size >= int64(Config.BufferSize) || acc.size == -1 {
buffers = int(int64(Config.BufferSize) / asyncBufferSize)
if acc.size >= int64(fs.Config.BufferSize) || acc.size == -1 {
buffers = int(int64(fs.Config.BufferSize) / asyncreader.BufferSize)
} else {
buffers = int(acc.size / asyncBufferSize)
buffers = int(acc.size / asyncreader.BufferSize)
}
// On big files add a buffer
if buffers > 0 {
in, err := newAsyncReader(acc.in, buffers)
in, err := asyncreader.New(acc.in, buffers)
if err != nil {
Errorf(acc.name, "Failed to make buffer: %v", err)
fs.Errorf(acc.name, "Failed to make buffer: %v", err)
} else {
acc.in = in
}
@ -409,7 +415,7 @@ func (acc *Account) GetReader() io.ReadCloser {
// StopBuffering stops the async buffer doing any more buffering
func (acc *Account) StopBuffering() {
if asyncIn, ok := acc.in.(*asyncReader); ok {
if asyncIn, ok := acc.in.(*asyncreader.AsyncReader); ok {
asyncIn.Abandon()
}
}
@ -484,7 +490,7 @@ func (acc *Account) read(in io.Reader, p []byte) (n int, err error) {
if tokenBucket != nil {
tbErr := tokenBucket.WaitN(context.Background(), n)
if tbErr != nil {
Errorf(nil, "Token bucket error: %v", err)
fs.Errorf(nil, "Token bucket error: %v", err)
}
}
tokenBucketMu.Unlock()
@ -572,14 +578,14 @@ func (acc *Account) String() string {
}
}
name := []rune(acc.name)
if Config.StatsFileNameLength > 0 {
if len(name) > Config.StatsFileNameLength {
where := len(name) - Config.StatsFileNameLength
if fs.Config.StatsFileNameLength > 0 {
if len(name) > fs.Config.StatsFileNameLength {
where := len(name) - fs.Config.StatsFileNameLength
name = append([]rune{'.', '.', '.'}, name[where:]...)
}
}
if Config.DataRateUnit == "bits" {
if fs.Config.DataRateUnit == "bits" {
cur = cur * 8
}
@ -588,12 +594,12 @@ func (acc *Account) String() string {
percentageDone = int(100 * float64(a) / float64(b))
}
done := fmt.Sprintf("%2d%% /%s", percentageDone, SizeSuffix(b))
done := fmt.Sprintf("%2d%% /%s", percentageDone, fs.SizeSuffix(b))
return fmt.Sprintf("%45s: %s, %s/s, %s",
string(name),
done,
SizeSuffix(cur),
fs.SizeSuffix(cur),
etas,
)
}
@ -633,10 +639,10 @@ func (a *accountStream) Read(p []byte) (n int, err error) {
// AccountByPart turns off whole file accounting
//
// Returns the current account or nil if not found
func AccountByPart(obj Object) *Account {
func AccountByPart(obj fs.Object) *Account {
acc := Stats.inProgress.get(obj.Remote())
if acc == nil {
Debugf(obj, "Didn't find object to account part transfer")
fs.Debugf(obj, "Didn't find object to account part transfer")
return nil
}
acc.disableWholeFileAccounting()
@ -647,7 +653,7 @@ func AccountByPart(obj Object) *Account {
//
// It disables the whole file counter and returns an io.Reader to wrap
// a segment of the transfer.
func AccountPart(obj Object, in io.Reader) io.Reader {
func AccountPart(obj fs.Object, in io.Reader) io.Reader {
acc := AccountByPart(obj)
if acc == nil {
return in

2
fs/accounting_other.go → fs/accounting/accounting_other.go

@ -3,7 +3,7 @@
// +build !darwin,!dragonfly,!freebsd,!linux,!netbsd,!openbsd,!solaris
package fs
package accounting
// startSignalHandler() is Unix specific and does nothing under non-Unix
// platforms.

6
fs/accounting_unix.go → fs/accounting/accounting_unix.go

@ -3,12 +3,14 @@
// +build darwin dragonfly freebsd linux netbsd openbsd solaris
package fs
package accounting
import (
"os"
"os/signal"
"syscall"
"github.com/ncw/rclone/fs"
)
// startSignalHandler() sets a signal handler to catch SIGUSR2 and toggle throttling.
@ -28,7 +30,7 @@ func startSignalHandler() {
s = "enabled"
}
tokenBucketMu.Unlock()
Logf(nil, "Bandwidth limit %s by user", s)
fs.Logf(nil, "Bandwidth limit %s by user", s)
}
}()
}

46
fs/buffer.go → fs/asyncreader/asyncreader.go

@ -1,14 +1,18 @@
package fs
// Package asyncreader provides an asynchronous reader which reads
// independently of write
package asyncreader
import (
"io"
"sync"
"github.com/ncw/rclone/lib/readers"
"github.com/pkg/errors"
)
const (
asyncBufferSize = 1024 * 1024
// BufferSize is the default size of the async buffer
BufferSize = 1024 * 1024
softStartInitial = 4 * 1024
)
@ -18,11 +22,11 @@ var asyncBufferPool = sync.Pool{
var errorStreamAbandoned = errors.New("stream abandoned")
// asyncReader will do async read-ahead from the input reader
// AsyncReader will do async read-ahead from the input reader
// and make the data available as an io.Reader.
// This should be fully transparent, except that once an error
// has been returned from the Reader, it will not recover.
type asyncReader struct {
type AsyncReader struct {
in io.ReadCloser // Input reader
ready chan *buffer // Buffers ready to be handed to the reader
token chan struct{} // Tokens which allow a buffer to be taken
@ -36,25 +40,25 @@ type asyncReader struct {
mu sync.Mutex // lock for Read/WriteTo/Abandon/Close
}
// newAsyncReader returns a reader that will asynchronously read from
// the supplied Reader into a number of buffers each of size asyncBufferSize
// New returns a reader that will asynchronously read from
// the supplied Reader into a number of buffers each of size BufferSize
// It will start reading from the input at once, maybe even before this
// function has returned.
// The input can be read from the returned reader.
// When done use Close to release the buffers and close the supplied input.
func newAsyncReader(rd io.ReadCloser, buffers int) (*asyncReader, error) {
func New(rd io.ReadCloser, buffers int) (*AsyncReader, error) {
if buffers <= 0 {
return nil, errors.New("number of buffers too small")
}
if rd == nil {
return nil, errors.New("nil reader supplied")
}
a := &asyncReader{}
a := &AsyncReader{}
a.init(rd, buffers)
return a, nil
}
func (a *asyncReader) init(rd io.ReadCloser, buffers int) {
func (a *AsyncReader) init(rd io.ReadCloser, buffers int) {
a.in = rd
a.ready = make(chan *buffer, buffers)
a.token = make(chan struct{}, buffers)
@ -78,7 +82,7 @@ func (a *asyncReader) init(rd io.ReadCloser, buffers int) {
select {
case <-a.token:
b := a.getBuffer()
if a.size < asyncBufferSize {
if a.size < BufferSize {
b.buf = b.buf[:a.size]
a.size <<= 1
}
@ -95,19 +99,19 @@ func (a *asyncReader) init(rd io.ReadCloser, buffers int) {
}
// return the buffer to the pool (clearing it)
func (a *asyncReader) putBuffer(b *buffer) {
func (a *AsyncReader) putBuffer(b *buffer) {
b.clear()
asyncBufferPool.Put(b)
}
// get a buffer from the pool
func (a *asyncReader) getBuffer() *buffer {
func (a *AsyncReader) getBuffer() *buffer {
b := asyncBufferPool.Get().(*buffer)
return b
}
// Read will return the next available data.
func (a *asyncReader) fill() (err error) {
func (a *AsyncReader) fill() (err error) {
if a.cur.isEmpty() {
if a.cur != nil {
a.putBuffer(a.cur)
@ -128,7 +132,7 @@ func (a *asyncReader) fill() (err error) {
}
// Read will return the next available data.
func (a *asyncReader) Read(p []byte) (n int, err error) {
func (a *AsyncReader) Read(p []byte) (n int, err error) {
a.mu.Lock()
defer a.mu.Unlock()
@ -153,7 +157,7 @@ func (a *asyncReader) Read(p []byte) (n int, err error) {
// WriteTo writes data to w until there's no more data to write or when an error occurs.
// The return value n is the number of bytes written.
// Any error encountered during the write is also returned.
func (a *asyncReader) WriteTo(w io.Writer) (n int64, err error) {
func (a *AsyncReader) WriteTo(w io.Writer) (n int64, err error) {
a.mu.Lock()
defer a.mu.Unlock()
@ -177,8 +181,8 @@ func (a *asyncReader) WriteTo(w io.Writer) (n int64, err error) {
}
// Abandon will ensure that the underlying async reader is shut down.
// It will NOT close the input supplied on newAsyncReader.
func (a *asyncReader) Abandon() {
// It will NOT close the input supplied on New.
func (a *AsyncReader) Abandon() {
select {
case <-a.exit:
// Do nothing if reader routine already exited
@ -202,8 +206,8 @@ func (a *asyncReader) Abandon() {
}
// Close will ensure that the underlying async reader is shut down.
// It will also close the input supplied on newAsyncReader.
func (a *asyncReader) Close() (err error) {
// It will also close the input supplied on New.
func (a *AsyncReader) Close() (err error) {
a.Abandon()
if a.closed {
return nil
@ -223,7 +227,7 @@ type buffer struct {
func newBuffer() *buffer {
return &buffer{
buf: make([]byte, asyncBufferSize),
buf: make([]byte, BufferSize),
err: nil,
}
}
@ -252,7 +256,7 @@ func (b *buffer) isEmpty() bool {
// Any error encountered during the read is returned.
func (b *buffer) read(rd io.Reader) error {
var n int
n, b.err = ReadFill(rd, b.buf)
n, b.err = readers.ReadFill(rd, b.buf)
b.buf = b.buf[0:n]
b.offset = 0
return b.err

24
fs/buffer_test.go → fs/asyncreader/asyncreader_test.go

@ -1,4 +1,4 @@
package fs
package asyncreader
import (
"bufio"
@ -17,7 +17,7 @@ import (
func TestAsyncReader(t *testing.T) {
buf := ioutil.NopCloser(bytes.NewBufferString("Testbuffer"))
ar, err := newAsyncReader(buf, 4)
ar, err := New(buf, 4)
require.NoError(t, err)
var dst = make([]byte, 100)
@ -42,7 +42,7 @@ func TestAsyncReader(t *testing.T) {
// Test Close without reading everything
buf = ioutil.NopCloser(bytes.NewBuffer(make([]byte, 50000)))
ar, err = newAsyncReader(buf, 4)
ar, err = New(buf, 4)
require.NoError(t, err)
err = ar.Close()
require.NoError(t, err)
@ -51,7 +51,7 @@ func TestAsyncReader(t *testing.T) {
func TestAsyncWriteTo(t *testing.T) {
buf := ioutil.NopCloser(bytes.NewBufferString("Testbuffer"))
ar, err := newAsyncReader(buf, 4)
ar, err := New(buf, 4)
require.NoError(t, err)
var dst = &bytes.Buffer{}
@ -70,14 +70,14 @@ func TestAsyncWriteTo(t *testing.T) {
func TestAsyncReaderErrors(t *testing.T) {
// test nil reader
_, err := newAsyncReader(nil, 4)
_, err := New(nil, 4)
require.Error(t, err)
// invalid buffer number
buf := ioutil.NopCloser(bytes.NewBufferString("Testbuffer"))
_, err = newAsyncReader(buf, 0)
_, err = New(buf, 0)
require.Error(t, err)
_, err = newAsyncReader(buf, -1)
_, err = New(buf, -1)
require.Error(t, err)
}
@ -157,9 +157,9 @@ func TestAsyncReaderSizes(t *testing.T) {
bufsize := bufsizes[k]
read := readmaker.fn(strings.NewReader(text))
buf := bufio.NewReaderSize(read, bufsize)
ar, _ := newAsyncReader(ioutil.NopCloser(buf), l)
ar, _ := New(ioutil.NopCloser(buf), l)
s := bufreader.fn(ar)
// "timeout" expects the Reader to recover, asyncReader does not.
// "timeout" expects the Reader to recover, AsyncReader does not.
if s != text && readmaker.name != "timeout" {
t.Errorf("reader=%s fn=%s bufsize=%d want=%q got=%q",
readmaker.name, bufreader.name, bufsize, text, s)
@ -196,14 +196,14 @@ func TestAsyncReaderWriteTo(t *testing.T) {
bufsize := bufsizes[k]
read := readmaker.fn(strings.NewReader(text))
buf := bufio.NewReaderSize(read, bufsize)
ar, _ := newAsyncReader(ioutil.NopCloser(buf), l)
ar, _ := New(ioutil.NopCloser(buf), l)
dst := &bytes.Buffer{}
_, err := ar.WriteTo(dst)
if err != nil && err != io.EOF && err != iotest.ErrTimeout {
t.Fatal("Copy:", err)
}
s := dst.String()
// "timeout" expects the Reader to recover, asyncReader does not.
// "timeout" expects the Reader to recover, AsyncReader does not.
if s != text && readmaker.name != "timeout" {
t.Errorf("reader=%s fn=%s bufsize=%d want=%q got=%q",
readmaker.name, bufreader.name, bufsize, text, s)
@ -243,7 +243,7 @@ func (z *zeroReader) Close() error {
// Test closing and abandoning
func testAsyncReaderClose(t *testing.T, writeto bool) {
zr := &zeroReader{}
a, err := newAsyncReader(zr, 16)
a, err := New(zr, 16)
require.NoError(t, err)
var copyN int64
var copyErr error

132
fs/bwtimetable.go Normal file

@ -0,0 +1,132 @@
package fs
import (
"fmt"
"strconv"
"strings"
"time"
"github.com/pkg/errors"
)
// BwTimeSlot represents a bandwidth configuration at a point in time.
type BwTimeSlot struct {
HHMM int
Bandwidth SizeSuffix
}
// BwTimetable contains all configured time slots.
type BwTimetable []BwTimeSlot
// String returns a printable representation of BwTimetable.
func (x BwTimetable) String() string {
ret := []string{}
for _, ts := range x {
ret = append(ret, fmt.Sprintf("%04.4d,%s", ts.HHMM, ts.Bandwidth.String()))
}
return strings.Join(ret, " ")
}
// Set the bandwidth timetable.
func (x *BwTimetable) Set(s string) error {
// The timetable is formatted as:
// "hh:mm,bandwidth hh:mm,banwidth..." ex: "10:00,10G 11:30,1G 18:00,off"
// If only a single bandwidth identifier is provided, we assume constant bandwidth.
if len(s) == 0 {
return errors.New("empty string")
}
// Single value without time specification.
if !strings.Contains(s, " ") && !strings.Contains(s, ",") {
ts := BwTimeSlot{}
if err := ts.Bandwidth.Set(s); err != nil {
return err
}
ts.HHMM = 0
*x = BwTimetable{ts}
return nil
}
for _, tok := range strings.Split(s, " ") {
tv := strings.Split(tok, ",")
// Format must be HH:MM,BW
if len(tv) != 2 {
return errors.Errorf("invalid time/bandwidth specification: %q", tok)
}
// Basic timespec sanity checking
HHMM := tv[0]
if len(HHMM) != 5 {
return errors.Errorf("invalid time specification (hh:mm): %q", HHMM)
}
hh, err := strconv.Atoi(HHMM[0:2])
if err != nil {
return errors.Errorf("invalid hour in time specification %q: %v", HHMM, err)
}
if hh < 0 || hh > 23 {
return errors.Errorf("invalid hour (must be between 00 and 23): %q", hh)
}
mm, err := strconv.Atoi(HHMM[3:])
if err != nil {
return errors.Errorf("invalid minute in time specification: %q: %v", HHMM, err)
}
if mm < 0 || mm > 59 {
return errors.Errorf("invalid minute (must be between 00 and 59): %q", hh)
}
ts := BwTimeSlot{
HHMM: (hh * 100) + mm,
}
// Bandwidth limit for this time slot.
if err := ts.Bandwidth.Set(tv[1]); err != nil {
return err
}
*x = append(*x, ts)
}
return nil
}
// LimitAt returns a BwTimeSlot for the time requested.
func (x BwTimetable) LimitAt(tt time.Time) BwTimeSlot {
// If the timetable is empty, we return an unlimited BwTimeSlot starting at midnight.
if len(x) == 0 {
return BwTimeSlot{HHMM: 0, Bandwidth: -1}
}
HHMM := tt.Hour()*100 + tt.Minute()
// By default, we return the last element in the timetable. This
// satisfies two conditions: 1) If there's only one element it
// will always be selected, and 2) The last element of the table
// will "wrap around" until overriden by an earlier time slot.
// there's only one time slot in the timetable.
ret := x[len(x)-1]
mindif := 0
first := true
// Look for most recent time slot.
for _, ts := range x {
// Ignore the past
if HHMM < ts.HHMM {
continue
}
dif := ((HHMM / 100 * 60) + (HHMM % 100)) - ((ts.HHMM / 100 * 60) + (ts.HHMM % 100))
if first {
mindif = dif
first = false
}
if dif <= mindif {
mindif = dif
ret = ts
}
}
return ret
}
// Type of the value
func (x BwTimetable) Type() string {
return "BwTimetable"
}

113
fs/bwtimetable_test.go Normal file

@ -0,0 +1,113 @@
package fs
import (
"testing"
"time"
"github.com/spf13/pflag"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Check it satisfies the interface
var _ pflag.Value = (*BwTimetable)(nil)
func TestBwTimetableSet(t *testing.T) {
for _, test := range []struct {
in string
want BwTimetable
err bool
}{
{"", BwTimetable{}, true},
{"0", BwTimetable{BwTimeSlot{HHMM: 0, Bandwidth: 0}}, false},
{"666", BwTimetable{BwTimeSlot{HHMM: 0, Bandwidth: 666 * 1024}}, false},
{"10:20,666", BwTimetable{BwTimeSlot{HHMM: 1020, Bandwidth: 666 * 1024}}, false},
{
"11:00,333 13:40,666 23:50,10M 23:59,off",
BwTimetable{
BwTimeSlot{HHMM: 1100, Bandwidth: 333 * 1024},
BwTimeSlot{HHMM: 1340, Bandwidth: 666 * 1024},
BwTimeSlot{HHMM: 2350, Bandwidth: 10 * 1024 * 1024},
BwTimeSlot{HHMM: 2359, Bandwidth: -1},
},
false,
},
{"bad,bad", BwTimetable{}, true},
{"bad bad", BwTimetable{}, true},
{"bad", BwTimetable{}, true},
{"1000X", BwTimetable{}, true},
{"2401,666", BwTimetable{}, true},
{"1061,666", BwTimetable{}, true},
} {
tt := BwTimetable{}
err := tt.Set(test.in)
if test.err {
require.Error(t, err)
} else {
require.NoError(t, err)
}
assert.Equal(t, test.want, tt)
}
}
func TestBwTimetableLimitAt(t *testing.T) {
for _, test := range []struct {
tt BwTimetable
now time.Time
want BwTimeSlot
}{
{
BwTimetable{},
time.Date(2017, time.April, 20, 15, 0, 0, 0, time.UTC),
BwTimeSlot{HHMM: 0, Bandwidth: -1},
},
{
BwTimetable{BwTimeSlot{HHMM: 1100, Bandwidth: 333 * 1024}},
time.Date(2017, time.April, 20, 15, 0, 0, 0, time.UTC),
BwTimeSlot{HHMM: 1100, Bandwidth: 333 * 1024},
},
{
BwTimetable{
BwTimeSlot{HHMM: 1100, Bandwidth: 333 * 1024},
BwTimeSlot{HHMM: 1300, Bandwidth: 666 * 1024},
BwTimeSlot{HHMM: 2301, Bandwidth: 1024 * 1024},
BwTimeSlot{HHMM: 2350, Bandwidth: -1},
},
time.Date(2017, time.April, 20, 10, 15, 0, 0, time.UTC),
BwTimeSlot{HHMM: 2350, Bandwidth: -1},
},
{
BwTimetable{
BwTimeSlot{HHMM: 1100, Bandwidth: 333 * 1024},
BwTimeSlot{HHMM: 1300, Bandwidth: 666 * 1024},
BwTimeSlot{HHMM: 2301, Bandwidth: 1024 * 1024},
BwTimeSlot{HHMM: 2350, Bandwidth: -1},
},
time.Date(2017, time.April, 20, 11, 0, 0, 0, time.UTC),
BwTimeSlot{HHMM: 1100, Bandwidth: 333 * 1024},
},
{
BwTimetable{
BwTimeSlot{HHMM: 1100, Bandwidth: 333 * 1024},
BwTimeSlot{HHMM: 1300, Bandwidth: 666 * 1024},
BwTimeSlot{HHMM: 2301, Bandwidth: 1024 * 1024},
BwTimeSlot{HHMM: 2350, Bandwidth: -1},
},
time.Date(2017, time.April, 20, 13, 1, 0, 0, time.UTC),
BwTimeSlot{HHMM: 1300, Bandwidth: 666 * 1024},
},
{
BwTimetable{
BwTimeSlot{HHMM: 1100, Bandwidth: 333 * 1024},
BwTimeSlot{HHMM: 1300, Bandwidth: 666 * 1024},
BwTimeSlot{HHMM: 2301, Bandwidth: 1024 * 1024},
BwTimeSlot{HHMM: 2350, Bandwidth: -1},
},
time.Date(2017, time.April, 20, 23, 59, 0, 0, time.UTC),
BwTimeSlot{HHMM: 2350, Bandwidth: -1},
},
} {
slot := test.tt.LimitAt(test.now)
assert.Equal(t, test.want, slot)
}
}

1538
fs/config.go

File diff suppressed because it is too large Load Diff

1159
fs/config/config.go Normal file

File diff suppressed because it is too large Load Diff

2
fs/config_other.go → fs/config/config_other.go

@ -3,7 +3,7 @@
// +build !darwin,!dragonfly,!freebsd,!linux,!netbsd,!openbsd,!solaris
package fs
package config
// attemptCopyGroups tries to keep the group the same, which only makes sense
// for system with user-group-world permission model.

2
fs/config_read_password.go → fs/config/config_read_password.go

@ -4,7 +4,7 @@
// +build !solaris,!plan9
package fs
package config
import (
"fmt"

2
fs/config_read_password_unsupported.go → fs/config/config_read_password_unsupported.go

@ -4,7 +4,7 @@
// +build solaris plan9
package fs
package config
// ReadPassword reads a password with echoing it to the terminal.
func ReadPassword() string {

105
fs/config_test.go → fs/config/config_test.go

@ -1,45 +1,15 @@
package fs
package config
import (
"bytes"
"crypto/rand"
"io/ioutil"
"os"
"testing"
"github.com/ncw/rclone/fs"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestObscure(t *testing.T) {
for _, test := range []struct {
in string
want string
iv string
}{
{"", "YWFhYWFhYWFhYWFhYWFhYQ", "aaaaaaaaaaaaaaaa"},
{"potato", "YWFhYWFhYWFhYWFhYWFhYXMaGgIlEQ", "aaaaaaaaaaaaaaaa"},
{"potato", "YmJiYmJiYmJiYmJiYmJiYp3gcEWbAw", "bbbbbbbbbbbbbbbb"},
} {
cryptRand = bytes.NewBufferString(test.iv)
got, err := Obscure(test.in)
cryptRand = rand.Reader
assert.NoError(t, err)
assert.Equal(t, test.want, got)
recoveredIn, err := Reveal(got)
assert.NoError(t, err)
assert.Equal(t, test.in, recoveredIn, "not bidirectional")
// Now the Must variants
cryptRand = bytes.NewBufferString(test.iv)
got = MustObscure(test.in)
cryptRand = rand.Reader
assert.Equal(t, test.want, got)
recoveredIn = MustReveal(got)
assert.Equal(t, test.in, recoveredIn, "not bidirectional")
}
}
func TestCRUD(t *testing.T) {
configKey = nil // reset password
// create temp config file
@ -54,39 +24,47 @@ func TestCRUD(t *testing.T) {
// temporarily adapt configuration
oldOsStdout := os.Stdout
oldConfigFile := configFile
oldConfig := Config
oldConfigPath := ConfigPath
oldConfig := fs.Config
oldConfigData := configData
oldReadLine := ReadLine
os.Stdout = nil
configFile = &path
Config = &ConfigInfo{}
ConfigPath = path
fs.Config = &fs.ConfigInfo{}
configData = nil
defer func() {
os.Stdout = oldOsStdout
configFile = oldConfigFile
ConfigPath = oldConfigPath
ReadLine = oldReadLine
Config = oldConfig
fs.Config = oldConfig
configData = oldConfigData
}()
LoadConfig()
assert.Equal(t, []string{}, configData.GetSectionList())
// Fake a remote
fs.Register(&fs.RegInfo{Name: "config_test_remote"})
// add new remote
i := 0
ReadLine = func() string {
answers := []string{
"local", // type is local
"1", // yes, disable long filenames
"config_test_remote", // type
"y", // looks good, save
}
i = i + 1
return answers[i-1]
}
NewRemote("test")
assert.Equal(t, []string{"test"}, configData.GetSectionList())
// Reload the config file to workaround this bug
// https://github.com/Unknwon/goconfig/issues/39
configData, err = loadConfigFile()
require.NoError(t, err)
// normal rename, test → asdf
ReadLine = func() string { return "asdf" }
RenameRemote("test")
@ -226,50 +204,3 @@ func hashedKeyCompare(t *testing.T, a, b string, shouldMatch bool) {
assert.NotEqual(t, k1, k2)
}
}
func TestDumpFlagsString(t *testing.T) {
assert.Equal(t, "", DumpFlags(0).String())
assert.Equal(t, "headers", (DumpHeaders).String())
assert.Equal(t, "headers,bodies", (DumpHeaders | DumpBodies).String())
assert.Equal(t, "headers,bodies,requests,responses,auth,filters", (DumpHeaders | DumpBodies | DumpRequests | DumpResponses | DumpAuth | DumpFilters).String())
assert.Equal(t, "headers,Unknown-0x8000", (DumpHeaders | DumpFlags(0x8000)).String())
}
func TestDumpFlagsSet(t *testing.T) {
for _, test := range []struct {
in string
want DumpFlags
wantErr string
}{
{"", DumpFlags(0), ""},
{"bodies", DumpBodies, ""},
{"bodies,headers,auth", DumpBodies | DumpHeaders | DumpAuth, ""},
{"bodies,headers,auth", DumpBodies | DumpHeaders | DumpAuth, ""},
{"headers,bodies,requests,responses,auth,filters", DumpHeaders | DumpBodies | DumpRequests | DumpResponses | DumpAuth | DumpFilters, ""},
{"headers,bodies,unknown,auth", 0, "Unknown dump flag \"unknown\""},
} {
f := DumpFlags(-1)
initial := f
err := f.Set(test.in)
if err != nil {
if test.wantErr == "" {
t.Errorf("Got an error when not expecting one on %q: %v", test.in, err)
} else {
assert.Contains(t, err.Error(), test.wantErr)
}
assert.Equal(t, initial, f, test.want)
} else {
if test.wantErr != "" {
t.Errorf("Got no error when expecting one on %q", test.in)
} else {
assert.Equal(t, test.want, f)
}
}
}
}
func TestDumpFlagsType(t *testing.T) {
f := DumpFlags(0)
assert.Equal(t, "string", f.Type())
}

6
fs/config_unix.go → fs/config/config_unix.go

@ -3,13 +3,15 @@
// +build darwin dragonfly freebsd linux netbsd openbsd solaris
package fs
package config
import (
"os"
"os/user"
"strconv"
"syscall"
"github.com/ncw/rclone/fs"
)
// attemptCopyGroups tries to keep the group the same. User will be the one
@ -29,7 +31,7 @@ func attemptCopyGroup(fromPath, toPath string) {
}
}
if err = os.Chown(toPath, uid, int(stat.Gid)); err != nil {
Debugf(nil, "Failed to keep previous owner of config file: %v", err)
fs.Debugf(nil, "Failed to keep previous owner of config file: %v", err)
}
}
}

162
fs/config/configflags/configflags.go Normal file

@ -0,0 +1,162 @@
// Package configflags defines the flags used by rclone. It is
// decoupled into a separate package so it can be replaced.
package configflags
// Options set by command line flags
import (
"log"
"net"
"path/filepath"
"strings"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/flags"
"github.com/spf13/pflag"
)
var (
// these will get interpreted into fs.Config via SetFlags() below
verbose int
quiet bool
dumpHeaders bool
dumpBodies bool
deleteBefore bool
deleteDuring bool
deleteAfter bool
bindAddr string
disableFeatures string
)
// AddFlags adds the non filing system specific flags to the command
func AddFlags(flagSet *pflag.FlagSet) {
// NB defaults which aren't the zero for the type should be set in fs/config.go NewConfig
flags.CountVarP(flagSet, &verbose, "verbose", "v", "Print lots more stuff (repeat for more)")
flags.BoolVarP(flagSet, &quiet, "quiet", "q", false, "Print as little stuff as possible")
flags.DurationVarP(flagSet, &fs.Config.ModifyWindow, "modify-window", "", fs.Config.ModifyWindow, "Max time diff to be considered the same")
flags.IntVarP(flagSet, &fs.Config.Checkers, "checkers", "", fs.Config.Checkers, "Number of checkers to run in parallel.")
flags.IntVarP(flagSet, &fs.Config.Transfers, "transfers", "", fs.Config.Transfers, "Number of file transfers to run in parallel.")
flags.StringVarP(flagSet, &config.ConfigPath, "config", "", config.ConfigPath, "Config file.")
flags.StringVarP(flagSet, &config.CacheDir, "cache-dir", "", config.CacheDir, "Directory rclone will use for caching.")
flags.BoolVarP(flagSet, &fs.Config.CheckSum, "checksum", "c", fs.Config.CheckSum, "Skip based on checksum & size, not mod-time & size")
flags.BoolVarP(flagSet, &fs.Config.SizeOnly, "size-only", "", fs.Config.SizeOnly, "Skip based on size only, not mod-time or checksum")
flags.BoolVarP(flagSet, &fs.Config.IgnoreTimes, "ignore-times", "I", fs.Config.IgnoreTimes, "Don't skip files that match size and time - transfer all files")
flags.BoolVarP(flagSet, &fs.Config.IgnoreExisting, "ignore-existing", "", fs.Config.IgnoreExisting, "Skip all files that exist on destination")
flags.BoolVarP(flagSet, &fs.Config.DryRun, "dry-run", "n", fs.Config.DryRun, "Do a trial run with no permanent changes")
flags.DurationVarP(flagSet, &fs.Config.ConnectTimeout, "contimeout", "", fs.Config.ConnectTimeout, "Connect timeout")
flags.DurationVarP(flagSet, &fs.Config.Timeout, "timeout", "", fs.Config.Timeout, "IO idle timeout")
flags.BoolVarP(flagSet, &dumpHeaders, "dump-headers", "", false, "Dump HTTP bodies - may contain sensitive info")
flags.BoolVarP(flagSet, &dumpBodies, "dump-bodies", "", false, "Dump HTTP headers and bodies - may contain sensitive info")
flags.BoolVarP(flagSet, &fs.Config.InsecureSkipVerify, "no-check-certificate", "", fs.Config.InsecureSkipVerify, "Do not verify the server SSL certificate. Insecure.")
flags.BoolVarP(flagSet, &fs.Config.AskPassword, "ask-password", "", fs.Config.AskPassword, "Allow prompt for password for encrypted configuration.")
flags.BoolVarP(flagSet, &deleteBefore, "delete-before", "", false, "When synchronizing, delete files on destination before transfering")
flags.BoolVarP(flagSet, &deleteDuring, "delete-during", "", false, "When synchronizing, delete files during transfer (default)")
flags.BoolVarP(flagSet, &deleteAfter, "delete-after", "", false, "When synchronizing, delete files on destination after transfering")
flags.BoolVarP(flagSet, &fs.Config.TrackRenames, "track-renames", "", fs.Config.TrackRenames, "When synchronizing, track file renames and do a server side move if possible")
flags.IntVarP(flagSet, &fs.Config.LowLevelRetries, "low-level-retries", "", fs.Config.LowLevelRetries, "Number of low level retries to do.")
flags.BoolVarP(flagSet, &fs.Config.UpdateOlder, "update", "u", fs.Config.UpdateOlder, "Skip files that are newer on the destination.")
flags.BoolVarP(flagSet, &fs.Config.NoGzip, "no-gzip-encoding", "", fs.Config.NoGzip, "Don't set Accept-Encoding: gzip.")
flags.IntVarP(flagSet, &fs.Config.MaxDepth, "max-depth", "", fs.Config.MaxDepth, "If set limits the recursion depth to this.")
flags.BoolVarP(flagSet, &fs.Config.IgnoreSize, "ignore-size", "", false, "Ignore size when skipping use mod-time or checksum.")
flags.BoolVarP(flagSet, &fs.Config.IgnoreChecksum, "ignore-checksum", "", fs.Config.IgnoreChecksum, "Skip post copy check of checksums.")
flags.BoolVarP(flagSet, &fs.Config.NoTraverse, "no-traverse", "", fs.Config.NoTraverse, "Don't traverse destination file system on copy.")
flags.BoolVarP(flagSet, &fs.Config.NoUpdateModTime, "no-update-modtime", "", fs.Config.NoUpdateModTime, "Don't update destination mod-time if files identical.")
flags.StringVarP(flagSet, &fs.Config.BackupDir, "backup-dir", "", fs.Config.BackupDir, "Make backups into hierarchy based in DIR.")
flags.StringVarP(flagSet, &fs.Config.Suffix, "suffix", "", fs.Config.Suffix, "Suffix for use with --backup-dir.")
flags.BoolVarP(flagSet, &fs.Config.UseListR, "fast-list", "", fs.Config.UseListR, "Use recursive list if available. Uses more memory but fewer transactions.")
flags.Float64VarP(flagSet, &fs.Config.TPSLimit, "tpslimit", "", fs.Config.TPSLimit, "Limit HTTP transactions per second to this.")
flags.IntVarP(flagSet, &fs.Config.TPSLimitBurst, "tpslimit-burst", "", fs.Config.TPSLimitBurst, "Max burst of transactions for --tpslimit.")
flags.StringVarP(flagSet, &bindAddr, "bind", "", "", "Local address to bind to for outgoing connections, IPv4, IPv6 or name.")
flags.StringVarP(flagSet, &disableFeatures, "disable", "", "", "Disable a comma separated list of features. Use help to see a list.")
flags.StringVarP(flagSet, &fs.Config.UserAgent, "user-agent", "", fs.Config.UserAgent, "Set the user-agent to a specified string. The default is rclone/ version")
flags.BoolVarP(flagSet, &fs.Config.Immutable, "immutable", "", fs.Config.Immutable, "Do not modify files. Fail if existing files have been modified.")
flags.BoolVarP(flagSet, &fs.Config.AutoConfirm, "auto-confirm", "", fs.Config.AutoConfirm, "If enabled, do not request console confirmation.")
flags.IntVarP(flagSet, &fs.Config.StatsFileNameLength, "stats-file-name-length", "", fs.Config.StatsFileNameLength, "Max file name length in stats. 0 for no limit")
flags.FVarP(flagSet, &fs.Config.LogLevel, "log-level", "", "Log level DEBUG|INFO|NOTICE|ERROR")
flags.FVarP(flagSet, &fs.Config.StatsLogLevel, "stats-log-level", "", "Log level to show --stats output DEBUG|INFO|NOTICE|ERROR")
flags.FVarP(flagSet, &fs.Config.BwLimit, "bwlimit", "", "Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable.")
flags.FVarP(flagSet, &fs.Config.BufferSize, "buffer-size", "", "Buffer size when copying files.")
flags.FVarP(flagSet, &fs.Config.StreamingUploadCutoff, "streaming-upload-cutoff", "", "Cutoff for switching to chunked upload if file size is unknown. Upload starts after reaching cutoff or when file ends.")
flags.FVarP(flagSet, &fs.Config.Dump, "dump", "", "List of items to dump from: "+fs.DumpFlagsList)
}
// SetFlags converts any flags into config which weren't straight foward
func SetFlags() {
fs.Config.LogLevel = fs.LogLevelNotice
if verbose >= 2 {
fs.Config.LogLevel = fs.LogLevelDebug
} else if verbose >= 1 {
fs.Config.LogLevel = fs.LogLevelInfo
}
if quiet {
if verbose > 0 {
log.Fatalf("Can't set -v and -q")
}
fs.Config.LogLevel = fs.LogLevelError
}
logLevelFlag := pflag.Lookup("log-level")
if logLevelFlag != nil && logLevelFlag.Changed {
if verbose > 0 {
log.Fatalf("Can't set -v and --log-level")
}
if quiet {
log.Fatalf("Can't set -q and --log-level")
}
}
if dumpHeaders {
fs.Config.Dump |= fs.DumpHeaders
fs.Infof(nil, "--dump-headers is obsolete - please use --dump headers instead")
}
if dumpBodies {
fs.Config.Dump |= fs.DumpBodies
fs.Infof(nil, "--dump-bodies is obsolete - please use --dump bodies instead")
}
switch {
case deleteBefore && (deleteDuring || deleteAfter),
deleteDuring && deleteAfter:
log.Fatalf(`Only one of --delete-before, --delete-during or --delete-after can be used.`)
case deleteBefore:
fs.Config.DeleteMode = fs.DeleteModeBefore
case deleteDuring:
fs.Config.DeleteMode = fs.DeleteModeDuring
case deleteAfter:
fs.Config.DeleteMode = fs.DeleteModeAfter
default:
fs.Config.DeleteMode = fs.DeleteModeDefault
}
if fs.Config.IgnoreSize && fs.Config.SizeOnly {
log.Fatalf(`Can't use --size-only and --ignore-size together.`)
}
if fs.Config.Suffix != "" && fs.Config.BackupDir == "" {
log.Fatalf(`Can only use --suffix with --backup-dir.`)
}
if bindAddr != "" {
addrs, err := net.LookupIP(bindAddr)
if err != nil {
log.Fatalf("--bind: Failed to parse %q as IP address: %v", bindAddr, err)
}
if len(addrs) != 1 {
log.Fatalf("--bind: Expecting 1 IP address for %q but got %d", bindAddr, len(addrs))
}
fs.Config.BindAddr = addrs[0]
}
if disableFeatures != "" {
if disableFeatures == "help" {
log.Fatalf("Possible backend features are: %s\n", strings.Join(new(fs.Features).List(), ", "))
}
fs.Config.DisableFeatures = strings.Split(disableFeatures, ",")
}
// Make the config file absolute
configPath, err := filepath.Abs(config.ConfigPath)
if err == nil {
config.ConfigPath = configPath
}
}

245
fs/flags.go → fs/config/flags/flags.go

@ -1,239 +1,17 @@
// This contains helper functions for managing flags
package fs
// Package flags contains enahnced versions of spf13/pflag flag
// routines which will read from the environment also.
package flags
import (
"fmt"
"log"
"math"
"os"
"strconv"
"strings"
"time"
"github.com/pkg/errors"
"github.com/ncw/rclone/fs"
"github.com/spf13/pflag"
)
// SizeSuffix is parsed by flag with k/M/G suffixes
type SizeSuffix int64
// Turn SizeSuffix into a string and a suffix
func (x SizeSuffix) string() (string, string) {
scaled := float64(0)
suffix := ""
switch {
case x < 0:
return "off", ""
case x == 0:
return "0", ""
case x < 1024:
scaled = float64(x)
suffix = ""
case x < 1024*1024:
scaled = float64(x) / 1024
suffix = "k"
case x < 1024*1024*1024:
scaled = float64(x) / 1024 / 1024
suffix = "M"
default:
scaled = float64(x) / 1024 / 1024 / 1024
suffix = "G"
}
if math.Floor(scaled) == scaled {
return fmt.Sprintf("%.0f", scaled), suffix
}
return fmt.Sprintf("%.3f", scaled), suffix
}
// String turns SizeSuffix into a string
func (x SizeSuffix) String() string {
val, suffix := x.string()
return val + suffix
}
// Unit turns SizeSuffix into a string with a unit
func (x SizeSuffix) Unit(unit string) string {
val, suffix := x.string()
if val == "off" {
return val
}
return val + " " + suffix + unit
}
// Set a SizeSuffix
func (x *SizeSuffix) Set(s string) error {
if len(s) == 0 {
return errors.New("empty string")
}
if strings.ToLower(s) == "off" {
*x = -1
return nil
}
suffix := s[len(s)-1]
suffixLen := 1
var multiplier float64
switch suffix {
case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '.':
suffixLen = 0
multiplier = 1 << 10
case 'b', 'B':
multiplier = 1
case 'k', 'K':
multiplier = 1 << 10
case 'm', 'M':
multiplier = 1 << 20
case 'g', 'G':
multiplier = 1 << 30
default:
return errors.Errorf("bad suffix %q", suffix)
}
s = s[:len(s)-suffixLen]
value, err := strconv.ParseFloat(s, 64)
if err != nil {
return err
}
if value < 0 {
return errors.Errorf("size can't be negative %q", s)
}
value *= multiplier
*x = SizeSuffix(value)
return nil
}
// Type of the value
func (x *SizeSuffix) Type() string {
return "int64"
}
// Check it satisfies the interface
var _ pflag.Value = (*SizeSuffix)(nil)
// BwTimeSlot represents a bandwidth configuration at a point in time.
type BwTimeSlot struct {
hhmm int
bandwidth SizeSuffix
}
// BwTimetable contains all configured time slots.
type BwTimetable []BwTimeSlot
// String returns a printable representation of BwTimetable.
func (x BwTimetable) String() string {
ret := []string{}
for _, ts := range x {
ret = append(ret, fmt.Sprintf("%04.4d,%s", ts.hhmm, ts.bandwidth.String()))
}
return strings.Join(ret, " ")
}
// Set the bandwidth timetable.
func (x *BwTimetable) Set(s string) error {
// The timetable is formatted as:
// "hh:mm,bandwidth hh:mm,banwidth..." ex: "10:00,10G 11:30,1G 18:00,off"
// If only a single bandwidth identifier is provided, we assume constant bandwidth.
if len(s) == 0 {
return errors.New("empty string")
}
// Single value without time specification.
if !strings.Contains(s, " ") && !strings.Contains(s, ",") {
ts := BwTimeSlot{}
if err := ts.bandwidth.Set(s); err != nil {
return err
}
ts.hhmm = 0
*x = BwTimetable{ts}
return nil
}
for _, tok := range strings.Split(s, " ") {
tv := strings.Split(tok, ",")
// Format must be HH:MM,BW
if len(tv) != 2 {
return errors.Errorf("invalid time/bandwidth specification: %q", tok)
}
// Basic timespec sanity checking
hhmm := tv[0]
if len(hhmm) != 5 {
return errors.Errorf("invalid time specification (hh:mm): %q", hhmm)
}
hh, err := strconv.Atoi(hhmm[0:2])
if err != nil {
return errors.Errorf("invalid hour in time specification %q: %v", hhmm, err)
}
if hh < 0 || hh > 23 {
return errors.Errorf("invalid hour (must be between 00 and 23): %q", hh)
}
mm, err := strconv.Atoi(hhmm[3:])
if err != nil {
return errors.Errorf("invalid minute in time specification: %q: %v", hhmm, err)
}
if mm < 0 || mm > 59 {
return errors.Errorf("invalid minute (must be between 00 and 59): %q", hh)
}
ts := BwTimeSlot{
hhmm: (hh * 100) + mm,
}
// Bandwidth limit for this time slot.
if err := ts.bandwidth.Set(tv[1]); err != nil {
return err
}
*x = append(*x, ts)
}
return nil
}
// LimitAt returns a BwTimeSlot for the time requested.
func (x BwTimetable) LimitAt(tt time.Time) BwTimeSlot {
// If the timetable is empty, we return an unlimited BwTimeSlot starting at midnight.
if len(x) == 0 {
return BwTimeSlot{hhmm: 0, bandwidth: -1}
}
hhmm := tt.Hour()*100 + tt.Minute()
// By default, we return the last element in the timetable. This
// satisfies two conditions: 1) If there's only one element it
// will always be selected, and 2) The last element of the table
// will "wrap around" until overriden by an earlier time slot.
// there's only one time slot in the timetable.
ret := x[len(x)-1]
mindif := 0
first := true
// Look for most recent time slot.
for _, ts := range x {
// Ignore the past
if hhmm < ts.hhmm {
continue
}
dif := ((hhmm / 100 * 60) + (hhmm % 100)) - ((ts.hhmm / 100 * 60) + (ts.hhmm % 100))
if first {
mindif = dif
first = false
}
if dif <= mindif {
mindif = dif
ret = ts
}
}
return ret
}
// Type of the value
func (x BwTimetable) Type() string {
return "BwTimetable"
}
// Check it satisfies the interface
var _ pflag.Value = (*BwTimetable)(nil)
// optionToEnv converts an option name, eg "ignore-size" into an
// environment name "RCLONE_IGNORE_SIZE"
func optionToEnv(name string) string {
@ -254,7 +32,7 @@ func setDefaultFromEnv(name string) {
if err != nil {
log.Fatalf("Invalid value for environment variable %q: %v", key, err)
}
Debugf(nil, "Set default for %q from %q to %q (%v)", name, key, newValue, flag.Value)
fs.Debugf(nil, "Set default for %q from %q to %q (%v)", name, key, newValue, flag.Value)
flag.DefValue = newValue
}
}
@ -302,6 +80,15 @@ func IntP(name, shorthand string, value int, usage string) (out *int) {
return out
}
// Int64P defines a flag which can be overridden by an environment variable
//
// It is a thin wrapper around pflag.IntP
func Int64P(name, shorthand string, value int64, usage string) (out *int64) {
out = pflag.Int64P(name, shorthand, value, usage)
setDefaultFromEnv(name)
return out
}
// IntVarP defines a flag which can be overridden by an environment variable
//
// It is a thin wrapper around pflag.IntVarP
@ -360,10 +147,10 @@ func VarP(value pflag.Value, name, shorthand, usage string) {
setDefaultFromEnv(name)
}
// FlagsVarP defines a flag which can be overridden by an environment variable
// FVarP defines a flag which can be overridden by an environment variable
//
// It is a thin wrapper around pflag.VarP
func FlagsVarP(flags *pflag.FlagSet, value pflag.Value, name, shorthand, usage string) {
func FVarP(flags *pflag.FlagSet, value pflag.Value, name, shorthand, usage string) {
flags.VarP(value, name, shorthand, usage)
setDefaultFromEnv(name)
}

95
fs/config/obscure.go Normal file

@ -0,0 +1,95 @@
// Obscure and Reveal config values
package config
import (
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"encoding/base64"
"io"
"log"
"github.com/pkg/errors"
)
// crypt internals
var (
cryptKey = []byte{
0x9c, 0x93, 0x5b, 0x48, 0x73, 0x0a, 0x55, 0x4d,
0x6b, 0xfd, 0x7c, 0x63, 0xc8, 0x86, 0xa9, 0x2b,
0xd3, 0x90, 0x19, 0x8e, 0xb8, 0x12, 0x8a, 0xfb,
0xf4, 0xde, 0x16, 0x2b, 0x8b, 0x95, 0xf6, 0x38,
}
cryptBlock cipher.Block
cryptRand = rand.Reader
)
// crypt transforms in to out using iv under AES-CTR.
//
// in and out may be the same buffer.
//
// Note encryption and decryption are the same operation
func crypt(out, in, iv []byte) error {
if cryptBlock == nil {
var err error
cryptBlock, err = aes.NewCipher(cryptKey)
if err != nil {
return err
}
}
stream := cipher.NewCTR(cryptBlock, iv)
stream.XORKeyStream(out, in)
return nil
}
// Obscure a value
//
// This is done by encrypting with AES-CTR
func Obscure(x string) (string, error) {
plaintext := []byte(x)
ciphertext := make([]byte, aes.BlockSize+len(plaintext))
iv := ciphertext[:aes.BlockSize]
if _, err := io.ReadFull(cryptRand, iv); err != nil {
return "", errors.Wrap(err, "failed to read iv")
}
if err := crypt(ciphertext[aes.BlockSize:], plaintext, iv); err != nil {
return "", errors.Wrap(err, "encrypt failed")
}
return base64.RawURLEncoding.EncodeToString(ciphertext), nil
}
// MustObscure obscures a value, exiting with a fatal error if it failed
func MustObscure(x string) string {
out, err := Obscure(x)
if err != nil {
log.Fatalf("Obscure failed: %v", err)
}
return out
}
// Reveal an obscured value
func Reveal(x string) (string, error) {
ciphertext, err := base64.RawURLEncoding.DecodeString(x)
if err != nil {
return "", errors.Wrap(err, "base64 decode failed when revealing password - is it obscured?")
}
if len(ciphertext) < aes.BlockSize {
return "", errors.New("input too short when revealing password - is it obscured?")
}
buf := ciphertext[aes.BlockSize:]
iv := ciphertext[:aes.BlockSize]
if err := crypt(buf, buf, iv); err != nil {
return "", errors.Wrap(err, "decrypt failed when revealing password - is it obscured?")
}
return string(buf), nil
}
// MustReveal reveals an obscured value, exiting with a fatal error if it failed
func MustReveal(x string) string {
out, err := Reveal(x)
if err != nil {
log.Fatalf("Reveal failed: %v", err)
}
return out
}

38
fs/config/obscure_test.go Normal file

@ -0,0 +1,38 @@
package config
import (
"bytes"
"crypto/rand"
"testing"
"github.com/stretchr/testify/assert"
)
func TestObscure(t *testing.T) {
for _, test := range []struct {
in string
want string
iv string
}{
{"", "YWFhYWFhYWFhYWFhYWFhYQ", "aaaaaaaaaaaaaaaa"},
{"potato", "YWFhYWFhYWFhYWFhYWFhYXMaGgIlEQ", "aaaaaaaaaaaaaaaa"},
{"potato", "YmJiYmJiYmJiYmJiYmJiYp3gcEWbAw", "bbbbbbbbbbbbbbbb"},
} {
cryptRand = bytes.NewBufferString(test.iv)
got, err := Obscure(test.in)
cryptRand = rand.Reader
assert.NoError(t, err)
assert.Equal(t, test.want, got)
recoveredIn, err := Reveal(got)
assert.NoError(t, err)
assert.Equal(t, test.in, recoveredIn, "not bidirectional")
// Now the Must variants
cryptRand = bytes.NewBufferString(test.iv)
got = MustObscure(test.in)
cryptRand = rand.Reader
assert.Equal(t, test.want, got)
recoveredIn = MustReveal(got)
assert.Equal(t, test.in, recoveredIn, "not bidirectional")
}
}

0
fs/testdata/enc-invalid.conf → fs/config/testdata/enc-invalid.conf vendored

0
fs/testdata/enc-short.conf → fs/config/testdata/enc-short.conf vendored

Some files were not shown because too many files have changed in this diff Show More