8.7 KiB
| title | description | date |
|---|---|---|
| B2 | Backblaze B2 | 2016-10-25 |
Backblaze B2
B2 is Backblaze's cloud storage system.
Paths are specified as remote:bucket (or remote: for the lsd
command.) You may put subdirectories in too, eg remote:bucket/path/to/dir.
Here is an example of making a b2 configuration. First run
rclone config
This will guide you through an interactive setup process. You will need your account number (a short hex number) and key (a long hex number) which you can get from the b2 control panel.
No remotes found - make a new one
n) New remote
q) Quit config
n/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
1 / Amazon Drive
\ "amazon cloud drive"
2 / Amazon S3 (also Dreamhost, Ceph, Minio)
\ "s3"
3 / Backblaze B2
\ "b2"
4 / Dropbox
\ "dropbox"
5 / Encrypt/Decrypt a remote
\ "crypt"
6 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
7 / Google Drive
\ "drive"
8 / Hubic
\ "hubic"
9 / Local Disk
\ "local"
10 / Microsoft OneDrive
\ "onedrive"
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
12 / SSH/SFTP Connection
\ "sftp"
13 / Yandex Disk
\ "yandex"
Storage> 3
Account ID
account> 123456789abc
Application Key
key> 0123456789abcdef0123456789abcdef0123456789
Endpoint for the service - leave blank normally.
endpoint>
Remote config
--------------------
[remote]
account = 123456789abc
key = 0123456789abcdef0123456789abcdef0123456789
endpoint =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
This remote is called remote and can now be used like this
See all buckets
rclone lsd remote:
Make a new bucket
rclone mkdir remote:bucket
List the contents of a bucket
rclone ls remote:bucket
Sync /home/local/directory to the remote bucket, deleting any
excess files in the bucket.
rclone sync /home/local/directory remote:bucket
--fast-list
This remote supports --fast-list which allows you to use fewer
transactions in exchange for more memory. See the rclone
docs for more details.
Modified time
The modified time is stored as metadata on the object as
X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01
in the Backblaze standard. Other tools should be able to use this as
a modified time.
Modified times are used in syncing and are fully supported except in the case of updating a modification time on an existing object. In this case the object will be uploaded again as B2 doesn't have an API method to set the modification time independent of doing an upload.
SHA1 checksums
The SHA1 checksums of the files are checked on upload and download and will be used in the syncing process.
Large files which are uploaded in chunks will store their SHA1 on the
object as X-Bz-Info-large_file_sha1 as recommended by Backblaze.
Transfers
Backblaze recommends that you do lots of transfers simultaneously for
maximum speed. In tests from my SSD equiped laptop the optimum
setting is about --transfers 32 though higher numbers may be used
for a slight speed improvement. The optimum number for you may vary
depending on your hardware, how big the files are, how much you want
to load your computer, etc. The default of --transfers 4 is
definitely too low for Backblaze B2 though.
Note that uploading big files (bigger than 200 MB by default) will use
a 96 MB RAM buffer by default. There can be at most --transfers of
these in use at any moment, so this sets the upper limit on the memory
used.
Versions
When rclone uploads a new version of a file it creates a new version
of it.
Likewise when you delete a file, the old version will be marked hidden
and still be available. Conversely, you may opt in to a "hard delete"
of files with the --b2-hard-delete flag which would permanently remove
the file instead of hiding it.
Old versions of files, where available, are visible using the
--b2-versions flag.
If you wish to remove all the old versions then you can use the
rclone cleanup remote:bucket command which will delete all the old
versions of files, leaving the current ones intact. You can also
supply a path and only old versions under that path will be deleted,
eg rclone cleanup remote:bucket/path/to/stuff.
When you purge a bucket, the current and the old versions will be
deleted then the bucket will be deleted.
However delete will cause the current versions of the files to
become hidden old versions.
Here is a session showing the listing and and retreival of an old
version followed by a cleanup of the old versions.
Show current version and all the versions with --b2-versions flag.
$ rclone -q ls b2:cleanup-test
9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
8 one-v2016-07-04-141032-000.txt
16 one-v2016-07-04-141003-000.txt
15 one-v2016-07-02-155621-000.txt
Retreive an old verson
$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
Clean up all the old versions and show that they've gone.
$ rclone -q cleanup b2:cleanup-test
$ rclone -q ls b2:cleanup-test
9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
Data usage
It is useful to know how many requests are sent to the server in different scenarios.
All copy commands send the following 4 requests:
/b2api/v1/b2_authorize_account
/b2api/v1/b2_create_bucket
/b2api/v1/b2_list_buckets
/b2api/v1/b2_list_file_names
The b2_list_file_names request will be sent once for every 1k files
in the remote path, providing the checksum and modification time of
the listed files. As of version 1.33 issue
#818 causes extra requests
to be sent when using B2 with Crypt. When a copy operation does not
require any files to be uploaded, no more requests will be sent.
Uploading files that do not require chunking, will send 2 requests per file upload:
/b2api/v1/b2_get_upload_url
/b2api/v1/b2_upload_file/
Uploading files requiring chunking, will send 2 requests (one each to start and finish the upload) and another 2 requests for each chunk:
/b2api/v1/b2_start_large_file
/b2api/v1/b2_get_upload_part_url
/b2api/v1/b2_upload_part/
/b2api/v1/b2_finish_large_file
B2 with crypt
When using B2 with crypt files are encrypted into a temporary
location and streamed from there. This is required to calculate the
encrypted file's checksum before beginning the upload. On Windows the
%TMPDIR% environment variable is used as the temporary location. If
the file requires chunking, both the chunking and encryption will take
place in memory.
Specific options
Here are the command line options specific to this cloud storage system.
--b2-chunk-size valuee=SIZE
When uploading large files chunk the file into this size. Note that
these chunks are buffered in memory and there might a maximum of
--transfers chunks in progress at once. 5,000,000 Bytes is the
minimim size (default 96M).
--b2-upload-cutoff=SIZE
Cutoff for switching to chunked upload (default 190.735 MiB == 200
MB). Files above this size will be uploaded in chunks of
--b2-chunk-size.
This value should be set no larger than 4.657GiB (== 5GB) as this is the largest file size that can be uploaded.
--b2-test-mode=FLAG
This is for debugging purposes only.
Setting FLAG to one of the strings below will cause b2 to return specific errors for debugging purposes.
fail_some_uploadsexpire_some_account_authorization_tokensforce_cap_exceeded
These will be set in the X-Bz-Test-Mode header which is documented
in the b2 integrations
checklist.
--b2-versions
When set rclone will show and act on older versions of files. For example
Listing without --b2-versions
$ rclone -q ls b2:cleanup-test
9 one.txt
And with
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
8 one-v2016-07-04-141032-000.txt
16 one-v2016-07-04-141003-000.txt
15 one-v2016-07-02-155621-000.txt
Showing that the current version is unchanged but older versions can be seen. These have the UTC date that they were uploaded to the server to the nearest millisecond appended to them.
Note that when using --b2-versions no file write operations are
permitted, so you can't upload files or delete them.