s3: add docs on how to add a new provider

This commit is contained in:
Nick Craig-Wood 2023-09-21 12:35:23 +01:00
parent eb3082a1eb
commit 9e80d48b03
2 changed files with 52 additions and 7 deletions

View File

@ -484,6 +484,46 @@ alphabetical order of full name of remote (e.g. `drive` is ordered as
Once you've written the docs, run `make serve` and check they look OK
in the web browser and the links (internal and external) all work.
## Adding a new s3 provider
It is quite easy to add a new S3 provider to rclone.
You'll need to modify the following files
- `backend/s3/s3.go`
- Add the provider to `providerOption` at the top of the file
- Add endpoints and other config for your provider gated on the provider in `fs.RegInfo`.
- Exclude your provider from genric config questions (eg `region` and `endpoint).
- Add the provider to the `setQuirks` function - see the documentation there.
- `docs/content/s3.md`
- Add the provider at the top of the page.
- Add a section about the provider linked from there.
- Add a transcript of a trial `rclone config` session
- Edit the transcript to remove things which might change in subsequent versions
- **Do not** alter or add to the autogenerated parts of `s3.md`
- **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3`
- `README.md` - this is the home page in github
- Add the provider and a link to the section you wrote in `docs/contents/s3.md`
- `docs/content/_index.md` - this is the home page of rclone.org
- Add the provider and a link to the section you wrote in `docs/contents/s3.md`
When adding the provider, endpoints, quirks, docs etc keep them in
alphabetical order by `Provider` name, but with `AWS` first and
`Other` last.
Once you've written the docs, run `make serve` and check they look OK
in the web browser and the links (internal and external) all work.
Once you've written the code, test `rclone config` works to your
satisfaction, and check the integration tests work `go test -v -remote
NewS3Provider:`. You may need to adjust the quirks to get them to
pass. Some providers just can't pass the tests with control characters
in the names so if these fail and the provider doesn't support
`urlEncodeListings` in the quirks then ignore them. Note that the
`SetTier` test may also fail on non AWS providers.
For an example of adding an s3 provider see [eb3082a1](https://github.com/rclone/rclone/commit/eb3082a1ebdb76d5625f14cedec3f5154a5e7b10).
## Writing a plugin ##
New features (backends, commands) can also be added "out-of-tree", through Go plugins.

View File

@ -70,6 +70,9 @@ import (
// NB if you add a new provider here, then add it in the setQuirks
// function and set the correct quirks. Test the quirks are correct by
// running the integration tests "go test -v -remote NewS3Provider:".
//
// See https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#adding-a-new-s3-provider
// for full information about how to add a new s3 provider.
var providerOption = fs.Option{
Name: fs.ConfigProvider,
Help: "Choose your S3 provider.",
@ -2998,15 +3001,17 @@ func setEndpointValueForIDriveE2(m configmap.Mapper) (err error) {
// There should be no testing against opt.Provider anywhere in the
// code except in here to localise the setting of the quirks.
//
// These should be differences from AWS S3
// Run the integration tests to check you have the quirks correct.
//
// go test -v -remote NewS3Provider:
func setQuirks(opt *Options) {
var (
listObjectsV2 = true
virtualHostStyle = true
urlEncodeListings = true
useMultipartEtag = true
useAcceptEncodingGzip = true
mightGzip = true // assume all providers might gzip until proven otherwise
listObjectsV2 = true // Always use ListObjectsV2 instead of ListObjects
virtualHostStyle = true // Use bucket.provider.com instead of putting the bucket in the URL
urlEncodeListings = true // URL encode the listings to help with control characters
useMultipartEtag = true // Set if Etags for multpart uploads are compatible with AWS
useAcceptEncodingGzip = true // Set Accept-Encoding: gzip
mightGzip = true // assume all providers might use content encoding gzip until proven otherwise
)
switch opt.Provider {
case "AWS":