vbatts/registry
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

Use multipart upload API in S3 Move method

Browse source 
This change to the S3 Move method uses S3's multipart upload API to copy
objects whose size exceeds a threshold.  Parts are copied concurrently.
The level of concurrency, part size, and threshold are all configurable
with reasonable defaults.

Using the multipart upload API has two benefits.

* The S3 Move method can now handle objects over 5 GB, fixing #886.

* Moving most objects, and espectially large ones, is faster.  For
  example, moving a 1 GB object averaged 30 seconds but now averages 10.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
This commit is contained in:
Noah Treuhaft 2016-08-15 17:12:24 -07:00
parent c810308d1b
commit 63468ef4a8
4 changed files with 305 additions and 61 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/configuration.md
View file

@ -99,6 +99,9 @@ information about each option that appears later in this page.
secure: true
v4auth: true
chunksize: 5242880
multipartcopychunksize: 33554432
multipartcopymaxconcurrency: 100
multipartcopythresholdsize: 33554432
rootdirectory: /s3/object/name/prefix
swift:
username: username
@ -380,6 +383,9 @@ Permitted values are `error`, `warn`, `info` and `debug`. The default is
secure: true
v4auth: true
chunksize: 5242880
multipartcopychunksize: 33554432
multipartcopymaxconcurrency: 100
multipartcopythresholdsize: 33554432
rootdirectory: /s3/object/name/prefix
swift:
username: username