This change to the S3 Move method uses S3's multipart upload API to copy
objects whose size exceeds a threshold. Parts are copied concurrently.
The level of concurrency, part size, and threshold are all configurable
with reasonable defaults.
Using the multipart upload API has two benefits.
* The S3 Move method can now handle objects over 5 GB, fixing #886.
* Moving most objects, and espectially large ones, is faster. For
example, moving a 1 GB object averaged 30 seconds but now averages 10.
Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
This is already supported by ncw/swift, so we just need to pass the
parameters from the storage driver.
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
* Add Object ACL Support to the S3 Storage Backend
Signed-off-by: Frank Chen <frankchn@gmail.com>
* Made changes per @RichardScothern's comments
Signed-off-by: Frank Chen <frankchn@gmail.com>
* Fix Typos
Signed-off-by: Frank Chen <frankchn@gmail.com>
- Fixing index metadata error
- Entering all the comments
- Updating with the build fixes
- Fix Microsoft link broken
- Fix collocate, colocate, to unambiguous co-locate
Signed-off-by: Mary Anthony <mary@docker.com>
Since the CloudFront middleware does not work without an S3 backend, it
became obvious that this documentation should exist within the S3
storage-driver documentation.
Signed-off-by: DJ Enriquez <dj.enriquez@infospace.com>
This page was missing styling once exported to HTML. Adding a
<!--[metadata]> block similar to the ones the other *.md files have
appears to solve the problem.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Renaming to index.md;rereading of Hugo showed me my mistake; removing commented out/Markdown has no comment feature
Updating with Olivier. Yay! It looks great
Signed-off-by: Mary Anthony <mary@docker.com>
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.
I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...
This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.
Fixes#620, #175.
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
Updating for tooling tests
Updating with the new sed scripts to protect links
updating with new image
Updating with comments
Signed-off-by: Mary Anthony <mary@docker.com>
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.
Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:
* All the objects data are stored with opaque UUID names prefixed
(e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
omap object identifier is the virtual directory name, the keys in
a specific are the relative filenames and the values the blob
object identifier (or empty value for a sub directory).
e.g. For the following hierarchy:
/directory1
/directory1/object1
/directory1/object2
/directory1/directory2/object3
The omap "/directory1" will contains the following key / values:
- "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
- "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
- "directory2" ""
The omap "/directory1/directory2" will contains:
- "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"
* The MOVE is implemented by changing the reference to a specific
blob in its parent virtual directory omap.
This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
