Commit graph

511 commits

Author SHA1 Message Date
Arthur Baars
7dee3d19d9 Storage: remove bufferedFileWriter (dead code)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:08 +00:00
Arthur Baars
586b3d47a7 Storage: blobwriter.Write/Seek test case
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:07 +00:00
Richard Scothern
6a248e115b Merge pull request #1321 from aibaars/gcs-fix-retry
GCS storage driver: fix retry function
2016-01-06 12:00:12 -08:00
Richard Scothern
e28a924e0c Merge pull request #1323 from aibaars/gcs-fix-list-empty
GCS Storagedriver: fix test failure caused by #1187
2016-01-06 11:18:34 -08:00
Arthur Baars
5c6fdc710f GCS Storagedriver: fix test failure caused by #1187
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 18:36:28 +00:00
Arthur Baars
bf1e41a9f2 GCS driver: fix retry function
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 17:57:22 +00:00
Stephen Day
5529baa918 Merge pull request #1249 from k4leung4/print-error-msg
Print error for failed HTTP auth request.
2016-01-05 12:23:05 -08:00
Stephen Day
1c6c0c8a2d Merge pull request #1097 from hopkings2008/master
use the scheme and host from x-forward-proto and x-forward-host if the…
2015-12-30 16:35:20 -08:00
Stephen Day
e4d08a35ca Merge pull request #1304 from BrianBland/fsDriverRelaxedPermissions
Relaxes filesystem driver permissions to 0777 (dirs) and 0666 (files)
2015-12-29 19:06:45 -08:00
Kenny Leung
731befec93 Merge branch 'print-error-msg' of https://github.com/k4leung4/distribution into print-error-msg
Changed to use typed error instead of formatted string.
Added tests for new public method.

Signed-off-by: Kenny Leung <kleung@google.com>
2015-12-29 16:26:13 -08:00
Brian Bland
2394deaedc Merge pull request #1303 from BrianBland/fixDisabledStorageRedirects
Serve blobs when a storage driver supports redirects but are disabled
2015-12-29 13:28:45 -08:00
Kenny Leung
b89c4e8cbf Print error for failed HTTP auth request.
Signed-off-by: Kenny Leung <kleung@google.com>
2015-12-29 11:39:20 -08:00
Brian Bland
165507a622 Relaxes filesystem driver permissions to 0777 (dirs) and 0666 (files)
Leaves any further permissions restrictions to the process umask

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-28 15:22:28 -08:00
Brian Bland
cf4fdc1be0 Serve blobs when a storage driver supports redirects but are disabled
Fixes issue where an error was returned instead of serving the blob

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-28 11:08:39 -08:00
yuzou
9c7dc47d80 use the scheme and host from x-forward-proto and x-forward-host if they exits and correct the scheme for Location header during image upload
Signed-off-by: yuzou <zouyu7@huawei.com>
2015-12-28 17:28:32 +08:00
Richard Scothern
13b56c9d20 Merge pull request #1268 from RichardScothern/manifest-refactor-impl
Implementation of the Manifest Service API refactor.
2015-12-17 17:32:55 -08:00
Richard Scothern
8efb9ca329 Implementation of the Manifest Service API refactor.
Add a generic Manifest interface to represent manifests in the registry and
remove references to schema specific manifests.

Add a ManifestBuilder to construct Manifest objects. Concrete manifest builders
will exist for each manifest type and implementations will contain manifest
specific data used to build a manifest.

Remove Signatures() from Repository interface.

Signatures are relevant only to schema1 manifests.  Move access to the signature
store inside the schema1 manifestStore.  Add some API tests to verify
signature roundtripping.

schema1
-------

Change the way data is stored in schema1.Manifest to enable Payload() to be used
to return complete Manifest JSON from the HTTP handler without knowledge of the
schema1 protocol.

tags
----

Move tag functionality to a seperate TagService and update ManifestService
to use the new interfaces.  Implement a driver based tagService to be backward
compatible with the current tag service.

Add a proxyTagService to enable the registry to get a digest for remote manifests
from a tag.

manifest store
--------------

Remove revision store and move all signing functionality into the signed manifeststore.

manifest registration
---------------------

Add a mechanism to register manifest media types and to allow different manifest
types to be Unmarshalled correctly.

client
------

Add ManifestServiceOptions to client functions to allow tags to be passed into Put and
Get for building correct registry URLs.  Change functional arguments to be an interface type
to allow passing data without mutating shared state.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2015-12-17 17:09:14 -08:00
Stephen Day
f72093134f Merge pull request #1257 from aaronlehmann/remove-extra-stat
Remove unnecessary stat from blob Get method
2015-12-17 14:10:12 -08:00
Stephen Day
0fef25389d Merge pull request #1271 from aaronlehmann/remove-tarsum
Remove tarsum support for digest package
2015-12-16 14:24:11 -08:00
Aaron Lehmann
a077202f88 Remove tarsum support for digest package
tarsum is not actually used by the registry. Remove support for it.

Convert numerous uses in unit tests to SHA256.

Update docs to remove mentions of tarsums (which were often inaccurate).

Remove tarsum dependency.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-15 17:22:18 -08:00
Aaron Lehmann
58232e50cf Simplify digest.FromBytes calling convention
The current implementation of digest.FromBytes returns an error. This
error can never be non-nil, but its presence in the function signature
means each call site needs error handling code for an error that is
always nil.

I verified that none of the hash.Hash implementations in the standard
library can return an error on Write. Nor can any of the hash.Hash
implementations vendored in distribution.

This commit changes digest.FromBytes not to return an error. If Write
returns an error, it will panic, but as discussed above, this should
never happen.

This commit also avoids using a bytes.Reader to feed data into the hash
function in FromBytes. This makes the hypothetical case that would panic
a bit more explicit, and should also be more performant.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-14 14:30:51 -08:00
Aaron Lehmann
4ebaacfcda Remove unnecessary stat from blob Get method
This calls Stat before Open, which should be unnecessary because Open
can handle the case of a nonexistent blob. Removing the Stat saves a
round trip.

This is similar to the removal of stat in Open in #1226.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-11 15:13:03 -08:00
Richard Scothern
e6363e8a20 Merge pull request #1187 from stevvooe/check-storage-drivers-list-path-not-found
[WIP] registry/storage/driver: checking that non-existent path returns PathNotFoundError
2015-12-08 16:32:02 -08:00
Stephen J Day
d68acc869e storage/driver/s3: adjust s3 driver to return unmunged path
This fixes both the s3 driver and the oss driver to return the unmunged path
when returning errors.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 15:48:55 -08:00
Stephen Day
3032cd2aa9 Merge pull request #1246 from stevvooe/decrease-driver-testsuite-allocations
storage/driver: decrease memory allocation done during testsuite
2015-12-08 14:52:59 -08:00
Kenny Leung
d38e02c52f Print error for failed HTTP auth request.
Signed-off-by: Kenny Leung <kleung@google.com>
2015-12-08 14:24:03 -08:00
Li Yi
533c912d3e Fix the issue for listing root directory
Change-Id: I1c6181fa4e5666bd2e6ec69cb608c4778ae0fe48
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-08 12:04:04 -08:00
Li Yi
3a5c6446d8 Fix for stevvooe:check-storage-drivers-list-path-not-found in OSS driver
Change-Id: I5e96fe761d3833c962084fd2d597f47e8a72e7c2
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-08 12:04:04 -08:00
davidli
aa08ced9d7 driver/swift: treat empty object list as a PathNotFoundError
Swift returns an empty object list when trying to read a non-existing object path, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: David li <wenquan.li@hpe.com>
2015-12-08 12:04:04 -08:00
Vincent Giersch
c39158d48c driver/rados: treat OMAP EIO as a PathNotFoundError
RADOS returns a -EIO when trying to read a non-existing OMAP, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
2015-12-08 12:04:04 -08:00
Stephen J Day
10f7b7bf95 storage/driver/s3: correct response on list of missing directory
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
dc5b71afb0 storage/driver/base: use correct error format style
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
c46d32bfbb driver/filesystem: address filesystem driver on behavior of List
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
4829e9685e registry/storage/driver: checking that non-existent path returns PathNotFoundError
Issue #1186 describes a condition where a null tags response is returned when
using the s3 driver. The issue seems to be related to a missing
PathNotFoundError in s3. This change adds a test for that to get an idea of the
lack of compliance across storage drivers. If the failures are manageable,
we'll add this test condition and fix the s3 driver.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen Day
51c5b868b1 Merge pull request #1238 from noxiouz/fix_comment
Fix comment for PathRegexp
2015-12-08 10:41:16 -08:00
Stephen Day
9bf537dede Merge pull request #1241 from pdevine/catalog-speedup
Make the catalog more efficient
2015-12-08 10:20:51 -08:00
Richard Scothern
e7e0b72ae0 Merge pull request #1243 from RichardScothern/client-header-message
Add clearer messaging around missing content-length headers.
2015-12-08 09:54:18 -08:00
Stephen J Day
be2985a35d storage/driver: decrease memory allocation done during testsuite
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-07 19:05:13 -08:00
Patrick Devine
ecb84029ec Make the catalog more efficient
This change removes the sort() from the Repositories() function since
we're now guaranteed to have a lexigraphically sorted walk.

Signed-off-by: Patrick Devine <patrick.devine@docker.com>
2015-12-07 12:33:01 -08:00
Richard Scothern
3b7e1a986a Merge pull request #1157 from lebauce/swift-bulk-delete-fixes
Use bulk delete to remove segments in Swift driver
2015-12-07 10:56:32 -08:00
Richard Scothern
fb2142147f Add clearer messaging around missing content-length headers.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-12-07 10:17:49 -08:00
Anton Tiurin
d6cc32965e Fix comment for PathRegexp
Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-12-04 22:14:21 +03:00
Sylvain Baubeau
b596464d38 Use bulk delete to remove segments in Swift driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-12-04 12:16:49 +01:00
Stephen Day
fb89fd30f8 Merge pull request #1231 from tonistiigi/digest-length-check
Validate digest length on parsing
2015-12-03 12:17:16 -08:00
Stephen Day
0a71df72a1 Merge pull request #1211 from tt/remove-name-verification
Remove name verification
2015-12-03 11:59:21 -08:00
Stephen Day
0aee52fa3d Merge pull request #1227 from stevvooe/walk-sorted
storage: enforce sorted traversal during Walk
2015-12-02 18:37:44 -08:00
Tonis Tiigi
1f5f9bad39 Validate digest length on parsing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2015-12-02 16:04:06 -08:00
Aaron Lehmann
bf2cc0a9d6 Avoid stat round-trips when fetching a blob
Without this commit, three round-trips are required to fetch a blob with
a progress bar. The first is a call to Stat (HEAD request), to determine
the size. Then Open is called, which also calls Stat, and finally
performs a GET request.

Only the GET request is actually needed. The size of the blob can be
sniffed from Content-Length in the GET response.

This commit changes HTTPReadSeeker to automatically detect the size from
Content-Length instead of requiring it to be passed in. The Stat call is
removed from Open because it is no longer necessary.

HTTPReadSeeker now takes an additional errorHandler callback argument which
translates an unsuccessful HTTP response into an appropriate API-level
error. Using a callback for this makes it possible to avoid leaking the
repsonse body to Read's caller, which would make lifecycle management
problematic.

Fixes #1223

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-02 14:21:13 -08:00
Stephen J Day
6693e9667c storage: add further tests for Walk implementation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-01 16:55:10 -08:00
Stephen J Day
93f92498ce storage: enforce sorted traversal during Walk
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-01 16:25:30 -08:00