Commit graph

116 commits

Author SHA1 Message Date
Richard Scothern
e6363e8a20 Merge pull request #1187 from stevvooe/check-storage-drivers-list-path-not-found
[WIP] registry/storage/driver: checking that non-existent path returns PathNotFoundError
2015-12-08 16:32:02 -08:00
Stephen J Day
d68acc869e storage/driver/s3: adjust s3 driver to return unmunged path
This fixes both the s3 driver and the oss driver to return the unmunged path
when returning errors.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 15:48:55 -08:00
Stephen Day
3032cd2aa9 Merge pull request #1246 from stevvooe/decrease-driver-testsuite-allocations
storage/driver: decrease memory allocation done during testsuite
2015-12-08 14:52:59 -08:00
Li Yi
533c912d3e Fix the issue for listing root directory
Change-Id: I1c6181fa4e5666bd2e6ec69cb608c4778ae0fe48
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-08 12:04:04 -08:00
Li Yi
3a5c6446d8 Fix for stevvooe:check-storage-drivers-list-path-not-found in OSS driver
Change-Id: I5e96fe761d3833c962084fd2d597f47e8a72e7c2
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-08 12:04:04 -08:00
davidli
aa08ced9d7 driver/swift: treat empty object list as a PathNotFoundError
Swift returns an empty object list when trying to read a non-existing object path, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: David li <wenquan.li@hpe.com>
2015-12-08 12:04:04 -08:00
Vincent Giersch
c39158d48c driver/rados: treat OMAP EIO as a PathNotFoundError
RADOS returns a -EIO when trying to read a non-existing OMAP, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
2015-12-08 12:04:04 -08:00
Stephen J Day
10f7b7bf95 storage/driver/s3: correct response on list of missing directory
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
dc5b71afb0 storage/driver/base: use correct error format style
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
c46d32bfbb driver/filesystem: address filesystem driver on behavior of List
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
4829e9685e registry/storage/driver: checking that non-existent path returns PathNotFoundError
Issue #1186 describes a condition where a null tags response is returned when
using the s3 driver. The issue seems to be related to a missing
PathNotFoundError in s3. This change adds a test for that to get an idea of the
lack of compliance across storage drivers. If the failures are manageable,
we'll add this test condition and fix the s3 driver.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen Day
51c5b868b1 Merge pull request #1238 from noxiouz/fix_comment
Fix comment for PathRegexp
2015-12-08 10:41:16 -08:00
Stephen J Day
be2985a35d storage/driver: decrease memory allocation done during testsuite
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-07 19:05:13 -08:00
Anton Tiurin
d6cc32965e Fix comment for PathRegexp
Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-12-04 22:14:21 +03:00
Sylvain Baubeau
b596464d38 Use bulk delete to remove segments in Swift driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-12-04 12:16:49 +01:00
Richard Scothern
aaf448b2de Merge pull request #1141 from lebauce/swift-no-missing-segment
Ensure read after write for segments
2015-11-10 10:39:43 -08:00
Aaron Lehmann
accfa46f9b Fix empty delete requests with Swift fs driver
The Delete method lists objects under the given path and tries to delete
all of them with a bulk delete request. If the path has no objects
underneath it, the body of this request will be empty, which causes
HTTP-level issues. Specifically, Go's HTTP client senses the empty
request buffer and doesn't include a Content-Length, which causes the
Swift server to fail the request.

This commit fixes the problem by avoiding sending empty bulk delete
requests. This is the correct thing to do anyway, since there's no
reason to request deletion of zero objects.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-11-09 15:33:05 -08:00
Richard Scothern
e79324edd8 Add a generic error type to capture non-typed errors
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-03 11:19:44 -08:00
amitshukla
7840a5bc8f Fix for issue 664: https://github.com/docker/distribution/issues/664
Errors thrown by storage drivers don't have the name of the driver, causing user
confusion about whether the error is coming from Docker or from a storage driver.
This change adds the storage driver name to each error message.

This required changing ErrUnsupportedDriver to a type, leading to code changes
whenever ErrUnsupportedDriver is used.  The tests check whether the driver name
appears in the error message.

Signed-off-by: Amit Shukla <amit.shukla@docker.com>
2015-11-03 11:19:17 -08:00
Sylvain Baubeau
34c1d0ed50 Ensure read after write for segments
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 17:09:52 +01:00
Sylvain Baubeau
11546b5309 Add support for temporary URL for Swift driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 09:59:50 +01:00
Arthur Baars
98ad17f757 Storage driver for: Google Cloud Storage (gcs)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2015-10-28 20:21:56 +00:00
Richard Scothern
1039e2dc26 Merge pull request #950 from jackpgriffin/master
Skip creating swift container if already exists
2015-10-01 10:15:54 -07:00
Nuutti Kotivuori
84e7c07c42 Remove initial access check from S3 driver
In the S3 storage driver there is currently an initial access permission check by listing the bucket. If this check fails, registry will panic and exit.

However, this check is broken in two ways. First of all it strips the final slash from the root directory path, meaning that any access permissions which limit access to a single directory will fail, because S3 treats the path as strict prefix match. Secondly it fails to strip any leading slash that might be present, unlike the other access places, which means that the path used is different as a leading slash is allowed and significant in a filename in S3.

Since there is also a periodic health check which correctly checks access permissions and shows the error more cleanly, the best solution seems to be to just remove this initial access check.

Signed-off-by: Nuutti Kotivuori <nuutti.kotivuori@poplatek.fi>
2015-09-19 12:47:12 +03:00
Jack Griffin
8ceca304b0 Skip creating swift container if already exists
Signed-off-by: Jack Griffin <jackpg14@gmail.com>
2015-09-04 18:53:17 +00:00
Hua Wang
8f5f6a4e59 Add TrustId parameter to swift driver
github/ncw/swift has added support for trust, so let's add it.

Signed-off-by: Hua Wang <wanghua.humble@gmail.com>
2015-09-02 15:05:29 +08:00
Vincent Giersch
9bf231e0fa fix(rados): Create OMAP for root directory
When using the RADOS driver, the hierarchy of the files is stored
in OMAPs, but the root OMAP was not created and a call to List("/")
was returning an error instead of returned the first level files
stored. This patches creates an OMAP for "/" and excludes the listed
directory from the list of files returned.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
2015-08-10 23:46:33 +02:00
tgic
f01a0694c1 remove unused code and fix todo format
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
tgic
90595c7ed9 fix goimports
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
tgic
235ccc0590 add include_oss build tag
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
Li Yi
9c27080c7a Update the comment for the consistency model
Change-Id: Iee49afeda1c11d6af8c0f26c96d8ccc328c22757
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
Li Yi
5d34d31739 Update the comments for consistence model
Change-Id: I161522ee51f247fb17e42844b3699bd9031e34e8
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
faee422420 fix testcase TestReadStreamWithOffset incompatible with oss
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
a9c3f86ce0 fix oss: got 403 in TestContinueStreamAppendLarge
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
Li Yi
440664a109 Update the OSS test case for latest code change
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
fc20dd72d6 check access key and secret before run 2015-07-31 13:22:55 +08:00
tgic
d28a3fa28a add endpoint support 2015-07-31 13:22:54 +08:00
Li Yi
46148721e1 Add the secure access with HTTPS
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Li Yi
bffce5722e Fix the warning of golint
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Li Yi
d79179884a Support OSS driver
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Stephen Day
96efb19952 Merge pull request #493 from nevermosby/storage-support-openstack-swift
Storage support openstack swift
2015-07-22 23:53:36 -07:00
davidli
b2935158b2 Remove IPC support from test file
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:10 +02:00
Sylvain Baubeau
81765f8cbb Catch either missing containers or objects
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:10 +02:00
Sylvain Baubeau
52d28ec81a Do not use Swift server side copy for manifests to handle >5G files
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:10 +02:00
Sylvain Baubeau
000dec3c6f Inline Swift errors handling
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:10 +02:00
Sylvain Baubeau
661f197f68 Retrieve all the objects using pagination
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
704e082254 Do not create objects for directories
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
f190aa4a7c Refactor segment path concatenation code
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
0807282859 Use http.StatusRequestedRangeNotSatisfiable instead of error code
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
7a5aa32a64 Use file instead of filepath as it may cause troubles on Windows
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00