Commit graph

580 commits

Author SHA1 Message Date
Stephen J Day
a7c2dceea5 Define and implement layer info cache
This changeset defines the interface for layer info caches. Layer info caches
speed up access to layer meta data accessed in storage driver backends. The
two main operations are tests for repository membership and resolving path and
size information for backend blobs.

Two implementations are available. The main implementation leverages redis to
store layer info. An alternative implementation simply caches layer info in
maps, which should speed up resolution for less sophisticated implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:09 -07:00
Stephen Day
dff57726f9 Merge pull request #295 from jlhawn/use_resumable_digest
digest, registry/storage: use resumable digest
2015-04-02 20:12:57 -07:00
Stephen Day
721b39d0cb Merge pull request #314 from stevvooe/lockdown-uuid-route
registry/api/v2: stronger validation for uuid field in urls
2015-04-02 13:57:17 -07:00
Josh Hawn
b96de45be8 Use resumable digest for efficient upload finish
By using a resumable digester and storing the state of upload digests between
subsequent upload chunks, finalizing an upload no longer requires reading back
all of the uploaded data to verify the client's expected digest.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-02 11:20:31 -07:00
Stephen J Day
06acde06cb Avoid crash on invalid Move arguments
This chnage prevents a crash when moving from a non-existent directory that has
a file as a parent. To prevent this, we simply check that the node is a
directory and throws an error if it is not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 19:03:02 -07:00
Stephen J Day
6eb804a1ec Stronger validation for uuid field in urls
This change adds strong validation for the uuid variable for v2 routes. This is
a minor specification change but is okay since the uuid field is controlled by
the server. The character set is restricted to avoid path traversal, allowing
for alphanumeric values and urlsafe base64 encoding.

This change has no effect on client implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 18:57:59 -07:00
Stephen J Day
38ae1cb461 Add redis pool to registry webapp
Redis has been integrated with the web application for use with various
services. The configuraiton exposes connection details, timeouts and pool
parameters. Documentation has been updated accordingly.

A few convenience methods have been added to the context package to get loggers
with certain fields, exposing some missing functionality from logrus.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 16:27:24 -07:00
Stephen Day
9ee35877e3 Merge pull request #294 from fkautz/pr_out_updating_msopentech_azure_sdk_for_go_to_latest_master
Updating MSOpenTech/azure-sdk-for-go to latest master
2015-03-24 13:27:19 -07:00
Frederick F. Kautz IV
dffd1babd2 Updating MSOpenTech/azure-sdk-for-go to latest master
Signed-off-by: Frederick F. Kautz IV <fkautz@alumni.cmu.edu>
2015-03-23 21:59:21 -07:00
Ahmet Alp Balkan
594f733e03 storage/driver/azure: Allow non-default realms
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-03-23 20:41:16 -07:00
Josh Hawn
6d14019368 Refactor Layer interface to return a Handler
... Rather than ServeHTTP directly.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 21:59:07 -07:00
Josh Hawn
fdd6314776 Insert request method option storage driver URLFor
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 17:06:40 -07:00
Stephen Day
efb24490c7 Merge pull request #256 from ncdc/master
Send WWW-Authenticate header for silly auth
2015-03-11 19:01:23 -07:00
Andy Goldstein
4b5af16fdc Send WWW-Authenticate header for silly auth
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 15:20:15 -04:00
David Lawrence
83571e574c don't panic during a request when configuring repository middleware. Return a 500 with an appropriate error
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:59:02 -07:00
Andy Goldstein
5c3f53b70f Fix Godoc typos
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:29 -07:00
David Lawrence
6a72d1aefb Final polish to cloudfront and larger middleware refactor
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:29 -07:00
Andy Goldstein
30bcc17b85 Middleware!
Convert middleware in the config to be a map of type->[]Middleware

Add support for registry & repository middleware.

Some naming updates as well.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:28 -07:00
David Lawrence
952f39edff Refactoring cloudfactory layer handler into a more generic storage
middleware concept.

This also breaks the dependency the storage package had on goamz
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:28 -07:00
Josh Hawn
3e658d29a6 digest: Minor refactoring
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-10 16:44:19 -07:00
David Lawrence
eccae81c9e minor refactor + tests for app.go just to improve test coverage.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-09 16:31:30 -07:00
Andy Goldstein
ccfadc93aa Remove max repository component length restriction
Fixes #241

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-06 19:55:33 -05:00
Stephen Day
27baf9eb73 Merge pull request #239 from jlhawn/event_target_update
notifications: update notification event Target fields
2015-03-06 16:45:06 -08:00
Stephen Day
5a8bedcc9f Merge pull request #238 from endophage/canonical_sha256
digest, registry/storage, registry/handlers: switch to SHA256 as canonical digest
2015-03-06 15:27:14 -08:00
David Lawrence
2a786bfc23 fixing up tests to work with for non-tarsum future
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-06 14:54:23 -08:00
Stephen Day
48bf33c038 Merge pull request #243 from stevvooe/storagedriver-docs
doc: move storage driver readmes into docs
2015-03-06 11:25:42 -08:00
Josh Hawn
98daae176a Switch to SHA256 as canonical digest
Also support client digests linking to canonical digest.
2015-03-06 09:49:05 -08:00
Josh Hawn
4e3bf4bad4 Update notification event Target fields
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-05 18:01:50 -08:00
Stephen J Day
19061f347e doc: move storage driver readmes into docs
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-05 17:23:33 -08:00
Stephen J Day
008236cfef Implement immutable manifest reference support
This changeset implements immutable manifest references via the HTTP API. Most
of the changes follow from modifications to ManifestService. Once updates were
made across the repo to implement these changes, the http handlers were change
accordingly. The new methods on ManifestService will be broken out into a
tagging service in a later PR.

Unfortunately, due to complexities around managing the manifest tag index in an
eventually consistent manner, direct deletes of manifests have been disabled.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-04 21:40:55 -08:00
Stephen J Day
f46a1b73e8 spec: fetch manifests by tag or digest
Manifests are now fetched by a field called "reference", which may be a tag or
a digest. When using digests to reference a manifest, the data is immutable.
The routes and specification have been updated to allow this.

There are a few caveats to this approach:

1. It may be problematic to rely on data format to differentiate between a tag
   and a digest. Currently, they are disjoint but there may modifications on
   either side that break this guarantee.
2. The caching characteristics of returned content are very different for
   digest versus tag-based references. Digest urls can be cached forever while tag
   urls cannot.

Both of these are minimal caveats that we can live with in the future.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-04 21:39:36 -08:00
Stephen Day
0ecb468a33 Merge pull request #234 from ncdc/signature-service
Expose Signatures() on Repository
2015-03-04 18:24:36 -08:00
Josh Hawn
f258df6e6b Merge pull request #231 from stevvooe/remove-unnecessary-close
Remove unnecessary close in client
2015-03-04 18:19:35 -08:00
Andy Goldstein
a65662c10f Expose Signatures() on Repository
Add a SignatureService and expose it via Signatures() on Repository so
external integrations wrapping the registry can access signatures.

Move signature related code from revisionstore.go to signaturestore.go.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-04 20:56:11 -05:00
Stephen J Day
0b34048fe3 Remove unnecessary close in client
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-03 19:29:12 -08:00
Stephen Day
29d2414366 Merge pull request #218 from endophage/DIST-148
registry/storage: buffered wrapper for fileWriter
2015-03-03 18:08:28 -08:00
Stephen Day
bd6b150c4a Merge pull request #212 from stevvooe/add-documentation-name-validator
documentation for name validation grammar
2015-03-03 17:32:54 -08:00
David Lawrence
b870e3fdfb wrap buffered writer around filewriter
benchmarks added to filewriter_test, demonstrate buffered
version is ~5x faster on my hardware.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-03 16:23:21 -08:00
Stephen J Day
ac7af800fb documentation for name validation grammar
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-02 10:55:31 -08:00
Stephen Day
b1c8952c1a Merge pull request #213 from stevvooe/docker-upload-uuid
doc/spec, registry/handlers: specify and implement Docker-Upload-UUID
2015-03-02 10:51:15 -08:00
David Lawrence
3bf768a588 Adding test cases to confirm path traversal attempts are mitigated and
bad characters in URI return 404
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-02 09:15:49 -08:00
Stephen J Day
32f5965c06 Specify and implement Docker-Upload-UUID
This changeset adds support for a header to identify docker upload uuids. This
id can be used as a key to manage local state for resumable uploads. The goal
is remove the necessity for a client to parse the url to get an upload uuid.
The restrictions for clients to use the location header are still strongly in
place.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 16:43:47 -08:00
David Lawrence
871cf9dd01 Path prefix support for running registry somewhere other than root of server
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-02-26 16:04:43 -08:00
Stephen Day
00ce453315 Merge pull request #189 from donhcd/rootcertbundle
Rename auth.token.{rootCertBundle -> rootcertbundle}
2015-02-19 19:31:55 -08:00
Stephen Day
3725bda8ea Merge pull request #191 from stevvooe/missing-digest-bad-request
A digest missing parameter should result in a bad request
2015-02-19 18:30:58 -08:00
Stephen J Day
606c5c8c57 A digest missing parameter should result in a bad request
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-19 17:14:25 -08:00
Donald Huang
b87459b363 Rename auth.token.rootCertBundle yml field
Renames auth.token.rootCertBundle field in registry config to rootcertbundle so
that the REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE environment variable will override it.

See
()[https://github.com/docker/distribution/blob/master/configuration/parser.go#L155]

Signed-off-by: Donald Huang <don.hcd@gmail.com>
2015-02-20 00:46:24 +00:00
Andrey Kostov
58269e73fc Fix S3 driver's list when the root directory is either "" or "/" 2015-02-19 16:31:34 -08:00
Andrey Kostov
02718ee277 Add an empty root directory s3 driver specific test 2015-02-19 16:28:32 -08:00
Stephen J Day
3e906311c6 Add error return to Repository method on Registry
The method (Registry).Repository may now return an error. This is too allow
certain implementationt to validate the name or opt to not return a repository
under certain conditions.

In conjunction with this change, error declarations have been moved into a
single file in the distribution package. Several error declarations that had
remained in the storage package have been moved into distribution, as well. The
declarations for Layer and LayerUpload have also been moved into the main
registry file, as a result.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-13 16:27:33 -08:00
Stephen J Day
ed8827c3c2 Move notifications package to distribution
Since the notifications package is now decoupled from storage, we are moving it
to the root package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:40:35 -08:00
Stephen J Day
09bf752234 Remove Name from Layer and LayerUpload interface
A Layer or LayerUpload should not be coupled with the containing repository.
Remove the Name method and correctly reference from the repository where
appropriate.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:27:05 -08:00
Stephen J Day
553d48d618 Move layer interface definitions to distribution package
After consideration, it has been decided that the interfaces defined in the
storage package provide a good base for interacting with various registry
instances. Whether interacting with a remote API or a local, on-disk registry,
these types have proved flexible. By moving them here, they can become the
central components of interacting with distribution components.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:26:46 -08:00
Ahmet Alp Balkan
8728074d65 storagedriver/azure: Add README
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-02-11 14:40:51 -08:00
Stephen J Day
fac0f5412d Run goimports/gofmt on previous changes
After all of the perl refactoring, some import orderings were left asunder.
This commit corrects that.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00
Stephen J Day
6e4f9a2e3e Move storagedriver package to registry/storage/driver
This change is slightly more complex than previous package maves in that the
package name changed. To address this, we simply always reference the package
driver as storagedriver to avoid compatbility issues with existing code. While
unfortunate, this can be cleaned up over time.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00
Stephen J Day
71e7ac33ca Move storage package under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00
Stephen J Day
d6308bc62b Move client package under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:42:56 -08:00
Stephen J Day
c3b07952ad Move auth package under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-10 17:34:04 -08:00
Stephen J Day
3822e685a0 Move registry api definitions under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-10 17:32:22 -08:00
Stephen J Day
54ae545ed3 Move registry package into handler package
The goal is to free up the distribution/registry package to include common
registry types. This moves the webapp definitions out of the way to allow for
this change in the future.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-10 17:25:40 -08:00
Stephen J Day
287e11e1d4 Correctly return when repo name is not available
The branch that executes after a failed request authorization due to a missing
repo name now correctly returns an error. This is somewhat superficial since
the response would have already been executed. Although, unintended repository
operations may have occurred.

Documentations and comments have also been updated to be in line with
surrounding changes.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-10 15:19:02 -08:00
Stephen J Day
9bde7d9835 Integrate context with storage package
This changeset integrates context with the storage package. Debug messages have
been added to exported methods. Existing log messages will now include
contextual details through logger fields to aid in debugging. This integration
focuses on logging and may be followed up with a metric-oriented change in the
future.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-09 14:44:58 -08:00
Stephen J Day
3e84069959 Integrate contextual logging with regsitry app
This changeset integrates contextual logging into the registry web application.
Idiomatic context use is attempted within the current webapp layout. The
functionality is centered around making lifecycle objects (application and
request context) into contexts themselves. Relevant data has been moved into
the context where appropriate.  We still have some work to do to factor out the
registry.Context object and the dispatching functionality to remove some
awkward portions.

The api tests were slightly refactored to use a test environment to eliminate
common code.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-06 16:51:53 -08:00
Stephen Day
d5b7c41ada Merge pull request #140 from jlhawn/137_auth_context
Use context for auth access controllers
2015-02-04 19:25:10 -08:00
Josh Hawn
904b35a24f Use context for auth access controllers
The auth package has been updated to use "golang.org/x/net/context" for
passing information between the application and the auth backend.

AccessControllers should now set a "auth.user" context value to a AuthUser
struct containing a single "Name" field for now with possible, optional, values
in the future.

The "silly" auth backend always sets the name to "silly", while the "token" auth
backend will set the name to match the "subject" claim of the JWT.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-02-04 19:17:33 -08:00
Stephen J Day
1f06e4f816 Manifest PUT should return 202 Accepted status
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 18:27:40 -08:00
Stephen J Day
1089cae282 Separate request data from actor in Event
To clarify the role of actor, the request data that initiates an event has been
separated. The ActorRecord is pared down to just the username. This eliminates
confusion about where event related data should be added.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:32:37 -08:00
Stephen J Day
2aed7c2d0c Webhook notification support in registry webapp
Endpoints are now created at applications startup time, using notification
configuration. The instances are then added to a Broadcaster instance, which
becomes the main event sink for the application. At request time, an event
bridge is configured to listen to repository method calls. The actor and source
of the eventBridge are created from the requeest context and application,
respectively. The result is notifications are dispatched with calls to the
context's Repository instance and are queued to each endpoint via the
broadcaster.

This commit also adds the concept of a RequestID and App.InstanceID. The
request id uniquely identifies each request and the InstanceID uniquely
identifies a run of the registry. These identifiers can be used in the future
to correlate log messages with generated events to support rich debugging.

The fields of the app were slightly reorganized for clarity and a few horrid
util functions have been removed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:32:37 -08:00
Stephen J Day
b6270d9c14 Handle empty blob files more appropriately
Several API tests were added to ensure correct acceptance of zero-size and
empty tar files. This led to several changes in the storage backend around the
guarantees of remote file reading, which backs the layer and layer upload type.

In support of these changes, zero-length and empty checks have been added to
the digest package. These provide a sanity check against upstream tarsum
changes. The fileReader has been modified to be more robust when reading and
seeking on zero-length or non-existent files. The file no longer needs to exist
for the reader to be created. Seeks can now move beyond the end of the file,
causing reads to issue an io.EOF. This eliminates errors during certain race
conditions for reading files which should be detected by stat calls. As a part
of this, a few error types were factored out and the read buffer size was
increased to something more reasonable.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 13:01:49 -08:00
Stephen J Day
33a1f4ef7d Address server errors received during layer upload
This changeset addresses intermittent internal server errors encountered during
pushes.  The root cause has been isolated to layers that result in identical,
empty filesystems but may have some path declarations (imaginge "./"),
resulting in different tarsums. The main error message reported during these
upload problems was a 500 error, which was not correct.  Further investigation
showed the errors to be rooted in digest verification when finishing uploads.

Inspection of the surrounding code also identified a few issues. PutLayerChunk
was slightly refactered into PutLayerUploadComplete. Helper methods were
avoided to make handler less confusing. This simplification leveraged an
earlier change in the spec that moved non-complete chunk uploads to the PATCH
method. Simple logging was also added in the unknown error case that should
help to avoid mysterious 500 errors in the future.

At the same time, the glaring omission of a proper layer upload cancel method
was rectified. This has been added in this change so it is not missed in the
future.

In the future, we may want to refactor the handler code to be more
straightforward, hopefully letting us avoid these problems in the future.

Added test cases that reproduce these errors and drove these changes include
the following:

1. Push a layer with an empty body results in invalid blob upload.
2. Push a layer with a different tarsum (in this case, empty tar)
3. Deleting a layer upload works.
4. Getting status on a deleted layer upload returns 404.

Common functionality was grouped into shared functions to remove repitition.
The API tests will still require future love.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 21:26:35 -08:00
Stephen Day
cd647beb1d Merge pull request #89 from jlhawn/version_header
Add Docker Distribution API Version header
2015-01-20 13:42:12 -08:00
Josh Hawn
acfcc955de Add Docker Distribution API Version header
Setting a header for all responses can help clients better determine
if the server speaks the legacy v1 API or the v2 API. It is important
that the header be set *BEFORE* routing the request.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-20 13:26:05 -08:00
Stephen J Day
825da388a4 Update the registry app to use the new storage interfaces
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-16 18:33:28 -08:00
Stephen J Day
594263a3f5 Correctly handle missing layer upload
Because we guarded the error check, nil Upload on the handler was getting
through to unexpected branches. This directly handles the missing upload
ensuring its set as expected.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 16:09:45 -08:00
Stephen J Day
cd92071caa Directly manage layerUploadState in webapp
Most of this change follows from the modifications to the storage api. The
driving factor is the separation of layerUploadState from the storage backend,
leaving it to the web application to store and update it. As part of the
updates to meet changes in the storage api, support for the size parameter has
been completely removed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:51:13 -08:00
Stephen J Day
fdcfc56f7b Refactor handling of hmac state packing
This refactors the hmac state token to take control of the layerUploadState
json message, which has been removed from the storage backend. It also moves
away from the concept of a LayerUploadStateStore callback object, which was
short-lived. This allows for upload offset to be managed by the web application
logic in the face of an inconsistent backend. By controlling the upload offset
externally, we reduce the possibility of misreporting upload state to a client.

We may still want to modify the way this works after getting production
experience.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:50:39 -08:00
Brian Bland
9d3436c18e Fixes tests, moves layerhandler in config file 2015-01-08 17:29:22 -08:00
Brian Bland
b11d549fd0 Adds support for content redirects for layer downloads
Includes a delegate implementation which redirects to the URL generated
by the storagedriver, and a cloudfront implementation.
Satisfies proposal #49
2015-01-08 17:01:28 -08:00
Stephen J Day
c02f1a5507 Move registry package out of repo root
Since the repo is no longer just the registry, we are moving the registry web
application package out of the repo root into a sub-package. We may break down
the registry package further to separate webapp components and bring the client
package under it. This change accomplishes the task of freeing up the repo root
for a distribution-oriented package. A stub doc.go file is left in place to
declare intent.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-06 10:40:22 -08:00