The Delete method lists objects under the given path and tries to delete
all of them with a bulk delete request. If the path has no objects
underneath it, the body of this request will be empty, which causes
HTTP-level issues. Specifically, Go's HTTP client senses the empty
request buffer and doesn't include a Content-Length, which causes the
Swift server to fail the request.
This commit fixes the problem by avoiding sending empty bulk delete
requests. This is the correct thing to do anyway, since there's no
reason to request deletion of zero objects.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Previously, this error message would stringify as a pointer address,
which isn't particularly helpful.
This change breaks out the elements of the challenge object such that
the error is appropriately represented.
Signed-off-by: Ted Reed <ted.reed@gmail.com>
contains equal length History and FSLayer arrays.
This is required to prevent malformed manifests being put to the registry and
failing external verification checks.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Errors thrown by storage drivers don't have the name of the driver, causing user
confusion about whether the error is coming from Docker or from a storage driver.
This change adds the storage driver name to each error message.
This required changing ErrUnsupportedDriver to a type, leading to code changes
whenever ErrUnsupportedDriver is used. The tests check whether the driver name
appears in the error message.
Signed-off-by: Amit Shukla <amit.shukla@docker.com>
This extends the specification for the Bearer token response to include
information pertaining to when an issued Bearer token will expire.
This also allows the client to accept `access_token` as an alias for `token`.
Signed-off-by: Matt Moore <mattmoor@google.com>
The proxy scheduler implemented its own timer state machine. It's
simpler and more efficient to leverage the Go runtime's timer heap by
using time.AfterFunc.
This commit adds a time.Timer to each scheduler entry, and starts and
stops those timers as necessary. Then the mainloop goroutine and its
associated logic are not needed.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
To ensure that we only unmarshal the verified payload into the contained
manifest, we first copy the entire incoming buffer into Raw and then unmarshal
only the Payload portion of the incoming bytes. If the contents is later
verified, the caller can then be sure that the contents of the Manifest fields
can be trusted.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
There seems to be a need for a type that represents a way of pointing
to an image, irrespective of the implementation.
This patch defines a Reference interface and provides 3 implementations:
- TagReference: when only a tag is provided
- DigestReference: when a digest (according to the digest package) is
provided, can include optional tag as well
Validation of references are purely syntactic.
There is also a strong type for tags, analogous to digests, as well
as a strong type for Repository from which clients can access the
hostname alone, or the repository name without the hostname, or both
together via the String() method.
For Repository, the files names.go and names_test.go were moved from
the v2 package.
Signed-off-by: Tibor Vass <tibor@docker.com>
of readers joining current downloads. Concurrent requests for the same blob
will not block, but only the first instance will be comitted locally.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Fixes https://github.com/docker/distribution/issues/1062
This relaxes the naming restrictions places on Docker images to permit
valid hostnames according to [RFC-2396](https://www.ietf.org/rfc/rfc2396.txt).
It deviates from the RFC in the following ways:
1) Allow underscores where we allow hyphens (hostnames don't allow
underscores, which we must for backwards compatibility).
2) Leave "top-level" name segments unrestricted (domains require an
alpha character to begin a top-level domain, e.g. "com").
3) DO NOT allow a trailing dot, as permitted by FQDNs.
Signed-off-by: Matt Moore <mattmoor@google.com>
This allows the administrator to specify an externally-reachable URL for
the registry. It takes precedence over the X-Forwarded-Proto and
X-Forwarded-Host headers, and the hostname in the request.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Associate HTTP 401s with Authentication errors rather than Authorization
errors. Changes the meaning of the UNAUTHORIZED error to be authentication
specific.
Defines DENIED error code to be associated with authorization
errors which result in HTTP 403 responses.
Add 'No Such Repository' errors to more endpoints.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
The "testing" package adds some flags in its init function, so utilities
that import distribution code may print a page of extra testing flags in
their help output.
This commit solves the issue by moving an import of "testing" in the
registry/storage/cache package to a new
registry/storage/cache/cachecheck package, which is only imported by
tests.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Add "readonly" under the storage/maintenance section. When this is set
to true, uploads and deletions will return 503 Service Unavailable
errors.
Document the parameter and add some unit testing.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
In the S3 storage driver there is currently an initial access permission check by listing the bucket. If this check fails, registry will panic and exit.
However, this check is broken in two ways. First of all it strips the final slash from the root directory path, meaning that any access permissions which limit access to a single directory will fail, because S3 treats the path as strict prefix match. Secondly it fails to strip any leading slash that might be present, unlike the other access places, which means that the path used is different as a leading slash is allowed and significant in a filename in S3.
Since there is also a periodic health check which correctly checks access permissions and shows the error more cleanly, the best solution seems to be to just remove this initial access check.
Signed-off-by: Nuutti Kotivuori <nuutti.kotivuori@poplatek.fi>
By adding WithVersion to the context package, we can simplify context setup in
the application. This avoids some odd bugs where instantiation order can lead
to missing instance.id or version from log messages.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This solves a issue from #909 where instance.id was not printed in logs,
because this file was using the background context from
golang.org/x/net/context instead of
github.com/docker/distribution/context.
It's cleaner to standardize on one package, so this commit removes the
import of golang.org/x/net/context entirely. The Context interfaces
defined in both packages are the same, so other code using
golang.org/x/net/context can still pass its context to NewRegistry.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Use this command in cmd/registry/main.go.
Move debug server to the main command, and change Serve to be a
ListenAndServe function.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
As we begin our march towards multi-arch, we must prepare for the reality of
multiple manifest schemas. This is the beginning of a set of changes to
facilitate this. We are both moving this package into its target position where
it may live peacefully next to other manfiest versions.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To bring ErrorCode into liine with Go conventions, ErrorCode.Error() now
returns the "nice" value of the error code. This ensures error message assembly
works similar to commonly used Go conventions when directly using ErrorCode as
an error.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Also, add timeout and status code parameters to the HTTP checker, and
remove the threshold parameter for the file checker.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Update docs.
Change health_test.go tests to create their own registries and register
the checks there. The tests now call CheckStatus directly instead of
polling the HTTP handler, which returns results from the default
registry.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Add default storagedriver health check to example configuration files
with parameters matching the previous hardcoded configuration.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Add a section to the config file called "health". Within this section,
"filecheckers" and "httpcheckers" list checks to run. Each check
specifies a file or URI, a time interval for the check, and a threshold
specifying how many times the check must fail to reach an unhealthy
state.
Document the new options in docs/configuration.md.
Add unit testing for both types of checkers. Add an UnregisterAll
function in the health package to support the unit tests, and an
Unregister function for consistency with Register.
Fix a string conversion problem in the health package's HTTP checker.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Clean up calling convention for NewRegistryWithDriver to use functional
arguments.
This is a first step towards the refactor described in #215. I plan to
add additional options in the process of moving configurable items from
the App structure to the registry structure.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
A change in #763 to address review comments caused problems. Originally,
instrumentedResponseWriter implemented the CloseNotifier interface, and
would panic if it was wrapping something that did not implement that
interface. This was split into a separate instrumentedResponseWriterCN
type that implements CloseNotifier, so there's a fallback if
instrumentedResponseWriter ever needs to wrap something that does not
implement this interface.
instrumentedResponseWriter's Value method would end up upcasting either
type back to instrumentedResponseWriter, which does not implement the
interface. In effect, instrumentedResponseWriterCN was never visible to
the handler.
This fixes the problem by implementing a wrapper Value method for
instrumentedResponseWriterCN.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Change checkResponse to only expect the configured
X-Content-Type-Options header if it doesn't receive a 405 error, which
means the handler isn't registered for that method.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
The use of the pathMapper is no longer needed the way we have organized the
code base. The extra level of indirection has proved unnecessary and confusing
so we've opted to clean it up. In the future, we may require more flexibility,
but now it is simply not required.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Unfortunately, the 2.1 releease has written manfiest links into the wrong
directory. This doesn't affect new 2.1 deployments but fixing this to be 2.0
backwards compatible has broken 2.1.0 compatibility. To ensure we have
compatibility between 2.0, 2.1.0 and future releases, we now check one of
several locations to identify a manifest link.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Unfortunately, the refactor used the incorrect path for manifest links within a
repository. While this didn't stop the registry from working, it did break
compatibility with 2.0 deployments for manifest fetches.
Tests were added to ensure these are locked down to the appropriate paths.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The example configuration files add X-Content-Type-Options: nosniff.
Add coverage in existing registry/handlers unit tests.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
distribution errors. Fill in missing checks for mutations on a registry pull-through
cache. Add unit tests and update documentation.
Also, give v2.ErrorCodeUnsupported an HTTP status code, previously it was
defaulting to 500, now its 405 Method Not Allowed.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Several error codes are generally useful but tied to the v2 specification
definitions. This change moves these error code definitions into the common
package for use by the health package, which is not tied to the v2 API.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To ensure the ensure the web application is properly operating, we've added a
periodic health check for the storage driver. If the health check fails three
times in a row, the registry will serve 503 response status for any request
until the condition is resolved. The condition is reported in the response body
and via the /debug/health endpoint.
To ensure that all drivers will properly operate with this health check, a
function has been added to the driver testsuite.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
When using the RADOS driver, the hierarchy of the files is stored
in OMAPs, but the root OMAP was not created and a call to List("/")
was returning an error instead of returned the first level files
stored. This patches creates an OMAP for "/" and excludes the listed
directory from the list of files returned.
Signed-off-by: Vincent Giersch <vincent@giersch.fr>
Instead, provide a variant of instrumentedResponseWriter that does not
implement CloseNotifier, and use that when necessary. In
copyFullPayload, log instead of panicing when we encounter something
that doesn't implement CloseNotifier.
This is more complicated than I'd like, but it's necessary because
instrumentedResponseWriter must not embed CloseNotifier unless there's
really a CloseNotifier to embed.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
with a new `proxy` section in the configuration file.
Create a new registry type which delegates storage to a proxyBlobStore
and proxyManifestStore. These stores will pull through data if not present
locally. proxyBlobStore takes care not to write duplicate data to disk.
Add a scheduler to cleanup expired content. The scheduler runs as a background
goroutine. When a blob or manifest is pulled through from the remote registry,
an entry is added to the scheduler with a TTL. When the TTL expires the
scheduler calls a pre-specified function to remove the fetched resource.
Add token authentication to the registry middleware. Get a token at startup
and preload the credential store with the username and password supplied in the
config file.
Allow resumable digest functionality to be disabled at runtime and disable
it when the registry is a pull through cache.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
registry/storage/blob_test.go:149: arg d for printf verb %s of wrong type: github.com/docker/distribution.Descriptor
Signed-off-by: Doug Davis <dug@us.ibm.com>
The response code isn't actually sent to the client, because the
connection has already closed by this point. But it causes the status
code to appear as 499 in the logs instead of 0.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
When a client disconnects without completing a HTTP request, we were
attempting to process the partial request, which usually leads to a 400
error. These errors can pollute the logs and make it more difficult to
track down real bugs.
This change uses CloseNotifier to detect disconnects. In combination
with checking Content-Length, we can detect a disconnect before sending
the full payload, and avoid logging a 400 error.
This logic is only applied to PUT, POST, and PATCH endpoints, as these
are the places where disconnects during a request are most likely to
happen.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
includes the http response.
When debugging non-successful registry requests this will place
the error details and http status fields in the same log line
giving easier visibility to what error occured in the request.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Log a warning if the registry generates its own secret.
Update configuration doc, and remove the default secret from the
development config file.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This passed in the #744 before merge, but apparently the test changed
since the PR was created in ways that led to a new failures.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Storage drivers can implement a method called URLFor which can return a direct
url for a given path. The functionality allows the registry to direct clients
to download content directly from the backend storage. This is commonly used
with s3 and cloudfront. Under certain conditions, such as when the registry is
not local to the backend, these redirects can hurt performance and waste
incoming bandwidth on pulls. This feature addition allows one to disable this
feature, if required.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Conflicts:
configuration/configuration.go
registry/handlers/app.go
registry/storage/catalog_test.go
registry/storage/manifeststore_test.go
registry/storage/registry.go
Implement the delete API by implementing soft delete for layers
and blobs by removing link files and updating the blob descriptor
cache. Deletion is configurable - if it is disabled API calls
will return an unsupported error.
We invalidate the blob descriptor cache by changing the linkedBlobStore's
blobStatter to a blobDescriptorService and naming it blobAccessController.
Delete() is added throughout the relevant API to support this functionality.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Because the logger was incorrectly replaced while adding hooks, log output did
not include the version and instance ids. The main issue was the the
logrus.Entry was replaced with the logger, which included no context. Replacing
the logger on the context is not necessary when configuring hooks since we are
configuring the contexts logger directly.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This removes the erroneous http.Handler interface in favor a simple SetHeaders
method that only operattes on the response. Several unnecessary uses of pointer
types were also fixed up.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Some missing descriptions and error code for tags pagination was cleaned up to
ensure clarity. Specifically, we ensure the request variations are named and
the proper error codes are included.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The way Repositories() was initially called was somewhat different than
other parts of the client bindings because there was no way to instantiate a
Namespace. This change implements a NewRegistry() function which changes
it so that Repositories() can be called the way one would expect.
It doesn't implement any of the other functions of Namespaces.
Signed-off-by: Patrick Devine <patrick.devine@docker.com>
This change removes the Catalog Service and replaces it with a more
simplistic Repositories() method for obtaining a catalog of all
repositories. The Repositories method takes a pre-allocated slice
and fills it up to the size of the slice and returns the amount
filled. The catalog is returned lexicographically and will start
being filled from the last entry passed to Repositories(). If there
are no more entries to fill, io.EOF will be returned.
Signed-off-by: Patrick Devine <patrick.devine@docker.com>
Conflicts:
registry/client/repository.go
registry/handlers/api_test.go
This change adds a basic catalog endpoint to the API, which returns a list,
or partial list, of all of the repositories contained in the registry. Calls
to this endpoint are somewhat expensive, as every call requires walking a
large part of the registry.
Instead, to maintain a list of repositories, you would first call the catalog
endpoint to get an initial list, and then use the events API to maintain
any future repositories.
Signed-off-by: Patrick Devine <patrick.devine@docker.com>
Move the specification to use a Link header, rather than a "next" entry in the
json results. This prevents requiring clients from parsing the request body to
issue the next request. It also ensures that the returned response body does
not change in between requests.
The ordering of the specification has been slightly tweaked, as well. Listing
image tags has been moved after the catalog specification. Tag pagination now
heavily references catalog pagination.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This contains a proposal for a catalog API, provided access to the internal
contents of a registry instance. The API endpoint is prefixed with an
underscore, which is illegal in images names, to prevent collisions with
repositories names. To avoid issues with large result sets, a paginated version
of the API is proposed. We make an addition to the tags API to support
pagination to ensure the specification is conistent.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
After consideration, we've changed the main descriptor field name to for number
of bytes to "size" to match convention. While this may be a subjective
argument, commonly we refer to files by their "size" rather than their
"length". This will match other conventions, like `(FileInfo).Size()` and
methods on `io.SizeReaderAt`. Under more broad analysis, this argument doesn't
necessarily hold up. If anything, "size" is shorter than "length".
Signed-off-by: Stephen J Day <stephen.day@docker.com>
An error level log is already produced within app.authorized() if an
actual unexpected error occurs during authorization, so this warning
level log remains for auditability purposes, but should not be
considered an error condition.
Addresses #704
Signed-off-by: Brian Bland <brian.bland@docker.com>
This changeset provides a common http handler for serving errcodes. This should
unify http responses across webservices in the face of errors.
Several type assertions have been added, as well, to ensure the error interface
is implemented.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Add a functional argument to pass a digest to (ManifestService).GetByTag().
If the digest matches an empty manifest and nil error are returned.
See 1bc740b0d5 for server implementation.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
The main goal of this changeset is to allow repository name components to
consist of a single character. The number of components allowed and the slash
separation requirements have also been clarified.
To go along with this simplification, errant constants and unneeded error types
have been removed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Challenger manager interface is used to handle getting authorization challenges from an endpoint as well as extracting challenges from responses.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Replace ping logic with individual functions to extract API version and authorization challenges. The response from a ping operation can be passed into these function. If an error occurs in parsing, the version or challenge will not be used. Sending the ping request is the responsibility of the caller.
APIVersion has been converted from a string to a structure type. A parse function was added to convert from string to the structure type.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Move client auth into a separate package.
Separate ping from the authorizer and export Challenges type.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Timeouts should not be a discrete period of time, because they end
up being arbitrary and may be difficult to gauge correctly against
very large Docker layers. Rather, timeouts should be set at the
transport level using the SetDeadline attribute on a net.Conn
object.
Signed-off-by: Jon Poler <jonathan.poler@apcera.com>
This removes documentation and code related to IPC based storage driver
plugins. The existence of this functionality was an original feature goal but
is now not maintained and actively confusing incoming contributions. We will
likely explore some driver plugin mechanism in the future but we don't need
this laying around in the meantime.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
See: 3ea67df373/registry/handlers/app.go (L498)
Per the comment on line 498, this moves the logic of setting the http
status code into the serveJSON func, leaving the auth.Challenge.ServeHTTP()
func to just set the auth challenge header.
Signed-off-by: Doug Davis <dug@us.ibm.com>
This ensures that rados is not required when building the registry. This was
slightly tricky in that when the flags were applied, the rados package was
completely missing. This led to a problem where rados was basically unlistable
and untestable as a package. This was fixed by simply adding a doc.go file that
is included whether rados is built or not.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.
I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...
This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.
Fixes#620, #175.
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
After consideration, the basic authentication implementation has been
simplified to only support bcrypt entries in an htpasswd file. This greatly
increases the security of the implementation by reducing the possibility of
timing attacks and other problems trying to detect the password hash type.
Also, the htpasswd file is only parsed at startup, ensuring that the file can
be edited and not effect ongoing requests. Newly added passwords take effect on
restart. Subsequently, password hash entries are now stored in a map.
Test cases have been modified accordingly.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change refactors the basic authentication implementation to better follow
Go coding standards. Many types are no longer exported. The parser is now a
separate function from the authentication code. The standard functions
(*http.Request).BasicAuth/SetBasicAuth are now used where appropriate.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
In the request parameters lists `tag` was used instead of
`reference` present in the HTTP requests paths
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
This PR is for issue of "email after registry webapp panic" #41, improving my
previous design (closed).
It use self setting up hooks, to catch panic in web application.
And, send email in hooks handle directly, to no use new http server and
handler.
Signed-off-by: xiekeyang <keyangxie@126.com>
To make the definition of supported digests more clear, we have refactored the
digest package to have a special Algorithm type. This represents the digest's
prefix and we associated various supported hash implementations through
function calls.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The change relies on a refactor of the upstream resumable sha256/sha512 package
that opts to register implementations with the standard library. This allows
the resumable support to be detected where it matters, avoiding unnecessary and
complex code. It also ensures that consumers of the digest package don't need
to depend on the forked sha implementations.
We also get an optimization with this change. If the size of data written to a
digester is the same as the file size, we check to see if the digest has been
verified. This works if the blob is written and committed in a single request.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Ensure that clients can use the blob descriptor cache provider without needing
the redis package.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Changes behavior so ping doesn't happen if /v2/ is anywhere in a request path, but instead only at the beginning. This fixes attempts to ping on redirected URLs.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The transport package no longer requires importing distribution for the ReadSeekCloser, instead declares its own.
Added comments on the Authenication handler in session.
Added todo on http seek reader to highlight its lack of belonging to the client transport.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Add check for unauthorized error code and explicitly set the error code if the content could not be parsed.
Updated repository test for unauthorized tests and nit feedback.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Each type no longer requires holding a reference to repository.
Added implementation for signatures get.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Update comments and TODOs
Fix switch style
Updated parse http response to take in reader
Add Cancel implementation
Update blobstore variable name
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.
Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:
* All the objects data are stored with opaque UUID names prefixed
(e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
omap object identifier is the virtual directory name, the keys in
a specific are the relative filenames and the values the blob
object identifier (or empty value for a sub directory).
e.g. For the following hierarchy:
/directory1
/directory1/object1
/directory1/object2
/directory1/directory2/object3
The omap "/directory1" will contains the following key / values:
- "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
- "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
- "directory2" ""
The omap "/directory1/directory2" will contains:
- "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"
* The MOVE is implemented by changing the reference to a specific
blob in its parent virtual directory omap.
This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
Repository creation now just takes in an http.RoundTripper. Authenticated requests or requests which require additional headers should use the NewTransport function along with a request modifier (such an an authentication handler).
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Wrapping the reader in a NopCloser is necessary to prevent the http library from closing the input reader.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Refactory authorizer to take a set of authentication handlers for different authentication schemes returned by an unauthorized HTTP requst.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Adds functionality to create a Repository client which connects to a remote endpoint.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.
The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.
"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.
Recommend Review Approach
-------------------------
This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.
To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.
Storage
-------
The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links. Eventually, we
will fully consolidate this storage.
The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.
The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.
Cache
-----
Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.
One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.
Handlers
--------
The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.
Configuration
-------------
One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.
Notifications
-------------
Changes the `notification` package are simply to support the interface
changes.
Context
-------
A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
- Set an Etag header
- Check If-None-Match and respond appropriately
- Set a Cache-Control header with a default of 1 week
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Allow to use a unix socket as a listener.
To specify an endpoint type we use an optional configuration
field 'net', as there's no way to distinguish a relative
socket path from a hostname.
Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
- Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it.
- Add PATCH support for streaming data on upload.
- Add messaging to specification that PATCH with content range is currently not supported.
- Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported.
closes#470
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This deals with a memory leak, caused by goroutines, experienced when using the
s3 driver. Unfortunately, this section of the code leaks goroutines like a
sieve. There is probably some refactoring that could be done to avoid this but
instead, we have a done channel that will cause waiting goroutines to exit.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
- Change driver interface to take a context as its first argument
- Make newFileReader take a context as its first argument
- Make newFileWriter take a context as its first argument
- Make blobstore exists and delete take a context as a first argument
- Pass the layerreader's context to the storage layer
- Pass the app's context to purgeuploads
- Store the app's context into the blobstore (was previously null)
- Pass the trace'd context to the storage drivers
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
According to the Apache mod_proxy docs, X-Forwarded-Host can be a
comma-separated list of hosts, to which each proxy appends the requested
host. We want to grab only the first from this comma-separated list
to get the original requested Host when building URLs.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This adds a missing return statement. It is not strictly needed since if the
io.Copy fails, the Finish operation will fail. Currently, the client reports
both errors where this new code will correctly only report the io.Copy error.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The code using values from the yaml package wasn't careful enought with the
possible incoming types. Turns out, it is just an int but we've made this
section somewhat bulletproof in case that package changes the behavior.
This code likely never worked. The configuration system should be decoupled
from the object instantiation.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Rather than accept the resulting of a layer validation, we retry up to three
times, backing off 100ms after each try. The thought is that we allow s3 files
to make their way into the correct location increasing the liklihood the
verification can proceed, if possible.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
When the registry starts a background timer will periodically
scan the upload directories on the file system every 24 hours
and delete any files older than 1 week. An initial jitter
intends to avoid contention on the filesystem where multiple
registries with the same storage driver are started
simultaneously.
Ensure that the status is logged in the context by instantiating before the
request is routed to handlers. While this requires some level of hacking to
acheive, the result is that the context value of "http.request.status" is as
accurate as possible for each request.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Registry is intended to be used as a repository service than an abstract collection of repositories. Namespace better describes a collection of repositories retrievable by name.
The registry service serves any repository in the global scope.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This moves the instance id out of the app so that it is associated with an
instantiation of the runtime. The instance id is stored on the background
context. This allows allow contexts using the main background context to
include an instance id for log messages. It also simplifies the application
slightly.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The original implementation wrote to different locations in a shared slice.
While this is theoretically okay, we end up thrashing the cpu cache since
multiple slice members may be on the same cache line. So, even though each
thread has its own memory location, there may be contention over the cache
line. This changes the code to aggregate to a slice in a single goroutine.
In reality, this change likely won't have any performance impact. The theory
proposed above hasn't really even been tested. Either way, we can consider it
and possibly go forward.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Rather than enforce lowercase paths for all drivers, support for
case-sensitivity has been deferred to the driver. There are a few caveats to
this approach:
1. There are possible security implications for tags that only differ in their
case. For instance, a tag "A" may be equivalent to tag "a" on certain file
system backends.
2. All system paths should not use case-sensitive identifiers where possible.
This might be problematic in a blob store that uses case-sensitive ids. For
now, since digest hex ids are all case-insensitive, this will not be an issue.
The recommend workaround is to not run the registry on a case-insensitive
filesystem driver in security sensitive applications.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To avoid compounded round trips leading to slow retrieval of manifests with a
large number of signatures, the fetch of signatures has been parallelized. This
simply spawns a goroutine for each path, coordinated with a sync.WaitGroup.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
For consistency with other systems, the redis and caching monitoring data has
been moved under the "registry" section in expvar. This ensures the entire
registry state is kept to a single section.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This allows one to better control the usage of the cache and turn it off
completely. The storage configuration module was modified to allow parameters
to be passed to just the storage implementation, rather than to the driver.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset integrates the layer info cache with the registry webapp and
storage backend. The main benefit is to cache immutable layer meta data,
reducing backend roundtrips. The cache can be configured to use either redis or
an inmemory cache.
This provides massive performance benefits for HEAD http checks on layer blobs
and manifest verification.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset defines the interface for layer info caches. Layer info caches
speed up access to layer meta data accessed in storage driver backends. The
two main operations are tests for repository membership and resolving path and
size information for backend blobs.
Two implementations are available. The main implementation leverages redis to
store layer info. An alternative implementation simply caches layer info in
maps, which should speed up resolution for less sophisticated implementations.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
By using a resumable digester and storing the state of upload digests between
subsequent upload chunks, finalizing an upload no longer requires reading back
all of the uploaded data to verify the client's expected digest.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This chnage prevents a crash when moving from a non-existent directory that has
a file as a parent. To prevent this, we simply check that the node is a
directory and throws an error if it is not.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change adds strong validation for the uuid variable for v2 routes. This is
a minor specification change but is okay since the uuid field is controlled by
the server. The character set is restricted to avoid path traversal, allowing
for alphanumeric values and urlsafe base64 encoding.
This change has no effect on client implementations.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Redis has been integrated with the web application for use with various
services. The configuraiton exposes connection details, timeouts and pool
parameters. Documentation has been updated accordingly.
A few convenience methods have been added to the context package to get loggers
with certain fields, exposing some missing functionality from logrus.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
Convert middleware in the config to be a map of type->[]Middleware
Add support for registry & repository middleware.
Some naming updates as well.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
middleware concept.
This also breaks the dependency the storage package had on goamz
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
This changeset implements immutable manifest references via the HTTP API. Most
of the changes follow from modifications to ManifestService. Once updates were
made across the repo to implement these changes, the http handlers were change
accordingly. The new methods on ManifestService will be broken out into a
tagging service in a later PR.
Unfortunately, due to complexities around managing the manifest tag index in an
eventually consistent manner, direct deletes of manifests have been disabled.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Manifests are now fetched by a field called "reference", which may be a tag or
a digest. When using digests to reference a manifest, the data is immutable.
The routes and specification have been updated to allow this.
There are a few caveats to this approach:
1. It may be problematic to rely on data format to differentiate between a tag
and a digest. Currently, they are disjoint but there may modifications on
either side that break this guarantee.
2. The caching characteristics of returned content are very different for
digest versus tag-based references. Digest urls can be cached forever while tag
urls cannot.
Both of these are minimal caveats that we can live with in the future.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Add a SignatureService and expose it via Signatures() on Repository so
external integrations wrapping the registry can access signatures.
Move signature related code from revisionstore.go to signaturestore.go.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
benchmarks added to filewriter_test, demonstrate buffered
version is ~5x faster on my hardware.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
This changeset adds support for a header to identify docker upload uuids. This
id can be used as a key to manage local state for resumable uploads. The goal
is remove the necessity for a client to parse the url to get an upload uuid.
The restrictions for clients to use the location header are still strongly in
place.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The method (Registry).Repository may now return an error. This is too allow
certain implementationt to validate the name or opt to not return a repository
under certain conditions.
In conjunction with this change, error declarations have been moved into a
single file in the distribution package. Several error declarations that had
remained in the storage package have been moved into distribution, as well. The
declarations for Layer and LayerUpload have also been moved into the main
registry file, as a result.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Since the notifications package is now decoupled from storage, we are moving it
to the root package.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
A Layer or LayerUpload should not be coupled with the containing repository.
Remove the Name method and correctly reference from the repository where
appropriate.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
After consideration, it has been decided that the interfaces defined in the
storage package provide a good base for interacting with various registry
instances. Whether interacting with a remote API or a local, on-disk registry,
these types have proved flexible. By moving them here, they can become the
central components of interacting with distribution components.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
After all of the perl refactoring, some import orderings were left asunder.
This commit corrects that.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change is slightly more complex than previous package maves in that the
package name changed. To address this, we simply always reference the package
driver as storagedriver to avoid compatbility issues with existing code. While
unfortunate, this can be cleaned up over time.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The goal is to free up the distribution/registry package to include common
registry types. This moves the webapp definitions out of the way to allow for
this change in the future.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The branch that executes after a failed request authorization due to a missing
repo name now correctly returns an error. This is somewhat superficial since
the response would have already been executed. Although, unintended repository
operations may have occurred.
Documentations and comments have also been updated to be in line with
surrounding changes.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset integrates context with the storage package. Debug messages have
been added to exported methods. Existing log messages will now include
contextual details through logger fields to aid in debugging. This integration
focuses on logging and may be followed up with a metric-oriented change in the
future.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset integrates contextual logging into the registry web application.
Idiomatic context use is attempted within the current webapp layout. The
functionality is centered around making lifecycle objects (application and
request context) into contexts themselves. Relevant data has been moved into
the context where appropriate. We still have some work to do to factor out the
registry.Context object and the dispatching functionality to remove some
awkward portions.
The api tests were slightly refactored to use a test environment to eliminate
common code.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The auth package has been updated to use "golang.org/x/net/context" for
passing information between the application and the auth backend.
AccessControllers should now set a "auth.user" context value to a AuthUser
struct containing a single "Name" field for now with possible, optional, values
in the future.
The "silly" auth backend always sets the name to "silly", while the "token" auth
backend will set the name to match the "subject" claim of the JWT.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
To clarify the role of actor, the request data that initiates an event has been
separated. The ActorRecord is pared down to just the username. This eliminates
confusion about where event related data should be added.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Endpoints are now created at applications startup time, using notification
configuration. The instances are then added to a Broadcaster instance, which
becomes the main event sink for the application. At request time, an event
bridge is configured to listen to repository method calls. The actor and source
of the eventBridge are created from the requeest context and application,
respectively. The result is notifications are dispatched with calls to the
context's Repository instance and are queued to each endpoint via the
broadcaster.
This commit also adds the concept of a RequestID and App.InstanceID. The
request id uniquely identifies each request and the InstanceID uniquely
identifies a run of the registry. These identifiers can be used in the future
to correlate log messages with generated events to support rich debugging.
The fields of the app were slightly reorganized for clarity and a few horrid
util functions have been removed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Several API tests were added to ensure correct acceptance of zero-size and
empty tar files. This led to several changes in the storage backend around the
guarantees of remote file reading, which backs the layer and layer upload type.
In support of these changes, zero-length and empty checks have been added to
the digest package. These provide a sanity check against upstream tarsum
changes. The fileReader has been modified to be more robust when reading and
seeking on zero-length or non-existent files. The file no longer needs to exist
for the reader to be created. Seeks can now move beyond the end of the file,
causing reads to issue an io.EOF. This eliminates errors during certain race
conditions for reading files which should be detected by stat calls. As a part
of this, a few error types were factored out and the read buffer size was
increased to something more reasonable.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset addresses intermittent internal server errors encountered during
pushes. The root cause has been isolated to layers that result in identical,
empty filesystems but may have some path declarations (imaginge "./"),
resulting in different tarsums. The main error message reported during these
upload problems was a 500 error, which was not correct. Further investigation
showed the errors to be rooted in digest verification when finishing uploads.
Inspection of the surrounding code also identified a few issues. PutLayerChunk
was slightly refactered into PutLayerUploadComplete. Helper methods were
avoided to make handler less confusing. This simplification leveraged an
earlier change in the spec that moved non-complete chunk uploads to the PATCH
method. Simple logging was also added in the unknown error case that should
help to avoid mysterious 500 errors in the future.
At the same time, the glaring omission of a proper layer upload cancel method
was rectified. This has been added in this change so it is not missed in the
future.
In the future, we may want to refactor the handler code to be more
straightforward, hopefully letting us avoid these problems in the future.
Added test cases that reproduce these errors and drove these changes include
the following:
1. Push a layer with an empty body results in invalid blob upload.
2. Push a layer with a different tarsum (in this case, empty tar)
3. Deleting a layer upload works.
4. Getting status on a deleted layer upload returns 404.
Common functionality was grouped into shared functions to remove repitition.
The API tests will still require future love.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Setting a header for all responses can help clients better determine
if the server speaks the legacy v1 API or the v2 API. It is important
that the header be set *BEFORE* routing the request.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Because we guarded the error check, nil Upload on the handler was getting
through to unexpected branches. This directly handles the missing upload
ensuring its set as expected.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Most of this change follows from the modifications to the storage api. The
driving factor is the separation of layerUploadState from the storage backend,
leaving it to the web application to store and update it. As part of the
updates to meet changes in the storage api, support for the size parameter has
been completely removed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This refactors the hmac state token to take control of the layerUploadState
json message, which has been removed from the storage backend. It also moves
away from the concept of a LayerUploadStateStore callback object, which was
short-lived. This allows for upload offset to be managed by the web application
logic in the face of an inconsistent backend. By controlling the upload offset
externally, we reduce the possibility of misreporting upload state to a client.
We may still want to modify the way this works after getting production
experience.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
