benchmarks added to filewriter_test, demonstrate buffered
version is ~5x faster on my hardware.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
This changeset adds support for a header to identify docker upload uuids. This
id can be used as a key to manage local state for resumable uploads. The goal
is remove the necessity for a client to parse the url to get an upload uuid.
The restrictions for clients to use the location header are still strongly in
place.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The method (Registry).Repository may now return an error. This is too allow
certain implementationt to validate the name or opt to not return a repository
under certain conditions.
In conjunction with this change, error declarations have been moved into a
single file in the distribution package. Several error declarations that had
remained in the storage package have been moved into distribution, as well. The
declarations for Layer and LayerUpload have also been moved into the main
registry file, as a result.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Since the notifications package is now decoupled from storage, we are moving it
to the root package.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
A Layer or LayerUpload should not be coupled with the containing repository.
Remove the Name method and correctly reference from the repository where
appropriate.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
After consideration, it has been decided that the interfaces defined in the
storage package provide a good base for interacting with various registry
instances. Whether interacting with a remote API or a local, on-disk registry,
these types have proved flexible. By moving them here, they can become the
central components of interacting with distribution components.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
After all of the perl refactoring, some import orderings were left asunder.
This commit corrects that.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change is slightly more complex than previous package maves in that the
package name changed. To address this, we simply always reference the package
driver as storagedriver to avoid compatbility issues with existing code. While
unfortunate, this can be cleaned up over time.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The goal is to free up the distribution/registry package to include common
registry types. This moves the webapp definitions out of the way to allow for
this change in the future.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The branch that executes after a failed request authorization due to a missing
repo name now correctly returns an error. This is somewhat superficial since
the response would have already been executed. Although, unintended repository
operations may have occurred.
Documentations and comments have also been updated to be in line with
surrounding changes.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset integrates context with the storage package. Debug messages have
been added to exported methods. Existing log messages will now include
contextual details through logger fields to aid in debugging. This integration
focuses on logging and may be followed up with a metric-oriented change in the
future.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset integrates contextual logging into the registry web application.
Idiomatic context use is attempted within the current webapp layout. The
functionality is centered around making lifecycle objects (application and
request context) into contexts themselves. Relevant data has been moved into
the context where appropriate. We still have some work to do to factor out the
registry.Context object and the dispatching functionality to remove some
awkward portions.
The api tests were slightly refactored to use a test environment to eliminate
common code.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The auth package has been updated to use "golang.org/x/net/context" for
passing information between the application and the auth backend.
AccessControllers should now set a "auth.user" context value to a AuthUser
struct containing a single "Name" field for now with possible, optional, values
in the future.
The "silly" auth backend always sets the name to "silly", while the "token" auth
backend will set the name to match the "subject" claim of the JWT.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This modifies the "docker help" text so that it is no wider than 80 chars
and each description fits on one line. This will also try to use ~ when
possible
Added a test to make sure we don't go over 80 chars again.
Added a test to make sure we use ~
Applied rules/tests to all docker commands - not just main help text
Closes#10214
Signed-off-by: Doug Davis <dug@us.ibm.com>
To clarify the role of actor, the request data that initiates an event has been
separated. The ActorRecord is pared down to just the username. This eliminates
confusion about where event related data should be added.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Endpoints are now created at applications startup time, using notification
configuration. The instances are then added to a Broadcaster instance, which
becomes the main event sink for the application. At request time, an event
bridge is configured to listen to repository method calls. The actor and source
of the eventBridge are created from the requeest context and application,
respectively. The result is notifications are dispatched with calls to the
context's Repository instance and are queued to each endpoint via the
broadcaster.
This commit also adds the concept of a RequestID and App.InstanceID. The
request id uniquely identifies each request and the InstanceID uniquely
identifies a run of the registry. These identifiers can be used in the future
to correlate log messages with generated events to support rich debugging.
The fields of the app were slightly reorganized for clarity and a few horrid
util functions have been removed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Several API tests were added to ensure correct acceptance of zero-size and
empty tar files. This led to several changes in the storage backend around the
guarantees of remote file reading, which backs the layer and layer upload type.
In support of these changes, zero-length and empty checks have been added to
the digest package. These provide a sanity check against upstream tarsum
changes. The fileReader has been modified to be more robust when reading and
seeking on zero-length or non-existent files. The file no longer needs to exist
for the reader to be created. Seeks can now move beyond the end of the file,
causing reads to issue an io.EOF. This eliminates errors during certain race
conditions for reading files which should be detected by stat calls. As a part
of this, a few error types were factored out and the read buffer size was
increased to something more reasonable.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
When requesting a token, the basic auth header is always being set even
if there is no username value. This patch corrects this and does not set
the basic auth header if the username is empty.
Also fixes an issue where pulling all tags from a v2 registry succeeds
when the image does not actually exist on the registry.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
When getting the URL from a v2 registry url builder, it does not
honor the scheme from the endpoint object and will cause an https
endpoint to return urls starting with http.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This changeset addresses intermittent internal server errors encountered during
pushes. The root cause has been isolated to layers that result in identical,
empty filesystems but may have some path declarations (imaginge "./"),
resulting in different tarsums. The main error message reported during these
upload problems was a 500 error, which was not correct. Further investigation
showed the errors to be rooted in digest verification when finishing uploads.
Inspection of the surrounding code also identified a few issues. PutLayerChunk
was slightly refactered into PutLayerUploadComplete. Helper methods were
avoided to make handler less confusing. This simplification leveraged an
earlier change in the spec that moved non-complete chunk uploads to the PATCH
method. Simple logging was also added in the unknown error case that should
help to avoid mysterious 500 errors in the future.
At the same time, the glaring omission of a proper layer upload cancel method
was rectified. This has been added in this change so it is not missed in the
future.
In the future, we may want to refactor the handler code to be more
straightforward, hopefully letting us avoid these problems in the future.
Added test cases that reproduce these errors and drove these changes include
the following:
1. Push a layer with an empty body results in invalid blob upload.
2. Push a layer with a different tarsum (in this case, empty tar)
3. Deleting a layer upload works.
4. Getting status on a deleted layer upload returns 404.
Common functionality was grouped into shared functions to remove repitition.
The API tests will still require future love.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Under certain cases, such as when putting a manifest or check for the existence
of a layer, the status code checks in session_v2.go were too narrow for their
purpose. In the case of putting a manifest, the handler only cares that an
error is not returned. Whether it is a 304 or 202 does not matter, as long as
the server reports success. Having the client only accept specific http codes
inhibits future protocol evolution.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Since the Docker-Distribution-API-Version header value may contain multiple
space delimited versions as well as many instances of the header key, the
header value is now split on whitespace characters to iterate over all versions
that may be listed in one instance of the header.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
v2 ping now checks for a Docker-Distribution-API-Version
header that identifies the endpoint as "registry/2.0"
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Setting a header for all responses can help clients better determine
if the server speaks the legacy v1 API or the v2 API. It is important
that the header be set *BEFORE* routing the request.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Fixes#10129
Makes the .dockercfg more human parsable.
Also cleaned up the (technically) racey login test.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Token cache prevents the need to get a new token for every registry interaction.
Since the tokens are short lived, the cache expires after only a minute.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
While the v2 pull operation is writing the body of the layer blob to disk
it now computes the tarsum checksum of the archive before extracting it to
the backend storage driver. If the checksum does not match that from the
image manifest an error is raised.
Also adds more debug logging to the pull operation and fixes existing test
cases which were failing. Adds a reverse lookup constructor to the tarsum
package so that you can get a tarsum object using a checksum label.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Because docker core cannot vendor non-master Go dependencies, we need to remove
dependencies on registry package. The definition of digest.Digest has been
changed to a string and the regular expressions have been ported from
docker-registry/common library.
We'll likely change this be dependent on the registry in the future when the
API stabilizies and use of the master branch becomes the norm.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This package, ported from next-generation docker regsitry, includes route and
error definitions. These facilitate compliant V2 client implementation. The
portions of the HTTP API that are included in this package are considered to be
locked down and should only be changed through a careful change proposal.
Descriptor definitions package layout may change without affecting API behavior
until the exported Go API is ready to be locked down.
When the new registry stabilizes and becomes the master branch, this package
can be vendored from the registry.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Registry authorization token is now taken from the response body rather than
the repsonse header.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
summary of changes:
registry/auth.go
- More logging around the login functions
- split Login() out to handle different code paths for v1 (unchanged logic)
and v2 (does not currently do account creation)
- handling for either basic or token based login attempts
registry/authchallenge.go
- New File
- credit to Brian Bland <brian.bland@docker.com> (github: BrianBland)
- handles parsing of WWW-Authenticate response headers
registry/endpoint.go
- EVEN MOAR LOGGING
- Many edits throught to make the coad less dense. Sparse code is more
readable code.
- slit Ping() out to handle different code paths for v1 (unchanged logic)
and v2.
- Updated Endpoint struct type to include an entry for authorization
challenges discovered during ping of a v2 registry.
- If registry endpoint version is unknown, v2 code path is first attempted,
then fallback to v1 upon failure.
registry/service.go
- STILL MOAR LOGGING
- simplified the logic around starting the 'auth' job.
registry/session.go
- updated use of a registry.Endpoint struct field.
registry/token.go
- New File
- Handles getting token from the parameters of a token auth challenge.
- Modified from function written by Brian Bland (see above credit).
registry/types.go
- Removed 'DefaultAPIVersion' in lieu of 'APIVersionUnknown = 0'`
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Because we guarded the error check, nil Upload on the handler was getting
through to unexpected branches. This directly handles the missing upload
ensuring its set as expected.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Most of this change follows from the modifications to the storage api. The
driving factor is the separation of layerUploadState from the storage backend,
leaving it to the web application to store and update it. As part of the
updates to meet changes in the storage api, support for the size parameter has
been completely removed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This refactors the hmac state token to take control of the layerUploadState
json message, which has been removed from the storage backend. It also moves
away from the concept of a LayerUploadStateStore callback object, which was
short-lived. This allows for upload offset to be managed by the web application
logic in the face of an inconsistent backend. By controlling the upload offset
externally, we reduce the possibility of misreporting upload state to a client.
We may still want to modify the way this works after getting production
experience.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Passing RepositoryInfo to ResolveAuthConfig, pullRepository, and pushRepository
Moving --registry-mirror configuration to registry config
Created resolve_repository job
Repo names with 'index.docker.io' or 'docker.io' are now synonymous with omitting an index name.
Adding test for RepositoryInfo
Adding tests for opts.StringSetOpts and registry.ValidateMirror
Fixing search term use of repoInfo
Adding integration tests for registry mirror configuration
Normalizing LookupImage image name to match LocalName parsing rules
Normalizing repository LocalName to avoid multiple references to an official image
Removing errorOut use in tests
Removing TODO comment
gofmt changes
golint comments cleanup. renaming RegistryOptions => registry.Options, and RegistryServiceConfig => registry.ServiceConfig
Splitting out builtins.Registry and registry.NewService calls
Stray whitespace cleanup
Moving integration tests for Mirrors and InsecureRegistries into TestNewIndexInfo unit test
Factoring out ValidateRepositoryName from NewRepositoryInfo
Removing unused IndexServerURL
Allowing json marshaling of ServiceConfig. Exposing ServiceConfig in /info
Switching to CamelCase for json marshaling
PR cleanup; removing 'Is' prefix from boolean members. Removing unneeded json tags.
Removing non-cleanup related fix for 'localhost:[port]' in splitReposName
Merge fixes for gh9735
Fixing integration test
Reapplying #9754
Adding comment on config.IndexConfigs use from isSecureIndex
Remove unused error return value from isSecureIndex
Signed-off-by: Don Kjer <don.kjer@gmail.com>
Adding back comment in isSecureIndex
Signed-off-by: Don Kjer <don.kjer@gmail.com>
Since the repo is no longer just the registry, we are moving the registry web
application package out of the repo root into a sub-package. We may break down
the registry package further to separate webapp components and bring the client
package under it. This change accomplishes the task of freeing up the repo root
for a distribution-oriented package. A stub doc.go file is left in place to
declare intent.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
If `--insecure-registry mydomain.com` was specified, it would match a registry at mydomain.com on any port.
This was accidentally added in #9735 and is now being reverted.
Signed-off-by: Tibor Vass <teabee89@gmail.com>
This commit is patch for following comment
// TODO: This method should return the errors instead of masking them and returning false
Signed-off-by: Daehyeok Mun <daehyeok@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Incase of a 3xx redirect the var was being overshowed and ever changed
causing an infinite loop.
Fixes#9480
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
This ensures that ServerAddress is set, while previously it was getting
set after configFile.Configs.
Signed-off-by: Vaidas Jablonskis <jablonskis@gmail.com>
Fix issue with restoring the tag store and setting static configuration
from the daemon. i.e. the field on the TagStore struct must be made
internal or the json.Unmarshal in restore will overwrite the insecure
registries to be an empty struct.
Signed-off-by: Michael Crosby <michael@docker.com>
Conflicts:
graph/pull.go
graph/push.go
graph/tags.go
We do this to prevent leakage of information, we don't want people
to be able to probe for existing content.
According to RFC 2616, "This status code (404) is commonly used when the server does not
wish to reveal exactly why the request has been refused, or when no other response i
is applicable."
https://www.ietf.org/rfc/rfc2616.txt
10.4.4 403 Forbidden
The server understood the request, but is refusing to fulfill it.
Authorization will not help and the request SHOULD NOT be repeated.
If the request method was not HEAD and the server wishes to make
public why the request has not been fulfilled, it SHOULD describe the
reason for the refusal in the entity. If the server does not wish to
make this information available to the client, the status code 404
(Not Found) can be used instead.
10.4.5 404 Not Found
The server has not found anything matching the Request-URI. No
indication is given of whether the condition is temporary or
permanent. The 410 (Gone) status code SHOULD be used if the server
knows, through some internally configurable mechanism, that an old
resource is permanently unavailable and has no forwarding address.
This status code is commonly used when the server does not wish to
reveal exactly why the request has been refused, or when no other
response is applicable.
When docker is running through its certificates, it should continue
trying with a new certificate even if it gets back a 404 error code.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
Add support for pulling signed images from a version 2 registry.
Only official images within the library namespace will be pull from the
new registry and check the build signature.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This introduces Versions for TarSum checksums.
Fixes: https://github.com/docker/docker/issues/7526
It preserves current functionality and abstracts the interface for
future flexibility of hashing algorithms. As a POC, the VersionDev
Tarsum does not include the mtime in the checksum calculation, and would
solve https://github.com/docker/docker/issues/7387 though this is not a
settled Version is subject to change until a version number is assigned.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Per registry.doRequest, res and client might be nil in case of error
For example, dns resolution errors, /etc/docker/certs.d perms, failed
loading of x509 cert ...
This will make res.StatusCode and res.Body SEGFAULT.
Signed-off-by: Arthur Gautier <baloo@gandi.net>
To avoid conflicting with layer IDs, repository names must
not be tagged with names that collide with hexadecimal strings.
Signed-off-by: Eric Windisch <eric@windisch.us>
The cli interface works similar to other registry related commands:
docker search foo
... searches for foo on the official hub
docker search localhost:5000/foo
... does the same for the private reg at localhost:5000
Signed-off-by: Daniel Menet <membership@sontags.ch>
renaming this struct to more clearly be session, as that is what it
handles.
Splitting out files for easier readability.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
functions to pkg/parsers/kernel, and parsing filters to
pkg/parsers/filter. Adjust imports and package references.
Docker-DCO-1.1-Signed-off-by: Erik Hollensbe <github@hollensbe.org> (github: erikh)
This lets you specify custom client TLS certificates and CA root for a
specific registry hostname. Docker will then verify the registry
against the CA and present the client cert when talking to that
registry. This allows the registry to verify that the client has a
proper key, indicating that the client is allowed to access the
images.
A custom cert is configured by creating a directory in
/etc/docker/certs.d with the same name as the registry hostname. Inside
this directory all *.crt files are added as CA Roots (if none exists,
the system default is used) and pair of files <filename>.key and
<filename>.cert indicate a custom certificate to present to the registry.
If there are multiple certificates each one will be tried in
alphabetical order, proceeding to the next if we get a 403 of 5xx
response.
So, an example setup would be:
/etc/docker/certs.d/
└── localhost
├── client.cert
├── client.key
└── localhost.crt
A simple way to test this setup is to use an apache server to host a
registry. Just copy a registry tree into the apache root, here is an
example one containing the busybox image:
http://people.gnome.org/~alexl/v1.tar.gz
Then add this conf file as /etc/httpd/conf.d/registry.conf:
# This must be in the root context, otherwise it causes a re-negotiation
# which is not supported by the tls implementation in go
SSLVerifyClient optional_no_ca
<Location /v1>
Action cert-protected /cgi-bin/cert.cgi
SetHandler cert-protected
Header set x-docker-registry-version "0.6.2"
SetEnvIf Host (.*) custom_host=$1
Header set X-Docker-Endpoints "%{custom_host}e"
</Location>
And this as /var/www/cgi-bin/cert.cgi
#!/bin/bash
if [ "$HTTPS" != "on" ]; then
echo "Status: 403 Not using SSL"
echo "x-docker-registry-version: 0.6.2"
echo
exit 0
fi
if [ "$SSL_CLIENT_VERIFY" == "NONE" ]; then
echo "Status: 403 Client certificate invalid"
echo "x-docker-registry-version: 0.6.2"
echo
exit 0
fi
echo "Content-length: $(stat --printf='%s' $PATH_TRANSLATED)"
echo "x-docker-registry-version: 0.6.2"
echo "X-Docker-Endpoints: $SERVER_NAME"
echo "X-Docker-Size: 0"
echo
cat $PATH_TRANSLATED
This will return 403 for all accessed to /v1 unless *any* client cert
is presented. Obviously a real implementation would verify more details
about the certificate.
Example client certs can be generated with:
openssl genrsa -out client.key 1024
openssl req -new -x509 -text -key client.key -out client.cert
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
These constants don't need to use time.Duration(). Fixup this file since
it seems to be the only one using this style.
Docker-DCO-1.1-Signed-off-by: Brandon Philips <brandon.philips@coreos.com> (github: philips)
This continues the effort to separate all registry logic from the
deprecated `Server` object.
* 'search' is exposed by `github.com/dotcloud/docker/registry/Service`
* Added proper documentation of Search while I was at it
Docker-DCO-1.1-Signed-off-by: Solomon Hykes <solomon@docker.com> (github: shykes)
This is the first step towards separating the registry subsystem from
the deprecated `Server` object.
* New service `github.com/dotcloud/docker/registry/Service`
* The service is installed by default in `builtins`
* The service only exposes `auth` for now...
* ...Soon to be followed by `pull`, `push` and `search`.
Docker-DCO-1.1-Signed-off-by: Solomon Hykes <solomon@docker.com> (github: shykes)
roll version and standalone information into the _ping. And to support
Headers they are checked after the JSON is loaded (if there is anything
to load). To stay backwards compatible, if the _ping contents are not
able to unmarshal to RegistryInfo, do not stop, but continue with the
same behavior.
Docker-DCO-1.1-Signed-off-by: Vincent Batts <vbatts@redhat.com> (github: vbatts)
For a pull-only, static registry, there only a couple of headers that
need to be optional (that are presently required.
* X-Docker-Registry-Version
* X-Docker-Size
* X-Docker-Endpoints
Docker-DCO-1.1-Signed-off-by: Vincent Batts <vbatts@redhat.com> (github: vbatts)
Since docker uses cookiejar it doesn't need to manage cookies manually
anymore.
Managing cookie was duplicating it.
Docker-DCO-1.1-Signed-off-by: Fabio Falci <fabiofalci@gmail.com> (github: fabiofalci)
- Added an argument to the call() method in order to control the auth sharing
- Enabled it only for search. Pulls and pushes were enabled already.
- Grouped a few variable declarations
Docker-DCO-1.1-Signed-off-by: Roberto Hashioka <roberto.hashioka@docker.com> (github: rogaha)
RequestFactory is no longer a singleton (can be different for different instances of Registry)
Registry now has an indexEndpoint member
Registry methods that needed the indexEndpoint parameter no longer do so
Registry methods will only use token auth where applicable if basic auth is not enabled.
To improve the use of docker with a private registry the login
command is extended with a parameter for the server address.
While implementing i noticed that two problems hindered authentication to a
private registry:
1. the resolve of the authentication did not match during push
because the looked up key was for example localhost:8080 but
the stored one would have been https://localhost:8080
Besides The lookup needs to still work if the https->http fallback
is used
2. During pull of an image no authentication is sent, which
means all repositories are expected to be private.
These points are fixed now. The changes are implemented in
a way to be compatible to existing behavior both in the
API as also with the private registry.
Update:
- login does not require the full url any more, you can login
to the repository prefix:
example:
docker logon localhost:8080
Fixed corner corner cases:
- When login is done during pull and push the registry endpoint is used and
not the central index
- When Remote sends a 401 during pull, it is now correctly delegating to
CmdLogin
- After a Login is done pull and push are using the newly entered login data,
and not the previous ones. This one seems to be also broken in master, too.
- Auth config is now transfered in a parameter instead of the body when
/images/create is called.
