vbatts/registry
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

Merge pull request #211 from stevvooe/immutable-manifest-references

Browse source 
doc/spec, registry: immutable manifest reference support
This commit is contained in:
Stephen Day 2015-03-05 17:38:45 -08:00
commit b4cc5e3ecc
21 changed files with 521 additions and 178 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

79
doc/spec/api.md
View file

@ -106,9 +106,20 @@ changes. Only non-conflicting additions should be made to the API and accepted
changes should avoid preventing future changes from happening. changes should avoid preventing future changes from happening.
This section should be updated when changes are made to the specification, This section should be updated when changes are made to the specification,
indicating what is different. Optionally, we may start marking parts of the specification to correspond with the versions enumerated here. indicating what is different. Optionally, we may start marking parts of the
specification to correspond with the versions enumerated here.
<dl> <dl>
<dt>2.0.1</dt>
<dd>
<ul>
<li>Added support for immutable manifest references in manifest endpoints.</li>
<li>Deleting a manifest by tag has been deprecated.</li>
<li>Specified `Docker-Content-Digest` header for appropriate entities.</li>
<li>Added error code for unsupported operations.</li>
</ul>
</dd>
<dt>2.0</dt> <dt>2.0</dt>
<dd> <dd>
This is the baseline specification. This is the baseline specification.
@ -235,10 +246,11 @@ the V2 registry API, keyed by their tarsum digest.
The image manifest can be fetched with the following url: The image manifest can be fetched with the following url:
``` ```
GET /v2/<name>/manifests/<tag> GET /v2/<name>/manifests/<reference>
``` ```
The "name" and "tag" parameter identify the image and are required. The `name` and `reference` parameter identify the image and are required. The
reference may include a tag or digest.
A `404 Not Found` response will be returned if the image is unknown to the A `404 Not Found` response will be returned if the image is unknown to the
registry. If the image exists and the response is successful, the image registry. If the image exists and the response is successful, the image
@ -330,6 +342,7 @@ http specification). The response will look as follows:
``` ```
200 OK 200 OK
Content-Length: <length of blob> Content-Length: <length of blob>
Docker-Content-Digest: <digest>
``` ```
When this response is received, the client can assume that the layer is When this response is received, the client can assume that the layer is
@ -509,10 +522,14 @@ will receive a `201 Created` response:
201 Created 201 Created
Location: /v2/<name>/blobs/<tarsum> Location: /v2/<name>/blobs/<tarsum>
Content-Length: 0 Content-Length: 0
Docker-Content-Digest: <digest>
``` ```
The `Location` header will contain the registry URL to access the accepted The `Location` header will contain the registry URL to access the accepted
layer file. layer file. The `Docker-Content-Digest` header returns the canonical digest of
the uploaded blob which may differ from the provided digest. Most clients may
ignore the value but if it is used, the client should verify the value against
the uploaded blob data.
###### Digest Parameter ###### Digest Parameter
@ -574,7 +591,7 @@ client must restart the upload process.
Once all of the layers for an image are uploaded, the client can upload the Once all of the layers for an image are uploaded, the client can upload the
image manifest. An image can be pushed using the following request format: image manifest. An image can be pushed using the following request format:
PUT /v2/<name>/manifests/<tag> PUT /v2/<name>/manifests/<reference>
{ {
"name": <name>, "name": <name>,
@ -591,8 +608,8 @@ image manifest. An image can be pushed using the following request format:
... ...
} }
The `name` and `tag` fields of the response body must match those specified in The `name` and `reference` fields of the response body must match those specified in
the URL. the URL. The `reference` field may be a "tag" or a "digest".
If there is a problem with pushing the manifest, a relevant 4xx response will If there is a problem with pushing the manifest, a relevant 4xx response will
be returned with a JSON error message. Please see the _PUT Manifest section be returned with a JSON error message. Please see the _PUT Manifest section
@ -641,13 +658,14 @@ reduce copying.
### Deleting an Image ### Deleting an Image
An image may be deleted from the registry via its `name` and `tag`. A delete An image may be deleted from the registry via its `name` and `reference`. A
may be issued with the following request format: delete may be issued with the following request format:
DELETE /v2/<name>/manifests/<tag> DELETE /v2/<name>/manifests/<reference>
If the image exists and has been successfully deleted, the following response For deletes, `reference` *must* be a digest or the delete will fail. If the
will be issued: image exists and has been successfully deleted, the following response will be
issued:
202 Accepted 202 Accepted
Content-Length: None Content-Length: None
@ -677,9 +695,9 @@ A list of methods and URIs are covered in the table below:
-------|----|------|------------ -------|----|------|------------
| GET | `/v2/` | Base | Check that the endpoint implements Docker Registry API V2. | | GET | `/v2/` | Base | Check that the endpoint implements Docker Registry API V2. |
| GET | `/v2/<name>/tags/list` | Tags | Fetch the tags under the repository identified by `name`. | | GET | `/v2/<name>/tags/list` | Tags | Fetch the tags under the repository identified by `name`. |
| GET | `/v2/<name>/manifests/<tag>` | Manifest | Fetch the manifest identified by `name` and `tag`. | | GET | `/v2/<name>/manifests/<reference>` | Manifest | Fetch the manifest identified by `name` and `reference` where `reference` can be a tag or digest. |
| PUT | `/v2/<name>/manifests/<tag>` | Manifest | Put the manifest identified by `name` and `tag`. | | PUT | `/v2/<name>/manifests/<reference>` | Manifest | Put the manifest identified by `name` and `reference` where `reference` can be a tag or digest. |
| DELETE | `/v2/<name>/manifests/<tag>` | Manifest | Delete the manifest identified by `name` and `tag`. | | DELETE | `/v2/<name>/manifests/<reference>` | Manifest | Delete the manifest identified by `name` and `reference` where `reference` can be a tag or digest. |
| GET | `/v2/<name>/blobs/<digest>` | Blob | Retrieve the blob from the registry identified by `digest`. A `HEAD` request can also be issued to this endpoint to obtain resource information without receiving all data. | | GET | `/v2/<name>/blobs/<digest>` | Blob | Retrieve the blob from the registry identified by `digest`. A `HEAD` request can also be issued to this endpoint to obtain resource information without receiving all data. |
| POST | `/v2/<name>/blobs/uploads/` | Intiate Blob Upload | Initiate a resumable blob upload. If successful, an upload location will be provided to complete the upload. Optionally, if the `digest` parameter is present, the request body will be used to complete the upload in a single request. | | POST | `/v2/<name>/blobs/uploads/` | Intiate Blob Upload | Initiate a resumable blob upload. If successful, an upload location will be provided to complete the upload. Optionally, if the `digest` parameter is present, the request body will be used to complete the upload in a single request. |
| GET | `/v2/<name>/blobs/uploads/<uuid>` | Blob Upload | Retrieve status of upload identified by `uuid`. The primary purpose of this endpoint is to resolve the current status of a resumable upload. | | GET | `/v2/<name>/blobs/uploads/<uuid>` | Blob Upload | Retrieve status of upload identified by `uuid`. The primary purpose of this endpoint is to resolve the current status of a resumable upload. |
@ -697,6 +715,7 @@ The error codes encountered via the API are enumerated in the following table:
|Code|Message|Description| |Code|Message|Description|
-------|----|------|------------ -------|----|------|------------
`UNKNOWN` | unknown error | Generic error returned when the error does not have an API classification. `UNKNOWN` | unknown error | Generic error returned when the error does not have an API classification.
`UNSUPPORTED` | The operation is unsupported. | The operation was unsupported due to a missing implementation or invalid set of parameters.
`UNAUTHORIZED` | access to the requested resource is not authorized | The access controller denied access for the operation on a resource. Often this will be accompanied by a 401 Unauthorized response status. `UNAUTHORIZED` | access to the requested resource is not authorized | The access controller denied access for the operation on a resource. Often this will be accompanied by a 401 Unauthorized response status.
`DIGEST_INVALID` | provided digest did not match uploaded content | When a blob is uploaded, the registry will check that the content matches the digest provided by the client. The error may include a detail structure with the key "digest", including the invalid digest string. This error may also be returned when a manifest includes an invalid layer digest. `DIGEST_INVALID` | provided digest did not match uploaded content | When a blob is uploaded, the registry will check that the content matches the digest provided by the client. The error may include a detail structure with the key "digest", including the invalid digest string. This error may also be returned when a manifest includes an invalid layer digest.
`SIZE_INVALID` | provided length did not match content length | When a layer is uploaded, the provided size will be checked against the uploaded content. If they do not match, this error will be returned. `SIZE_INVALID` | provided length did not match content length | When a layer is uploaded, the provided size will be checked against the uploaded content. If they do not match, this error will be returned.
@ -933,12 +952,12 @@ Create, update and retrieve manifests.
#### GET Manifest #### GET Manifest
Fetch the manifest identified by `name` and `tag`. Fetch the manifest identified by `name` and `reference` where `reference` can be a tag or digest.
``` ```
GET /v2/<name>/manifests/<tag> GET /v2/<name>/manifests/<reference>
Host: <registry host> Host: <registry host>
Authorization: <scheme> <token> Authorization: <scheme> <token>
``` ```
@ -962,6 +981,7 @@ The following parameters should be specified on the request:
``` ```
200 OK 200 OK
Docker-Content-Digest: <digest>
Content-Type: application/json; charset=utf-8 Content-Type: application/json; charset=utf-8
{ {
@ -979,8 +999,13 @@ Content-Type: application/json; charset=utf-8
} }
``` ```
The manifest idenfied by `name` and `tag`. The contents can be used to identify and resolve resources required to run the specified image. The manifest idenfied by `name` and `reference`. The contents can be used to identify and resolve resources required to run the specified image.
The following headers will be returned with the response:
|Name|Description|
|----|-----------|
|`Docker-Content-Digest`|Digest of the targeted content for the request.|
@ -1003,7 +1028,7 @@ Content-Type: application/json; charset=utf-8
} }
``` ```
The name or tag was invalid. The name or reference was invalid.
@ -1080,12 +1105,12 @@ The error codes that may be included in the response body are enumerated below:
#### PUT Manifest #### PUT Manifest
Put the manifest identified by `name` and `tag`. Put the manifest identified by `name` and `reference` where `reference` can be a tag or digest.
``` ```
PUT /v2/<name>/manifests/<tag> PUT /v2/<name>/manifests/<reference>
Host: <registry host> Host: <registry host>
Authorization: <scheme> <token> Authorization: <scheme> <token>
Content-Type: application/json; charset=utf-8 Content-Type: application/json; charset=utf-8
@ -1126,6 +1151,7 @@ The following parameters should be specified on the request:
202 Accepted 202 Accepted
Location: <url> Location: <url>
Content-Length: 0 Content-Length: 0
Docker-Content-Digest: <digest>
``` ```
The manifest has been accepted by the registry and is stored under the specified `name` and `tag`. The manifest has been accepted by the registry and is stored under the specified `name` and `tag`.
@ -1136,6 +1162,7 @@ The following headers will be returned with the response:
|----|-----------| |----|-----------|
|`Location`|The canonical location url of the uploaded manifest.| |`Location`|The canonical location url of the uploaded manifest.|
|`Content-Length`|The `Content-Length` header must be zero and the body must be empty.| |`Content-Length`|The `Content-Length` header must be zero and the body must be empty.|
|`Docker-Content-Digest`|Digest of the targeted content for the request.|
@ -1277,12 +1304,12 @@ The error codes that may be included in the response body are enumerated below:
#### DELETE Manifest #### DELETE Manifest
Delete the manifest identified by `name` and `tag`. Delete the manifest identified by `name` and `reference` where `reference` can be a tag or digest.
``` ```
DELETE /v2/<name>/manifests/<tag> DELETE /v2/<name>/manifests/<reference>
Host: <registry host> Host: <registry host>
Authorization: <scheme> <token> Authorization: <scheme> <token>
``` ```
@ -1456,6 +1483,7 @@ The following parameters should be specified on the request:
``` ```
200 OK 200 OK
Content-Length: <length> Content-Length: <length>
Docker-Content-Digest: <digest>
Content-Type: application/octet-stream Content-Type: application/octet-stream
<blob binary data> <blob binary data>
@ -1468,12 +1496,14 @@ The following headers will be returned with the response:
|Name|Description| |Name|Description|
|----|-----------| |----|-----------|
|`Content-Length`|The length of the requested blob content.| |`Content-Length`|The length of the requested blob content.|
|`Docker-Content-Digest`|Digest of the targeted content for the request.|
###### On Success: Temporary Redirect ###### On Success: Temporary Redirect
``` ```
307 Temporary Redirect 307 Temporary Redirect
Location: <blob location> Location: <blob location>
Docker-Content-Digest: <digest>
``` ```
The blob identified by `digest` is available at the provided location. The blob identified by `digest` is available at the provided location.
@ -1483,6 +1513,7 @@ The following headers will be returned with the response:
|Name|Description| |Name|Description|
|----|-----------| |----|-----------|
|`Location`|The location where the layer should be accessible.| |`Location`|The location where the layer should be accessible.|
|`Docker-Content-Digest`|Digest of the targeted content for the request.|
@ -2345,6 +2376,7 @@ The following parameters should be specified on the request:
Location: <blob location> Location: <blob location>
Content-Range: <start of range>-<end of range, inclusive> Content-Range: <start of range>-<end of range, inclusive>
Content-Length: <length of chunk> Content-Length: <length of chunk>
Docker-Content-Digest: <digest>
``` ```
The upload has been completed and accepted by the registry. The canonical location will be available in the `Location` header. The upload has been completed and accepted by the registry. The canonical location will be available in the `Location` header.
@ -2356,6 +2388,7 @@ The following headers will be returned with the response:
|`Location`|| |`Location`||
|`Content-Range`|Range of bytes identifying the desired block of content represented by the body. Start must match the end of offset retrieved via status check. Note that this is a non-standard use of the `Content-Range` header.| |`Content-Range`|Range of bytes identifying the desired block of content represented by the body. Start must match the end of offset retrieved via status check. Note that this is a non-standard use of the `Content-Range` header.|
|`Content-Length`|Length of the chunk being uploaded, corresponding the length of the request body.| |`Content-Length`|Length of the chunk being uploaded, corresponding the length of the request body.|
|`Docker-Content-Digest`|Digest of the targeted content for the request.|

42
doc/spec/api.md.tmpl
View file

@ -106,9 +106,20 @@ changes. Only non-conflicting additions should be made to the API and accepted
changes should avoid preventing future changes from happening. changes should avoid preventing future changes from happening.
This section should be updated when changes are made to the specification, This section should be updated when changes are made to the specification,
indicating what is different. Optionally, we may start marking parts of the specification to correspond with the versions enumerated here. indicating what is different. Optionally, we may start marking parts of the
specification to correspond with the versions enumerated here.
<dl> <dl>
<dt>2.0.1</dt>
<dd>
<ul>
<li>Added support for immutable manifest references in manifest endpoints.</li>
<li>Deleting a manifest by tag has been deprecated.</li>
<li>Specified `Docker-Content-Digest` header for appropriate entities.</li>
<li>Added error code for unsupported operations.</li>
</ul>
</dd>
<dt>2.0</dt> <dt>2.0</dt>
<dd> <dd>
This is the baseline specification. This is the baseline specification.
@ -235,10 +246,11 @@ the V2 registry API, keyed by their tarsum digest.
The image manifest can be fetched with the following url: The image manifest can be fetched with the following url:
``` ```
GET /v2/<name>/manifests/<tag> GET /v2/<name>/manifests/<reference>
``` ```
The "name" and "tag" parameter identify the image and are required. The `name` and `reference` parameter identify the image and are required. The
reference may include a tag or digest.
A `404 Not Found` response will be returned if the image is unknown to the A `404 Not Found` response will be returned if the image is unknown to the
registry. If the image exists and the response is successful, the image registry. If the image exists and the response is successful, the image
@ -330,6 +342,7 @@ http specification). The response will look as follows:
``` ```
200 OK 200 OK
Content-Length: <length of blob> Content-Length: <length of blob>
Docker-Content-Digest: <digest>
``` ```
When this response is received, the client can assume that the layer is When this response is received, the client can assume that the layer is
@ -509,10 +522,14 @@ will receive a `201 Created` response:
201 Created 201 Created
Location: /v2/<name>/blobs/<tarsum> Location: /v2/<name>/blobs/<tarsum>
Content-Length: 0 Content-Length: 0
Docker-Content-Digest: <digest>
``` ```
The `Location` header will contain the registry URL to access the accepted The `Location` header will contain the registry URL to access the accepted
layer file. layer file. The `Docker-Content-Digest` header returns the canonical digest of
the uploaded blob which may differ from the provided digest. Most clients may
ignore the value but if it is used, the client should verify the value against
the uploaded blob data.
###### Digest Parameter ###### Digest Parameter
@ -574,7 +591,7 @@ client must restart the upload process.
Once all of the layers for an image are uploaded, the client can upload the Once all of the layers for an image are uploaded, the client can upload the
image manifest. An image can be pushed using the following request format: image manifest. An image can be pushed using the following request format:
PUT /v2/<name>/manifests/<tag> PUT /v2/<name>/manifests/<reference>
{ {
"name": <name>, "name": <name>,
@ -591,8 +608,8 @@ image manifest. An image can be pushed using the following request format:
... ...
} }
The `name` and `tag` fields of the response body must match those specified in The `name` and `reference` fields of the response body must match those specified in
the URL. the URL. The `reference` field may be a "tag" or a "digest".
If there is a problem with pushing the manifest, a relevant 4xx response will If there is a problem with pushing the manifest, a relevant 4xx response will
be returned with a JSON error message. Please see the _PUT Manifest section be returned with a JSON error message. Please see the _PUT Manifest section
@ -641,13 +658,14 @@ reduce copying.
### Deleting an Image ### Deleting an Image
An image may be deleted from the registry via its `name` and `tag`. A delete An image may be deleted from the registry via its `name` and `reference`. A
may be issued with the following request format: delete may be issued with the following request format:
DELETE /v2/<name>/manifests/<tag> DELETE /v2/<name>/manifests/<reference>
If the image exists and has been successfully deleted, the following response For deletes, `reference` *must* be a digest or the delete will fail. If the
will be issued: image exists and has been successfully deleted, the following response will be
issued:
202 Accepted 202 Accepted
Content-Length: None Content-Length: None

19
notifications/listener.go
View file

@ -67,8 +67,8 @@ type manifestServiceListener struct {
parent *repositoryListener parent *repositoryListener
} }
func (msl *manifestServiceListener) Get(tag string) (*manifest.SignedManifest, error) { func (msl *manifestServiceListener) Get(dgst digest.Digest) (*manifest.SignedManifest, error) {
sm, err := msl.ManifestService.Get(tag) sm, err := msl.ManifestService.Get(dgst)
if err == nil { if err == nil {
if err := msl.parent.listener.ManifestPulled(msl.parent.Repository, sm); err != nil { if err := msl.parent.listener.ManifestPulled(msl.parent.Repository, sm); err != nil {
logrus.Errorf("error dispatching manifest pull to listener: %v", err) logrus.Errorf("error dispatching manifest pull to listener: %v", err)
@ -78,8 +78,8 @@ func (msl *manifestServiceListener) Get(tag string) (*manifest.SignedManifest, e
return sm, err return sm, err
} }
func (msl *manifestServiceListener) Put(tag string, sm *manifest.SignedManifest) error { func (msl *manifestServiceListener) Put(sm *manifest.SignedManifest) error {
err := msl.ManifestService.Put(tag, sm) err := msl.ManifestService.Put(sm)
if err == nil { if err == nil {
if err := msl.parent.listener.ManifestPushed(msl.parent.Repository, sm); err != nil { if err := msl.parent.listener.ManifestPushed(msl.parent.Repository, sm); err != nil {
@ -90,6 +90,17 @@ func (msl *manifestServiceListener) Put(tag string, sm *manifest.SignedManifest)
return err return err
} }
func (msl *manifestServiceListener) GetByTag(tag string) (*manifest.SignedManifest, error) {
sm, err := msl.ManifestService.GetByTag(tag)
if err == nil {
if err := msl.parent.listener.ManifestPulled(msl.parent.Repository, sm); err != nil {
logrus.Errorf("error dispatching manifest pull to listener: %v", err)
}
}
return sm, err
}
type layerServiceListener struct { type layerServiceListener struct {
distribution.LayerService distribution.LayerService
parent *repositoryListener parent *repositoryListener

27
notifications/listener_test.go
View file

@ -32,7 +32,7 @@ func TestListener(t *testing.T) {
expectedOps := map[string]int{ expectedOps := map[string]int{
"manifest:push": 1, "manifest:push": 1,
"manifest:pull": 1, "manifest:pull": 2,
// "manifest:delete": 0, // deletes not supported for now // "manifest:delete": 0, // deletes not supported for now
"layer:push": 2, "layer:push": 2,
"layer:pull": 2, "layer:pull": 2,
@ -143,16 +143,35 @@ func checkExerciseRepository(t *testing.T, repository distribution.Repository) {
manifests := repository.Manifests() manifests := repository.Manifests()
if err := manifests.Put(tag, sm); err != nil { if err := manifests.Put(sm); err != nil {
t.Fatalf("unexpected error putting the manifest: %v", err) t.Fatalf("unexpected error putting the manifest: %v", err)
} }
fetched, err := manifests.Get(tag) p, err := sm.Payload()
if err != nil {
t.Fatalf("unexpected error getting manifest payload: %v", err)
}
dgst, err := digest.FromBytes(p)
if err != nil {
t.Fatalf("unexpected error digesting manifest payload: %v", err)
}
fetchedByManifest, err := manifests.Get(dgst)
if err != nil { if err != nil {
t.Fatalf("unexpected error fetching manifest: %v", err) t.Fatalf("unexpected error fetching manifest: %v", err)
} }
if fetched.Tag != fetched.Tag { if fetchedByManifest.Tag != sm.Tag {
t.Fatalf("retrieved unexpected manifest: %v", err)
}
fetched, err := manifests.GetByTag(tag)
if err != nil {
t.Fatalf("unexpected error fetching manifest: %v", err)
}
if fetched.Tag != fetchedByManifest.Tag {
t.Fatalf("retrieved unexpected manifest: %v", err) t.Fatalf("retrieved unexpected manifest: %v", err)
} }
} }

45
registry.go
View file

@ -34,38 +34,41 @@ type Repository interface {
// ManifestService provides operations on image manifests. // ManifestService provides operations on image manifests.
type ManifestService interface { type ManifestService interface {
// Exists returns true if the manifest exists.
Exists(dgst digest.Digest) (bool, error)
// Get retrieves the identified by the digest, if it exists.
Get(dgst digest.Digest) (*manifest.SignedManifest, error)
// Delete removes the manifest, if it exists.
Delete(dgst digest.Digest) error
// Put creates or updates the manifest.
Put(manifest *manifest.SignedManifest) error
// TODO(stevvooe): The methods after this message should be moved to a
// discrete TagService, per active proposals.
// Tags lists the tags under the named repository. // Tags lists the tags under the named repository.
Tags() ([]string, error) Tags() ([]string, error)
// Exists returns true if the manifest exists. // ExistsByTag returns true if the manifest exists.
Exists(tag string) (bool, error) ExistsByTag(tag string) (bool, error)
// Get retrieves the named manifest, if it exists. // GetByTag retrieves the named manifest, if it exists.
Get(tag string) (*manifest.SignedManifest, error) GetByTag(tag string) (*manifest.SignedManifest, error)
// Put creates or updates the named manifest.
// Put(tag string, manifest *manifest.SignedManifest) (digest.Digest, error)
Put(tag string, manifest *manifest.SignedManifest) error
// Delete removes the named manifest, if it exists.
Delete(tag string) error
// TODO(stevvooe): There are several changes that need to be done to this // TODO(stevvooe): There are several changes that need to be done to this
// interface: // interface:
// //
// 1. Get(tag string) should be GetByTag(tag string) // 1. Allow explicit tagging with Tag(digest digest.Digest, tag string)
// 2. Put(tag string, manifest *manifest.SignedManifest) should be // 2. Support reading tags with a re-entrant reader to avoid large
// Put(manifest *manifest.SignedManifest). The method can read the
// tag on manifest to automatically tag it in the repository.
// 3. Need a GetByDigest(dgst digest.Digest) method.
// 4. Allow explicit tagging with Tag(digest digest.Digest, tag string)
// 5. Support reading tags with a re-entrant reader to avoid large
// allocations in the registry. // allocations in the registry.
// 6. Long-term: Provide All() method that lets one scroll through all of // 3. Long-term: Provide All() method that lets one scroll through all of
// the manifest entries. // the manifest entries.
// 7. Long-term: break out concept of signing from manifests. This is // 4. Long-term: break out concept of signing from manifests. This is
// really a part of the distribution sprint. // really a part of the distribution sprint.
// 8. Long-term: Manifest should be an interface. This code shouldn't // 5. Long-term: Manifest should be an interface. This code shouldn't
// really be concerned with the storage format. // really be concerned with the storage format.
} }

33
registry/api/v2/descriptors.go
View file

@ -79,6 +79,13 @@ var (
Format: "<uuid>", Format: "<uuid>",
} }
digestHeader = ParameterDescriptor{
Name: "Docker-Content-Digest",
Description: "Digest of the targeted content for the request.",
Type: "digest",
Format: "<digest>",
}
unauthorizedResponse = ResponseDescriptor{ unauthorizedResponse = ResponseDescriptor{
Description: "The client does not have access to the repository.", Description: "The client does not have access to the repository.",
StatusCode: http.StatusUnauthorized, StatusCode: http.StatusUnauthorized,
@ -454,13 +461,13 @@ var routeDescriptors = []RouteDescriptor{
}, },
{ {
Name: RouteNameManifest, Name: RouteNameManifest,
Path: "/v2/{name:" + RepositoryNameRegexp.String() + "}/manifests/{tag:" + TagNameRegexp.String() + "}", Path: "/v2/{name:" + RepositoryNameRegexp.String() + "}/manifests/{reference:" + TagNameRegexp.String() + "|" + digest.DigestRegexp.String() + "}",
Entity: "Manifest", Entity: "Manifest",
Description: "Create, update and retrieve manifests.", Description: "Create, update and retrieve manifests.",
Methods: []MethodDescriptor{ Methods: []MethodDescriptor{
{ {
Method: "GET", Method: "GET",
Description: "Fetch the manifest identified by `name` and `tag`.", Description: "Fetch the manifest identified by `name` and `reference` where `reference` can be a tag or digest.",
Requests: []RequestDescriptor{ Requests: []RequestDescriptor{
{ {
Headers: []ParameterDescriptor{ Headers: []ParameterDescriptor{
@ -473,8 +480,11 @@ var routeDescriptors = []RouteDescriptor{
}, },
Successes: []ResponseDescriptor{ Successes: []ResponseDescriptor{
{ {
Description: "The manifest idenfied by `name` and `tag`. The contents can be used to identify and resolve resources required to run the specified image.", Description: "The manifest idenfied by `name` and `reference`. The contents can be used to identify and resolve resources required to run the specified image.",
StatusCode: http.StatusOK, StatusCode: http.StatusOK,
Headers: []ParameterDescriptor{
digestHeader,
},
Body: BodyDescriptor{ Body: BodyDescriptor{
ContentType: "application/json; charset=utf-8", ContentType: "application/json; charset=utf-8",
Format: manifestBody, Format: manifestBody,
@ -483,7 +493,7 @@ var routeDescriptors = []RouteDescriptor{
}, },
Failures: []ResponseDescriptor{ Failures: []ResponseDescriptor{
{ {
Description: "The name or tag was invalid.", Description: "The name or reference was invalid.",
StatusCode: http.StatusBadRequest, StatusCode: http.StatusBadRequest,
ErrorCodes: []ErrorCode{ ErrorCodes: []ErrorCode{
ErrorCodeNameInvalid, ErrorCodeNameInvalid,
@ -523,7 +533,7 @@ var routeDescriptors = []RouteDescriptor{
}, },
{ {
Method: "PUT", Method: "PUT",
Description: "Put the manifest identified by `name` and `tag`.", Description: "Put the manifest identified by `name` and `reference` where `reference` can be a tag or digest.",
Requests: []RequestDescriptor{ Requests: []RequestDescriptor{
{ {
Headers: []ParameterDescriptor{ Headers: []ParameterDescriptor{
@ -550,6 +560,7 @@ var routeDescriptors = []RouteDescriptor{
Format: "<url>", Format: "<url>",
}, },
contentLengthZeroHeader, contentLengthZeroHeader,
digestHeader,
}, },
}, },
}, },
@ -628,7 +639,7 @@ var routeDescriptors = []RouteDescriptor{
}, },
{ {
Method: "DELETE", Method: "DELETE",
Description: "Delete the manifest identified by `name` and `tag`.", Description: "Delete the manifest identified by `name` and `reference` where `reference` can be a tag or digest.",
Requests: []RequestDescriptor{ Requests: []RequestDescriptor{
{ {
Headers: []ParameterDescriptor{ Headers: []ParameterDescriptor{
@ -729,6 +740,7 @@ var routeDescriptors = []RouteDescriptor{
Description: "The length of the requested blob content.", Description: "The length of the requested blob content.",
Format: "<length>", Format: "<length>",
}, },
digestHeader,
}, },
Body: BodyDescriptor{ Body: BodyDescriptor{
ContentType: "application/octet-stream", ContentType: "application/octet-stream",
@ -745,6 +757,7 @@ var routeDescriptors = []RouteDescriptor{
Description: "The location where the layer should be accessible.", Description: "The location where the layer should be accessible.",
Format: "<blob location>", Format: "<blob location>",
}, },
digestHeader,
}, },
}, },
}, },
@ -1193,6 +1206,7 @@ var routeDescriptors = []RouteDescriptor{
Format: "<length of chunk>", Format: "<length of chunk>",
Description: "Length of the chunk being uploaded, corresponding the length of the request body.", Description: "Length of the chunk being uploaded, corresponding the length of the request body.",
}, },
digestHeader,
}, },
}, },
}, },
@ -1312,6 +1326,13 @@ var errorDescriptors = []ErrorDescriptor{
Description: `Generic error returned when the error does not have an Description: `Generic error returned when the error does not have an
API classification.`, API classification.`,
}, },
{
Code: ErrorCodeUnsupported,
Value: "UNSUPPORTED",
Message: "The operation is unsupported.",
Description: `The operation was unsupported due to a missing
implementation or invalid set of parameters.`,
},
{ {
Code: ErrorCodeUnauthorized, Code: ErrorCodeUnauthorized,
Value: "UNAUTHORIZED", Value: "UNAUTHORIZED",

3
registry/api/v2/errors.go
View file

@ -13,6 +13,9 @@ const (
// ErrorCodeUnknown is a catch-all for errors not defined below. // ErrorCodeUnknown is a catch-all for errors not defined below.
ErrorCodeUnknown ErrorCode = iota ErrorCodeUnknown ErrorCode = iota
// ErrorCodeUnsupported is returned when an operation is not supported.
ErrorCodeUnsupported
// ErrorCodeUnauthorized is returned if a request is not authorized. // ErrorCodeUnauthorized is returned if a request is not authorized.
ErrorCodeUnauthorized ErrorCodeUnauthorized

14
registry/api/v2/routes_test.go
View file

@ -40,7 +40,7 @@ func TestRouter(t *testing.T) {
RequestURI: "/v2/foo/manifests/bar", RequestURI: "/v2/foo/manifests/bar",
Vars: map[string]string{ Vars: map[string]string{
"name": "foo", "name": "foo",
"tag": "bar", "reference": "bar",
}, },
}, },
{ {
@ -48,7 +48,15 @@ func TestRouter(t *testing.T) {
RequestURI: "/v2/foo/bar/manifests/tag", RequestURI: "/v2/foo/bar/manifests/tag",
Vars: map[string]string{ Vars: map[string]string{
"name": "foo/bar", "name": "foo/bar",
"tag": "tag", "reference": "tag",
},
},
{
RouteName: RouteNameManifest,
RequestURI: "/v2/foo/bar/manifests/sha256:abcdef01234567890",
Vars: map[string]string{
"name": "foo/bar",
"reference": "sha256:abcdef01234567890",
}, },
}, },
{ {
@ -113,7 +121,7 @@ func TestRouter(t *testing.T) {
RequestURI: "/v2/foo/bar/manifests/manifests/tags", RequestURI: "/v2/foo/bar/manifests/manifests/tags",
Vars: map[string]string{ Vars: map[string]string{
"name": "foo/bar/manifests", "name": "foo/bar/manifests",
"tag": "tags", "reference": "tags",
}, },
}, },
{ {

7
registry/api/v2/urls.go
View file

@ -107,11 +107,12 @@ func (ub *URLBuilder) BuildTagsURL(name string) (string, error) {
return tagsURL.String(), nil return tagsURL.String(), nil
} }
// BuildManifestURL constructs a url for the manifest identified by name and tag. // BuildManifestURL constructs a url for the manifest identified by name and
func (ub *URLBuilder) BuildManifestURL(name, tag string) (string, error) { // reference. The argument reference may be either a tag or digest.
func (ub *URLBuilder) BuildManifestURL(name, reference string) (string, error) {
route := ub.cloneRoute(RouteNameManifest) route := ub.cloneRoute(RouteNameManifest)
manifestURL, err := route.URL("name", name, "tag", tag) manifestURL, err := route.URL("name", name, "reference", reference)
if err != nil { if err != nil {
return "", err return "", err
} }

60
registry/handlers/api_test.go
View file

@ -219,6 +219,7 @@ func TestLayerAPI(t *testing.T) {
checkResponse(t, "checking head on existing layer", resp, http.StatusOK) checkResponse(t, "checking head on existing layer", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{ checkHeaders(t, resp, http.Header{
"Content-Length": []string{fmt.Sprint(layerLength)}, "Content-Length": []string{fmt.Sprint(layerLength)},
"Docker-Content-Digest": []string{layerDigest.String()},
}) })
// ---------------- // ----------------
@ -231,6 +232,7 @@ func TestLayerAPI(t *testing.T) {
checkResponse(t, "fetching layer", resp, http.StatusOK) checkResponse(t, "fetching layer", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{ checkHeaders(t, resp, http.Header{
"Content-Length": []string{fmt.Sprint(layerLength)}, "Content-Length": []string{fmt.Sprint(layerLength)},
"Docker-Content-Digest": []string{layerDigest.String()},
}) })
// Verify the body // Verify the body
@ -286,6 +288,9 @@ func TestManifestAPI(t *testing.T) {
// -------------------------------- // --------------------------------
// Attempt to push unsigned manifest with missing layers // Attempt to push unsigned manifest with missing layers
unsignedManifest := &manifest.Manifest{ unsignedManifest := &manifest.Manifest{
Versioned: manifest.Versioned{
SchemaVersion: 1,
},
Name: imageName, Name: imageName,
Tag: tag, Tag: tag,
FSLayers: []manifest.FSLayer{ FSLayers: []manifest.FSLayer{
@ -343,9 +348,33 @@ func TestManifestAPI(t *testing.T) {
t.Fatalf("unexpected error signing manifest: %v", err) t.Fatalf("unexpected error signing manifest: %v", err)
} }
payload, err := signedManifest.Payload()
checkErr(t, err, "getting manifest payload")
dgst, err := digest.FromBytes(payload)
checkErr(t, err, "digesting manifest")
manifestDigestURL, err := env.builder.BuildManifestURL(imageName, dgst.String())
checkErr(t, err, "building manifest url")
resp = putManifest(t, "putting signed manifest", manifestURL, signedManifest) resp = putManifest(t, "putting signed manifest", manifestURL, signedManifest)
checkResponse(t, "putting signed manifest", resp, http.StatusAccepted) checkResponse(t, "putting signed manifest", resp, http.StatusAccepted)
checkHeaders(t, resp, http.Header{
"Location": []string{manifestDigestURL},
"Docker-Content-Digest": []string{dgst.String()},
})
// --------------------
// Push by digest -- should get same result
resp = putManifest(t, "putting signed manifest", manifestDigestURL, signedManifest)
checkResponse(t, "putting signed manifest", resp, http.StatusAccepted)
checkHeaders(t, resp, http.Header{
"Location": []string{manifestDigestURL},
"Docker-Content-Digest": []string{dgst.String()},
})
// ------------------
// Fetch by tag name
resp, err = http.Get(manifestURL) resp, err = http.Get(manifestURL)
if err != nil { if err != nil {
t.Fatalf("unexpected error fetching manifest: %v", err) t.Fatalf("unexpected error fetching manifest: %v", err)
@ -353,6 +382,9 @@ func TestManifestAPI(t *testing.T) {
defer resp.Body.Close() defer resp.Body.Close()
checkResponse(t, "fetching uploaded manifest", resp, http.StatusOK) checkResponse(t, "fetching uploaded manifest", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{
"Docker-Content-Digest": []string{dgst.String()},
})
var fetchedManifest manifest.SignedManifest var fetchedManifest manifest.SignedManifest
dec := json.NewDecoder(resp.Body) dec := json.NewDecoder(resp.Body)
@ -364,6 +396,27 @@ func TestManifestAPI(t *testing.T) {
t.Fatalf("manifests do not match") t.Fatalf("manifests do not match")
} }
// ---------------
// Fetch by digest
resp, err = http.Get(manifestDigestURL)
checkErr(t, err, "fetching manifest by digest")
defer resp.Body.Close()
checkResponse(t, "fetching uploaded manifest", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{
"Docker-Content-Digest": []string{dgst.String()},
})
var fetchedManifestByDigest manifest.SignedManifest
dec = json.NewDecoder(resp.Body)
if err := dec.Decode(&fetchedManifestByDigest); err != nil {
t.Fatalf("error decoding fetched manifest: %v", err)
}
if !bytes.Equal(fetchedManifestByDigest.Raw, signedManifest.Raw) {
t.Fatalf("manifests do not match")
}
// Ensure that the tag is listed. // Ensure that the tag is listed.
resp, err = http.Get(tagsURL) resp, err = http.Get(tagsURL)
if err != nil { if err != nil {
@ -536,6 +589,7 @@ func pushLayer(t *testing.T, ub *v2.URLBuilder, name string, dgst digest.Digest,
checkHeaders(t, resp, http.Header{ checkHeaders(t, resp, http.Header{
"Location": []string{expectedLayerURL}, "Location": []string{expectedLayerURL},
"Content-Length": []string{"0"}, "Content-Length": []string{"0"},
"Docker-Content-Digest": []string{dgst.String()},
}) })
return resp.Header.Get("Location") return resp.Header.Get("Location")
@ -634,3 +688,9 @@ func checkHeaders(t *testing.T, resp *http.Response, headers http.Header) {
} }
} }
} }
func checkErr(t *testing.T, err error, msg string) {
if err != nil {
t.Fatalf("unexpected error %s: %v", msg, err)
}
}

3
registry/handlers/app.go
View file

@ -277,9 +277,8 @@ func (app *App) context(w http.ResponseWriter, r *http.Request) *Context {
ctx = ctxu.WithLogger(ctx, ctxu.GetRequestLogger(ctx)) ctx = ctxu.WithLogger(ctx, ctxu.GetRequestLogger(ctx))
ctx = ctxu.WithLogger(ctx, ctxu.GetLogger(ctx, ctx = ctxu.WithLogger(ctx, ctxu.GetLogger(ctx,
"vars.name", "vars.name",
"vars.tag", "vars.reference",
"vars.digest", "vars.digest",
"vars.tag",
"vars.uuid")) "vars.uuid"))
context := &Context{ context := &Context{

2
registry/handlers/app_test.go
View file

@ -84,7 +84,7 @@ func TestAppDispatcher(t *testing.T) {
endpoint: v2.RouteNameManifest, endpoint: v2.RouteNameManifest,
vars: []string{ vars: []string{
"name", "foo/bar", "name", "foo/bar",
"tag", "sometag", "reference", "sometag",
}, },
}, },
{ {

4
registry/handlers/context.go
View file

@ -45,8 +45,8 @@ func getName(ctx context.Context) (name string) {
return ctxu.GetStringValue(ctx, "vars.name") return ctxu.GetStringValue(ctx, "vars.name")
} }
func getTag(ctx context.Context) (tag string) { func getReference(ctx context.Context) (reference string) {
return ctxu.GetStringValue(ctx, "vars.tag") return ctxu.GetStringValue(ctx, "vars.reference")
} }
var errDigestNotAvailable = fmt.Errorf("digest not available in context") var errDigestNotAvailable = fmt.Errorf("digest not available in context")

127
registry/handlers/images.go
View file

@ -4,6 +4,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
"strings"
"github.com/docker/distribution" "github.com/docker/distribution"
ctxu "github.com/docker/distribution/context" ctxu "github.com/docker/distribution/context"
@ -11,6 +12,7 @@ import (
"github.com/docker/distribution/manifest" "github.com/docker/distribution/manifest"
"github.com/docker/distribution/registry/api/v2" "github.com/docker/distribution/registry/api/v2"
"github.com/gorilla/handlers" "github.com/gorilla/handlers"
"golang.org/x/net/context"
) )
// imageManifestDispatcher takes the request context and builds the // imageManifestDispatcher takes the request context and builds the
@ -18,7 +20,14 @@ import (
func imageManifestDispatcher(ctx *Context, r *http.Request) http.Handler { func imageManifestDispatcher(ctx *Context, r *http.Request) http.Handler {
imageManifestHandler := &imageManifestHandler{ imageManifestHandler := &imageManifestHandler{
Context: ctx, Context: ctx,
Tag: getTag(ctx), }
reference := getReference(ctx)
dgst, err := digest.ParseDigest(reference)
if err != nil {
// We just have a tag
imageManifestHandler.Tag = reference
} else {
imageManifestHandler.Digest = dgst
} }
return handlers.MethodHandler{ return handlers.MethodHandler{
@ -32,14 +41,26 @@ func imageManifestDispatcher(ctx *Context, r *http.Request) http.Handler {
type imageManifestHandler struct { type imageManifestHandler struct {
*Context *Context
// One of tag or digest gets set, depending on what is present in context.
Tag string Tag string
Digest digest.Digest
} }
// GetImageManifest fetches the image manifest from the storage backend, if it exists. // GetImageManifest fetches the image manifest from the storage backend, if it exists.
func (imh *imageManifestHandler) GetImageManifest(w http.ResponseWriter, r *http.Request) { func (imh *imageManifestHandler) GetImageManifest(w http.ResponseWriter, r *http.Request) {
ctxu.GetLogger(imh).Debug("GetImageManifest") ctxu.GetLogger(imh).Debug("GetImageManifest")
manifests := imh.Repository.Manifests() manifests := imh.Repository.Manifests()
manifest, err := manifests.Get(imh.Tag)
var (
sm *manifest.SignedManifest
err error
)
if imh.Tag != "" {
sm, err = manifests.GetByTag(imh.Tag)
} else {
sm, err = manifests.Get(imh.Digest)
}
if err != nil { if err != nil {
imh.Errors.Push(v2.ErrorCodeManifestUnknown, err) imh.Errors.Push(v2.ErrorCodeManifestUnknown, err)
@ -47,9 +68,22 @@ func (imh *imageManifestHandler) GetImageManifest(w http.ResponseWriter, r *http
return return
} }
// Get the digest, if we don't already have it.
if imh.Digest == "" {
dgst, err := digestManifest(imh, sm)
if err != nil {
imh.Errors.Push(v2.ErrorCodeDigestInvalid, err)
w.WriteHeader(http.StatusBadRequest)
return
}
imh.Digest = dgst
}
w.Header().Set("Content-Type", "application/json; charset=utf-8") w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Content-Length", fmt.Sprint(len(manifest.Raw))) w.Header().Set("Content-Length", fmt.Sprint(len(sm.Raw)))
w.Write(manifest.Raw) w.Header().Set("Docker-Content-Digest", imh.Digest.String())
w.Write(sm.Raw)
} }
// PutImageManifest validates and stores and image in the registry. // PutImageManifest validates and stores and image in the registry.
@ -65,7 +99,37 @@ func (imh *imageManifestHandler) PutImageManifest(w http.ResponseWriter, r *http
return return
} }
if err := manifests.Put(imh.Tag, &manifest); err != nil { dgst, err := digestManifest(imh, &manifest)
if err != nil {
imh.Errors.Push(v2.ErrorCodeDigestInvalid, err)
w.WriteHeader(http.StatusBadRequest)
return
}
// Validate manifest tag or digest matches payload
if imh.Tag != "" {
if manifest.Tag != imh.Tag {
ctxu.GetLogger(imh).Errorf("invalid tag on manifest payload: %q != %q", manifest.Tag, imh.Tag)
imh.Errors.Push(v2.ErrorCodeTagInvalid)
w.WriteHeader(http.StatusBadRequest)
return
}
imh.Digest = dgst
} else if imh.Digest != "" {
if dgst != imh.Digest {
ctxu.GetLogger(imh).Errorf("payload digest does match: %q != %q", dgst, imh.Digest)
imh.Errors.Push(v2.ErrorCodeDigestInvalid)
w.WriteHeader(http.StatusBadRequest)
return
}
} else {
imh.Errors.Push(v2.ErrorCodeTagInvalid, "no tag or digest specified")
w.WriteHeader(http.StatusBadRequest)
return
}
if err := manifests.Put(&manifest); err != nil {
// TODO(stevvooe): These error handling switches really need to be // TODO(stevvooe): These error handling switches really need to be
// handled by an app global mapper. // handled by an app global mapper.
switch err := err.(type) { switch err := err.(type) {
@ -94,25 +158,54 @@ func (imh *imageManifestHandler) PutImageManifest(w http.ResponseWriter, r *http
return return
} }
// Construct a canonical url for the uploaded manifest.
location, err := imh.urlBuilder.BuildManifestURL(imh.Repository.Name(), imh.Digest.String())
if err != nil {
// NOTE(stevvooe): Given the behavior above, this absurdly unlikely to
// happen. We'll log the error here but proceed as if it worked. Worst
// case, we set an empty location header.
ctxu.GetLogger(imh).Errorf("error building manifest url from digest: %v", err)
}
w.Header().Set("Location", location)
w.Header().Set("Docker-Content-Digest", imh.Digest.String())
w.WriteHeader(http.StatusAccepted) w.WriteHeader(http.StatusAccepted)
} }
// DeleteImageManifest removes the image with the given tag from the registry. // DeleteImageManifest removes the image with the given tag from the registry.
func (imh *imageManifestHandler) DeleteImageManifest(w http.ResponseWriter, r *http.Request) { func (imh *imageManifestHandler) DeleteImageManifest(w http.ResponseWriter, r *http.Request) {
ctxu.GetLogger(imh).Debug("DeleteImageManifest") ctxu.GetLogger(imh).Debug("DeleteImageManifest")
manifests := imh.Repository.Manifests()
if err := manifests.Delete(imh.Tag); err != nil { // TODO(stevvooe): Unfortunately, at this point, manifest deletes are
switch err := err.(type) { // unsupported. There are issues with schema version 1 that make removing
case distribution.ErrManifestUnknown: // tag index entries a serious problem in eventually consistent storage.
imh.Errors.Push(v2.ErrorCodeManifestUnknown, err) // Once we work out schema version 2, the full deletion system will be
w.WriteHeader(http.StatusNotFound) // worked out and we can add support back.
default: imh.Errors.Push(v2.ErrorCodeUnsupported)
imh.Errors.Push(v2.ErrorCodeUnknown, err)
w.WriteHeader(http.StatusBadRequest) w.WriteHeader(http.StatusBadRequest)
} }
return
// digestManifest takes a digest of the given manifest. This belongs somewhere
// better but we'll wait for a refactoring cycle to find that real somewhere.
func digestManifest(ctx context.Context, sm *manifest.SignedManifest) (digest.Digest, error) {
p, err := sm.Payload()
if err != nil {
if !strings.Contains(err.Error(), "missing signature key") {
ctxu.GetLogger(ctx).Errorf("error getting manifest payload: %v", err)
return "", err
} }
w.Header().Set("Content-Length", "0") // NOTE(stevvooe): There are no signatures but we still have a
w.WriteHeader(http.StatusAccepted) // payload. The request will fail later but this is not the
// responsibility of this part of the code.
p = sm.Raw
}
dgst, err := digest.FromBytes(p)
if err != nil {
ctxu.GetLogger(ctx).Errorf("error digesting manifest: %v", err)
return "", err
}
return dgst, err
} }

2
registry/handlers/layer.go
View file

@ -64,6 +64,8 @@ func (lh *layerHandler) GetLayer(w http.ResponseWriter, r *http.Request) {
} }
defer layer.Close() defer layer.Close()
w.Header().Set("Docker-Content-Digest", lh.Digest.String())
if lh.layerHandler != nil { if lh.layerHandler != nil {
handler, _ := lh.layerHandler.Resolve(layer) handler, _ := lh.layerHandler.Resolve(layer)
if handler != nil { if handler != nil {

5
registry/handlers/layerupload.go
View file

@ -193,6 +193,10 @@ func (luh *layerUploadHandler) PutLayerUploadComplete(w http.ResponseWriter, r *
// TODO(stevvooe): Check the incoming range header here, per the // TODO(stevvooe): Check the incoming range header here, per the
// specification. LayerUpload should be seeked (sought?) to that position. // specification. LayerUpload should be seeked (sought?) to that position.
// TODO(stevvooe): Consider checking the error on this copy.
// Theoretically, problems should be detected during verification but we
// may miss a root cause.
// Read in the final chunk, if any. // Read in the final chunk, if any.
io.Copy(luh.Upload, r.Body) io.Copy(luh.Upload, r.Body)
@ -227,6 +231,7 @@ func (luh *layerUploadHandler) PutLayerUploadComplete(w http.ResponseWriter, r *
w.Header().Set("Location", layerURL) w.Header().Set("Location", layerURL)
w.Header().Set("Content-Length", "0") w.Header().Set("Content-Length", "0")
w.Header().Set("Docker-Content-Digest", layer.Digest().String())
w.WriteHeader(http.StatusCreated) w.WriteHeader(http.StatusCreated)
} }

75
registry/storage/manifeststore.go
View file

@ -5,6 +5,7 @@ import (
"github.com/docker/distribution" "github.com/docker/distribution"
ctxu "github.com/docker/distribution/context" ctxu "github.com/docker/distribution/context"
"github.com/docker/distribution/digest"
"github.com/docker/distribution/manifest" "github.com/docker/distribution/manifest"
"github.com/docker/libtrust" "github.com/docker/libtrust"
) )
@ -18,31 +19,17 @@ type manifestStore struct {
var _ distribution.ManifestService = &manifestStore{} var _ distribution.ManifestService = &manifestStore{}
// func (ms *manifestStore) Repository() Repository { func (ms *manifestStore) Exists(dgst digest.Digest) (bool, error) {
// return ms.repository
// }
func (ms *manifestStore) Tags() ([]string, error) {
ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Tags")
return ms.tagStore.tags()
}
func (ms *manifestStore) Exists(tag string) (bool, error) {
ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Exists") ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Exists")
return ms.tagStore.exists(tag) return ms.revisionStore.exists(dgst)
} }
func (ms *manifestStore) Get(tag string) (*manifest.SignedManifest, error) { func (ms *manifestStore) Get(dgst digest.Digest) (*manifest.SignedManifest, error) {
ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Get") ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Get")
dgst, err := ms.tagStore.resolve(tag)
if err != nil {
return nil, err
}
return ms.revisionStore.get(dgst) return ms.revisionStore.get(dgst)
} }
func (ms *manifestStore) Put(tag string, manifest *manifest.SignedManifest) error { func (ms *manifestStore) Put(manifest *manifest.SignedManifest) error {
ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Put") ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Put")
// TODO(stevvooe): Add check here to see if the revision is already // TODO(stevvooe): Add check here to see if the revision is already
@ -51,7 +38,7 @@ func (ms *manifestStore) Put(tag string, manifest *manifest.SignedManifest) erro
// indicating what happened. // indicating what happened.
// Verify the manifest. // Verify the manifest.
if err := ms.verifyManifest(tag, manifest); err != nil { if err := ms.verifyManifest(manifest); err != nil {
return err return err
} }
@ -62,46 +49,46 @@ func (ms *manifestStore) Put(tag string, manifest *manifest.SignedManifest) erro
} }
// Now, tag the manifest // Now, tag the manifest
return ms.tagStore.tag(tag, revision) return ms.tagStore.tag(manifest.Tag, revision)
} }
// Delete removes all revisions of the given tag. We may want to change these // Delete removes the revision of the specified manfiest.
// semantics in the future, but this will maintain consistency. The underlying func (ms *manifestStore) Delete(dgst digest.Digest) error {
// blobs are left alone. ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Delete - unsupported")
func (ms *manifestStore) Delete(tag string) error { return fmt.Errorf("deletion of manifests not supported")
ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Delete") }
revisions, err := ms.tagStore.revisions(tag) func (ms *manifestStore) Tags() ([]string, error) {
ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).Tags")
return ms.tagStore.tags()
}
func (ms *manifestStore) ExistsByTag(tag string) (bool, error) {
ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).ExistsByTag")
return ms.tagStore.exists(tag)
}
func (ms *manifestStore) GetByTag(tag string) (*manifest.SignedManifest, error) {
ctxu.GetLogger(ms.repository.ctx).Debug("(*manifestStore).GetByTag")
dgst, err := ms.tagStore.resolve(tag)
if err != nil { if err != nil {
return err return nil, err
} }
for _, revision := range revisions { return ms.revisionStore.get(dgst)
if err := ms.revisionStore.delete(revision); err != nil {
return err
}
}
return ms.tagStore.delete(tag)
} }
// verifyManifest ensures that the manifest content is valid from the // verifyManifest ensures that the manifest content is valid from the
// perspective of the registry. It ensures that the name and tag match and // perspective of the registry. It ensures that the signature is valid for the
// that the signature is valid for the enclosed payload. As a policy, the // enclosed payload. As a policy, the registry only tries to store valid
// registry only tries to store valid content, leaving trust policies of that // content, leaving trust policies of that content up to consumers.
// content up to consumers. func (ms *manifestStore) verifyManifest(mnfst *manifest.SignedManifest) error {
func (ms *manifestStore) verifyManifest(tag string, mnfst *manifest.SignedManifest) error {
var errs distribution.ErrManifestVerification var errs distribution.ErrManifestVerification
if mnfst.Name != ms.repository.Name() { if mnfst.Name != ms.repository.Name() {
// TODO(stevvooe): This needs to be an exported error // TODO(stevvooe): This needs to be an exported error
errs = append(errs, fmt.Errorf("repository name does not match manifest name")) errs = append(errs, fmt.Errorf("repository name does not match manifest name"))
} }
if mnfst.Tag != tag {
// TODO(stevvooe): This needs to be an exported error.
errs = append(errs, fmt.Errorf("tag does not match manifest tag"))
}
if _, err := manifest.Verify(mnfst); err != nil { if _, err := manifest.Verify(mnfst); err != nil {
switch err { switch err {
case libtrust.ErrMissingSignatureKey, libtrust.ErrInvalidJSONContent, libtrust.ErrMissingSignatureKey: case libtrust.ErrMissingSignatureKey, libtrust.ErrInvalidJSONContent, libtrust.ErrMissingSignatureKey:

89
registry/storage/manifeststore_test.go
View file

@ -9,25 +9,47 @@ import (
"github.com/docker/distribution" "github.com/docker/distribution"
"github.com/docker/distribution/digest" "github.com/docker/distribution/digest"
"github.com/docker/distribution/manifest" "github.com/docker/distribution/manifest"
"github.com/docker/distribution/registry/storage/driver"
"github.com/docker/distribution/registry/storage/driver/inmemory" "github.com/docker/distribution/registry/storage/driver/inmemory"
"github.com/docker/distribution/testutil" "github.com/docker/distribution/testutil"
"github.com/docker/libtrust" "github.com/docker/libtrust"
"golang.org/x/net/context" "golang.org/x/net/context"
) )
func TestManifestStorage(t *testing.T) { type manifestStoreTestEnv struct {
ctx context.Context
driver driver.StorageDriver
registry distribution.Registry
repository distribution.Repository
name string
tag string
}
func newManifestStoreTestEnv(t *testing.T, name, tag string) *manifestStoreTestEnv {
ctx := context.Background() ctx := context.Background()
name := "foo/bar"
tag := "thetag"
driver := inmemory.New() driver := inmemory.New()
registry := NewRegistryWithDriver(driver) registry := NewRegistryWithDriver(driver)
repo, err := registry.Repository(ctx, name) repo, err := registry.Repository(ctx, name)
if err != nil { if err != nil {
t.Fatalf("unexpected error getting repo: %v", err) t.Fatalf("unexpected error getting repo: %v", err)
} }
ms := repo.Manifests()
exists, err := ms.Exists(tag) return &manifestStoreTestEnv{
ctx: ctx,
driver: driver,
registry: registry,
repository: repo,
name: name,
tag: tag,
}
}
func TestManifestStorage(t *testing.T) {
env := newManifestStoreTestEnv(t, "foo/bar", "thetag")
ms := env.repository.Manifests()
exists, err := ms.ExistsByTag(env.tag)
if err != nil { if err != nil {
t.Fatalf("unexpected error checking manifest existence: %v", err) t.Fatalf("unexpected error checking manifest existence: %v", err)
} }
@ -36,7 +58,7 @@ func TestManifestStorage(t *testing.T) {
t.Fatalf("manifest should not exist") t.Fatalf("manifest should not exist")
} }
if _, err := ms.Get(tag); true { if _, err := ms.GetByTag(env.tag); true {
switch err.(type) { switch err.(type) {
case distribution.ErrManifestUnknown: case distribution.ErrManifestUnknown:
break break
@ -49,8 +71,8 @@ func TestManifestStorage(t *testing.T) {
Versioned: manifest.Versioned{ Versioned: manifest.Versioned{
SchemaVersion: 1, SchemaVersion: 1,
}, },
Name: name, Name: env.name,
Tag: tag, Tag: env.tag,
} }
// Build up some test layers and add them to the manifest, saving the // Build up some test layers and add them to the manifest, saving the
@ -79,7 +101,7 @@ func TestManifestStorage(t *testing.T) {
t.Fatalf("error signing manifest: %v", err) t.Fatalf("error signing manifest: %v", err)
} }
err = ms.Put(tag, sm) err = ms.Put(sm)
if err == nil { if err == nil {
t.Fatalf("expected errors putting manifest") t.Fatalf("expected errors putting manifest")
} }
@ -88,7 +110,7 @@ func TestManifestStorage(t *testing.T) {
// Now, upload the layers that were missing! // Now, upload the layers that were missing!
for dgst, rs := range testLayers { for dgst, rs := range testLayers {
upload, err := repo.Layers().Upload() upload, err := env.repository.Layers().Upload()
if err != nil { if err != nil {
t.Fatalf("unexpected error creating test upload: %v", err) t.Fatalf("unexpected error creating test upload: %v", err)
} }
@ -102,11 +124,11 @@ func TestManifestStorage(t *testing.T) {
} }
} }
if err = ms.Put(tag, sm); err != nil { if err = ms.Put(sm); err != nil {
t.Fatalf("unexpected error putting manifest: %v", err) t.Fatalf("unexpected error putting manifest: %v", err)
} }
exists, err = ms.Exists(tag) exists, err = ms.ExistsByTag(env.tag)
if err != nil { if err != nil {
t.Fatalf("unexpected error checking manifest existence: %v", err) t.Fatalf("unexpected error checking manifest existence: %v", err)
} }
@ -115,7 +137,7 @@ func TestManifestStorage(t *testing.T) {
t.Fatalf("manifest should exist") t.Fatalf("manifest should exist")
} }
fetchedManifest, err := ms.Get(tag) fetchedManifest, err := ms.GetByTag(env.tag)
if err != nil { if err != nil {
t.Fatalf("unexpected error fetching manifest: %v", err) t.Fatalf("unexpected error fetching manifest: %v", err)
} }
@ -134,6 +156,31 @@ func TestManifestStorage(t *testing.T) {
t.Fatalf("unexpected error extracting payload: %v", err) t.Fatalf("unexpected error extracting payload: %v", err)
} }
// Now that we have a payload, take a moment to check that the manifest is
// return by the payload digest.
dgst, err := digest.FromBytes(payload)
if err != nil {
t.Fatalf("error getting manifest digest: %v", err)
}
exists, err = ms.Exists(dgst)
if err != nil {
t.Fatalf("error checking manifest existence by digest: %v", err)
}
if !exists {
t.Fatalf("manifest %s should exist", dgst)
}
fetchedByDigest, err := ms.Get(dgst)
if err != nil {
t.Fatalf("unexpected error fetching manifest by digest: %v", err)
}
if !reflect.DeepEqual(fetchedByDigest, fetchedManifest) {
t.Fatalf("fetched manifest not equal: %#v != %#v", fetchedByDigest, fetchedManifest)
}
sigs, err := fetchedJWS.Signatures() sigs, err := fetchedJWS.Signatures()
if err != nil { if err != nil {
t.Fatalf("unable to extract signatures: %v", err) t.Fatalf("unable to extract signatures: %v", err)
@ -153,8 +200,8 @@ func TestManifestStorage(t *testing.T) {
t.Fatalf("unexpected tags returned: %v", tags) t.Fatalf("unexpected tags returned: %v", tags)
} }
if tags[0] != tag { if tags[0] != env.tag {
t.Fatalf("unexpected tag found in tags: %v != %v", tags, []string{tag}) t.Fatalf("unexpected tag found in tags: %v != %v", tags, []string{env.tag})
} }
// Now, push the same manifest with a different key // Now, push the same manifest with a different key
@ -182,11 +229,11 @@ func TestManifestStorage(t *testing.T) {
t.Fatalf("unexpected number of signatures: %d != %d", len(sigs2), 1) t.Fatalf("unexpected number of signatures: %d != %d", len(sigs2), 1)
} }
if err = ms.Put(tag, sm2); err != nil { if err = ms.Put(sm2); err != nil {
t.Fatalf("unexpected error putting manifest: %v", err) t.Fatalf("unexpected error putting manifest: %v", err)
} }
fetched, err := ms.Get(tag) fetched, err := ms.GetByTag(env.tag)
if err != nil { if err != nil {
t.Fatalf("unexpected error fetching manifest: %v", err) t.Fatalf("unexpected error fetching manifest: %v", err)
} }
@ -231,7 +278,11 @@ func TestManifestStorage(t *testing.T) {
} }
} }
if err := ms.Delete(tag); err != nil { // TODO(stevvooe): Currently, deletes are not supported due to some
t.Fatalf("unexpected error deleting manifest: %v", err) // complexity around managing tag indexes. We'll add this support back in
// when the manifest format has settled. For now, we expect an error for
// all deletes.
if err := ms.Delete(dgst); err == nil {
t.Fatalf("unexpected an error deleting manifest by digest: %v", err)
} }
} }

29
registry/storage/paths.go
View file

@ -76,7 +76,8 @@ const storagePathVersion = "v2"
// manifestTagPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/ // manifestTagPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/
// manifestTagCurrentPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/current/link // manifestTagCurrentPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/current/link
// manifestTagIndexPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/index/ // manifestTagIndexPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/index/
// manifestTagIndexEntryPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/index/<algorithm>/<hex digest>/link // manifestTagIndexEntryPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/index/<algorithm>/<hex digest>/
// manifestTagIndexEntryLinkPathSpec: <root>/v2/repositories/<name>/_manifests/tags/<tag>/index/<algorithm>/<hex digest>/link
// //
// Layers: // Layers:
// //
@ -199,6 +200,17 @@ func (pm *pathMapper) path(spec pathSpec) (string, error) {
} }
return path.Join(root, "index"), nil return path.Join(root, "index"), nil
case manifestTagIndexEntryLinkPathSpec:
root, err := pm.path(manifestTagIndexEntryPathSpec{
name: v.name,
tag: v.tag,
revision: v.revision,
})
if err != nil {
return "", err
}
return path.Join(root, "link"), nil
case manifestTagIndexEntryPathSpec: case manifestTagIndexEntryPathSpec:
root, err := pm.path(manifestTagIndexPathSpec{ root, err := pm.path(manifestTagIndexPathSpec{
name: v.name, name: v.name,
@ -213,7 +225,7 @@ func (pm *pathMapper) path(spec pathSpec) (string, error) {
return "", err return "", err
} }
return path.Join(root, path.Join(append(components, "link")...)), nil return path.Join(root, path.Join(components...)), nil
case layerLinkPathSpec: case layerLinkPathSpec:
components, err := digestPathComponents(v.digest, false) components, err := digestPathComponents(v.digest, false)
if err != nil { if err != nil {
@ -332,8 +344,7 @@ type manifestTagIndexPathSpec struct {
func (manifestTagIndexPathSpec) pathSpec() {} func (manifestTagIndexPathSpec) pathSpec() {}
// manifestTagIndexEntryPathSpec describes the link to a revisions of a // manifestTagIndexEntryPathSpec contains the entries of the index by revision.
// manifest with given tag within the index.
type manifestTagIndexEntryPathSpec struct { type manifestTagIndexEntryPathSpec struct {
name string name string
tag string tag string
@ -342,6 +353,16 @@ type manifestTagIndexEntryPathSpec struct {
func (manifestTagIndexEntryPathSpec) pathSpec() {} func (manifestTagIndexEntryPathSpec) pathSpec() {}
// manifestTagIndexEntryLinkPathSpec describes the link to a revisions of a
// manifest with given tag within the index.
type manifestTagIndexEntryLinkPathSpec struct {
name string
tag string
revision digest.Digest
}
func (manifestTagIndexEntryLinkPathSpec) pathSpec() {}
// layerLink specifies a path for a layer link, which is a file with a blob // layerLink specifies a path for a layer link, which is a file with a blob
// id. The layer link will contain a content addressable blob id reference // id. The layer link will contain a content addressable blob id reference
// into the blob store. The format of the contents is as follows: // into the blob store. The format of the contents is as follows:

8
registry/storage/paths_test.go
View file

@ -78,6 +78,14 @@ func TestPathMapper(t *testing.T) {
tag: "thetag", tag: "thetag",
revision: "sha256:abcdef0123456789", revision: "sha256:abcdef0123456789",
}, },
expected: "/pathmapper-test/repositories/foo/bar/_manifests/tags/thetag/index/sha256/abcdef0123456789",
},
{
spec: manifestTagIndexEntryLinkPathSpec{
name: "foo/bar",
tag: "thetag",
revision: "sha256:abcdef0123456789",
},
expected: "/pathmapper-test/repositories/foo/bar/_manifests/tags/thetag/index/sha256/abcdef0123456789/link", expected: "/pathmapper-test/repositories/foo/bar/_manifests/tags/thetag/index/sha256/abcdef0123456789/link",
}, },
{ {

2
registry/storage/tagstore.go
View file

@ -63,7 +63,7 @@ func (ts *tagStore) exists(tag string) (bool, error) {
// tag tags the digest with the given tag, updating the the store to point at // tag tags the digest with the given tag, updating the the store to point at
// the current tag. The digest must point to a manifest. // the current tag. The digest must point to a manifest.
func (ts *tagStore) tag(tag string, revision digest.Digest) error { func (ts *tagStore) tag(tag string, revision digest.Digest) error {
indexEntryPath, err := ts.pm.path(manifestTagIndexEntryPathSpec{ indexEntryPath, err := ts.pm.path(manifestTagIndexEntryLinkPathSpec{
name: ts.Name(), name: ts.Name(),
tag: tag, tag: tag,
revision: revision, revision: revision,