vbatts/containerd
1
0
Fork 0
Code Pull requests Releases Activity
containerd/cmd/dist/main.go

84 lines
1.7 KiB
Go
Raw Normal View History

cmd/dist: POC implementation of dist fetch With this changeset we introduce several new things. The first is the top-level dist command. This is a toolkit that implements various distribution primitives, such as fetching, unpacking and ingesting. The first component to this is a simple `fetch` command. It is a low-level command that takes a "remote", identified by a `locator`, and an object identifier. Keyed by the locator, this tool can identify a remote implementation to fetch the content and write it back to standard out. By allowing this to be the unit of pluggability in fetching content, we can have quite a bit of flexibility in how we retrieve images. The current `fetch` implementation provides anonymous access to docker hub images, through the namespace `docker.io`. As an example, one can fetch the manifest for `redis` with the following command: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json ``` Note that we have provided a mediatype "hint", nudging the fetch implementation to grab the correct endpoint. We can hash the output of that to fetch the same content by digest: ``` $ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256) ``` Note that the hint is now elided, since we have affixed the content to a particular hash. If you are not yet entertained, let's bring `jq` and `xargs` into the mix for maximum fun. The following incantation fetches the same manifest and downloads all layers into the convenience of `/dev/null`: ``` $ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null ``` This is just the beginning. We should be able to centralize configuration around fetch to implement a number of distribution methodologies that have been challenging or impossible up to this point. The `locator`, mentioned earlier, is a schemaless URL that provides a host and path that can be used to resolve the remote. By dispatching on this common identifier, we should be able to support almost any protocol and discovery mechanism imaginable. When this is more solidified, we can roll these up into higher-level operations that can be orchestrated through the `dist` tool or via GRPC. What a time to be alive! Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-20 03:03:44 +00:00
package main
import (
content: refactor content store for API After iterating on the GRPC API, the changes required for the actual implementation are now included in the content store. The begin change is the move to a single, atomic `Ingester.Writer` method for locking content ingestion on a key. From this, comes several new interface definitions. The main benefit here is the clarification between `Status` and `Info` that came out of the GPRC API. `Status` tells the status of a write, whereas `Info` is for querying metadata about various blobs. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 08:07:02 +00:00
contextpkg "context"
cmd/dist: POC implementation of dist fetch With this changeset we introduce several new things. The first is the top-level dist command. This is a toolkit that implements various distribution primitives, such as fetching, unpacking and ingesting. The first component to this is a simple `fetch` command. It is a low-level command that takes a "remote", identified by a `locator`, and an object identifier. Keyed by the locator, this tool can identify a remote implementation to fetch the content and write it back to standard out. By allowing this to be the unit of pluggability in fetching content, we can have quite a bit of flexibility in how we retrieve images. The current `fetch` implementation provides anonymous access to docker hub images, through the namespace `docker.io`. As an example, one can fetch the manifest for `redis` with the following command: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json ``` Note that we have provided a mediatype "hint", nudging the fetch implementation to grab the correct endpoint. We can hash the output of that to fetch the same content by digest: ``` $ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256) ``` Note that the hint is now elided, since we have affixed the content to a particular hash. If you are not yet entertained, let's bring `jq` and `xargs` into the mix for maximum fun. The following incantation fetches the same manifest and downloads all layers into the convenience of `/dev/null`: ``` $ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null ``` This is just the beginning. We should be able to centralize configuration around fetch to implement a number of distribution methodologies that have been challenging or impossible up to this point. The `locator`, mentioned earlier, is a schemaless URL that provides a host and path that can be used to resolve the remote. By dispatching on this common identifier, we should be able to support almost any protocol and discovery mechanism imaginable. When this is more solidified, we can roll these up into higher-level operations that can be orchestrated through the `dist` tool or via GRPC. What a time to be alive! Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-20 03:03:44 +00:00
"fmt"
"os"
"github.com/Sirupsen/logrus"
"github.com/docker/containerd"
"github.com/urfave/cli"
)
content: refactor content store for API After iterating on the GRPC API, the changes required for the actual implementation are now included in the content store. The begin change is the move to a single, atomic `Ingester.Writer` method for locking content ingestion on a key. From this, comes several new interface definitions. The main benefit here is the clarification between `Status` and `Info` that came out of the GPRC API. `Status` tells the status of a write, whereas `Info` is for querying metadata about various blobs. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 08:07:02 +00:00
var (
background = contextpkg.Background()
)
cmd/dist: POC implementation of dist fetch With this changeset we introduce several new things. The first is the top-level dist command. This is a toolkit that implements various distribution primitives, such as fetching, unpacking and ingesting. The first component to this is a simple `fetch` command. It is a low-level command that takes a "remote", identified by a `locator`, and an object identifier. Keyed by the locator, this tool can identify a remote implementation to fetch the content and write it back to standard out. By allowing this to be the unit of pluggability in fetching content, we can have quite a bit of flexibility in how we retrieve images. The current `fetch` implementation provides anonymous access to docker hub images, through the namespace `docker.io`. As an example, one can fetch the manifest for `redis` with the following command: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json ``` Note that we have provided a mediatype "hint", nudging the fetch implementation to grab the correct endpoint. We can hash the output of that to fetch the same content by digest: ``` $ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256) ``` Note that the hint is now elided, since we have affixed the content to a particular hash. If you are not yet entertained, let's bring `jq` and `xargs` into the mix for maximum fun. The following incantation fetches the same manifest and downloads all layers into the convenience of `/dev/null`: ``` $ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null ``` This is just the beginning. We should be able to centralize configuration around fetch to implement a number of distribution methodologies that have been challenging or impossible up to this point. The `locator`, mentioned earlier, is a schemaless URL that provides a host and path that can be used to resolve the remote. By dispatching on this common identifier, we should be able to support almost any protocol and discovery mechanism imaginable. When this is more solidified, we can roll these up into higher-level operations that can be orchestrated through the `dist` tool or via GRPC. What a time to be alive! Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-20 03:03:44 +00:00
func main() {
version: finish version setup This setup will now correctly set the version number from the git tag. When using `--version`, we will see the binary name, the package it was built from and a git hash based on the tag: ```console $./bin/dist -v ./bin/dist github.com/docker/containerd 0b45d91.m ``` Note that in the above example, if we set a tag of `v1.0.0-dev`, that will show up in the version number, as follows: ```console $./bin/dist -v ./bin/dist github.com/docker/containerd v1.0.0-dev ``` Once commits are made past that tag, the version number will be expressed relative to that tag and include a git hash: ```console $./bin/dist -v ./bin/dist github.com/docker/containerd v1.0.0-dev-1-g7953e96.m ``` Some these examples include a `.m` postfix. This indicates that the binary was build from a source tree with local modifications. We can add a dev tag to start getting 1.0 version numbers for test builds. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-22 21:16:06 +00:00
cli.VersionPrinter = func(c *cli.Context) {
fmt.Println(os.Args[0], containerd.Package, containerd.Version)
}
cmd/dist: POC implementation of dist fetch With this changeset we introduce several new things. The first is the top-level dist command. This is a toolkit that implements various distribution primitives, such as fetching, unpacking and ingesting. The first component to this is a simple `fetch` command. It is a low-level command that takes a "remote", identified by a `locator`, and an object identifier. Keyed by the locator, this tool can identify a remote implementation to fetch the content and write it back to standard out. By allowing this to be the unit of pluggability in fetching content, we can have quite a bit of flexibility in how we retrieve images. The current `fetch` implementation provides anonymous access to docker hub images, through the namespace `docker.io`. As an example, one can fetch the manifest for `redis` with the following command: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json ``` Note that we have provided a mediatype "hint", nudging the fetch implementation to grab the correct endpoint. We can hash the output of that to fetch the same content by digest: ``` $ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256) ``` Note that the hint is now elided, since we have affixed the content to a particular hash. If you are not yet entertained, let's bring `jq` and `xargs` into the mix for maximum fun. The following incantation fetches the same manifest and downloads all layers into the convenience of `/dev/null`: ``` $ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null ``` This is just the beginning. We should be able to centralize configuration around fetch to implement a number of distribution methodologies that have been challenging or impossible up to this point. The `locator`, mentioned earlier, is a schemaless URL that provides a host and path that can be used to resolve the remote. By dispatching on this common identifier, we should be able to support almost any protocol and discovery mechanism imaginable. When this is more solidified, we can roll these up into higher-level operations that can be orchestrated through the `dist` tool or via GRPC. What a time to be alive! Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-20 03:03:44 +00:00
app := cli.NewApp()
app.Name = "dist"
app.Version = containerd.Version
app.Usage = `
___ __
____/ (_)____/ /_
/ __ / / ___/ __/
/ /_/ / (__ ) /_
\__,_/_/____/\__/
distribution tool
`
app.Flags = []cli.Flag{
cli.BoolFlag{
Name: "debug",
Usage: "enable debug output in logs",
},
content: refactor content store for API After iterating on the GRPC API, the changes required for the actual implementation are now included in the content store. The begin change is the move to a single, atomic `Ingester.Writer` method for locking content ingestion on a key. From this, comes several new interface definitions. The main benefit here is the clarification between `Status` and `Info` that came out of the GPRC API. `Status` tells the status of a write, whereas `Info` is for querying metadata about various blobs. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 08:07:02 +00:00
cli.DurationFlag{
Name: "timeout",
Usage: "total timeout for fetch",
EnvVar: "CONTAINERD_FETCH_TIMEOUT",
},
cli.StringFlag{
content: cleanup service and interfaces After implementing pull, a few changes are required to the content store interface to make sure that the implementation works smoothly. Specifically, we work to make sure the predeclaration path for digests works the same between remote and local writers. Before, we were hesitent to require the the size and digest up front, but it became clear that having this provided significant benefit. There are also several cleanups related to naming. We now call the expected digest `Expected` consistently across the board and `Total` is used to mark the expected size. This whole effort comes together to provide a very smooth status reporting workflow for image pull and push. This will be more obvious when the bulk of pull code lands. There are a few other changes to make `content.WriteBlob` more broadly useful. In accordance with addition for predeclaring expected size when getting a `Writer`, `WriteBlob` now supports this fully. It will also resume downloads if provided an `io.Seeker` or `io.ReaderAt`. Coupled with the `httpReadSeeker` from `docker/distribution`, we should only be a lines of code away from resumable downloads. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-22 07:41:11 +00:00
// TODO(stevvooe): for now, we allow circumventing the GRPC. Once
// we have clear separation, this will likely go away.
content: refactor content store for API After iterating on the GRPC API, the changes required for the actual implementation are now included in the content store. The begin change is the move to a single, atomic `Ingester.Writer` method for locking content ingestion on a key. From this, comes several new interface definitions. The main benefit here is the clarification between `Status` and `Info` that came out of the GPRC API. `Status` tells the status of a write, whereas `Info` is for querying metadata about various blobs. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 08:07:02 +00:00
Name: "root",
Usage: "path to content store root",
content: cleanup service and interfaces After implementing pull, a few changes are required to the content store interface to make sure that the implementation works smoothly. Specifically, we work to make sure the predeclaration path for digests works the same between remote and local writers. Before, we were hesitent to require the the size and digest up front, but it became clear that having this provided significant benefit. There are also several cleanups related to naming. We now call the expected digest `Expected` consistently across the board and `Total` is used to mark the expected size. This whole effort comes together to provide a very smooth status reporting workflow for image pull and push. This will be more obvious when the bulk of pull code lands. There are a few other changes to make `content.WriteBlob` more broadly useful. In accordance with addition for predeclaring expected size when getting a `Writer`, `WriteBlob` now supports this fully. It will also resume downloads if provided an `io.Seeker` or `io.ReaderAt`. Coupled with the `httpReadSeeker` from `docker/distribution`, we should only be a lines of code away from resumable downloads. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-22 07:41:11 +00:00
Value: "/var/lib/containerd",
content: refactor content store for API After iterating on the GRPC API, the changes required for the actual implementation are now included in the content store. The begin change is the move to a single, atomic `Ingester.Writer` method for locking content ingestion on a key. From this, comes several new interface definitions. The main benefit here is the clarification between `Status` and `Info` that came out of the GPRC API. `Status` tells the status of a write, whereas `Info` is for querying metadata about various blobs. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 08:07:02 +00:00
},
cmd/dist: port commands over to use GRPC content store Following from the rest of the work in this branch, we now are porting the dist command to work directly against the containerd content API. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-18 00:49:59 +00:00
cli.StringFlag{
Name: "socket, s",
Usage: "socket path for containerd's GRPC server",
Value: "/run/containerd/containerd.sock",
},
cmd/dist: POC implementation of dist fetch With this changeset we introduce several new things. The first is the top-level dist command. This is a toolkit that implements various distribution primitives, such as fetching, unpacking and ingesting. The first component to this is a simple `fetch` command. It is a low-level command that takes a "remote", identified by a `locator`, and an object identifier. Keyed by the locator, this tool can identify a remote implementation to fetch the content and write it back to standard out. By allowing this to be the unit of pluggability in fetching content, we can have quite a bit of flexibility in how we retrieve images. The current `fetch` implementation provides anonymous access to docker hub images, through the namespace `docker.io`. As an example, one can fetch the manifest for `redis` with the following command: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json ``` Note that we have provided a mediatype "hint", nudging the fetch implementation to grab the correct endpoint. We can hash the output of that to fetch the same content by digest: ``` $ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256) ``` Note that the hint is now elided, since we have affixed the content to a particular hash. If you are not yet entertained, let's bring `jq` and `xargs` into the mix for maximum fun. The following incantation fetches the same manifest and downloads all layers into the convenience of `/dev/null`: ``` $ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null ``` This is just the beginning. We should be able to centralize configuration around fetch to implement a number of distribution methodologies that have been challenging or impossible up to this point. The `locator`, mentioned earlier, is a schemaless URL that provides a host and path that can be used to resolve the remote. By dispatching on this common identifier, we should be able to support almost any protocol and discovery mechanism imaginable. When this is more solidified, we can roll these up into higher-level operations that can be orchestrated through the `dist` tool or via GRPC. What a time to be alive! Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-20 03:03:44 +00:00
}
app.Commands = []cli.Command{
fetchCommand,
dist: expand functionality of the dist tool With this change, we add the following commands to the dist tool: - `ingest`: verify and accept content into storage - `active`: display active ingest processes - `list`: list content in storage - `path`: provide a path to a blob by digest - `delete`: remove a piece of content from storage We demonstrate the utility with the following shell pipeline: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | \ jq -r '.layers[] | "./dist fetch docker.io/library/redis "+.digest + "| ./dist ingest --expected-digest "+.digest+" --expected-size "+(.size | tostring) +" docker.io/library/redis@"+.digest' | xargs -I{} -P10 -n1 sh -c "{}" ``` The above fetches a manifest, pipes it to jq, which assembles a shell pipeline to ingest each layer into the content store. Because the transactions are keyed by their digest, concurrent downloads and downloads of repeated content are ignored. Each process is then executed parallel using xargs. Put shortly, this is a parallel layer download. In a separate shell session, could monitor the active downloads with the following: ``` $ watch -n0.2 ./dist active ``` For now, the content is downloaded into `.content` in the current working directory. To watch the contents of this directory, you can use the following: ``` $ watch -n0.2 tree .content ``` This will help to understand what is going on internally. To get access to the layers, you can use the path command: ``` $./dist path sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa /home/sjd/go/src/github.com/docker/containerd/.content/blobs/sha256/010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa ``` When you are done, you can clear out the content with the classic xargs pipeline: ``` $ ./dist list -q | xargs ./dist delete ``` Note that this is mostly a POC. Things like failed downloads and abandoned download cleanup aren't quite handled. We'll probably make adjustments around how content store transactions are handled to address this. From here, we'll build out full image pull and create tooling to get runtime bundles from the fetched content. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-26 22:08:56 +00:00
ingestCommand,
activeCommand,
content: refactor content store for API After iterating on the GRPC API, the changes required for the actual implementation are now included in the content store. The begin change is the move to a single, atomic `Ingester.Writer` method for locking content ingestion on a key. From this, comes several new interface definitions. The main benefit here is the clarification between `Status` and `Info` that came out of the GPRC API. `Status` tells the status of a write, whereas `Info` is for querying metadata about various blobs. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 08:07:02 +00:00
getCommand,
dist: expand functionality of the dist tool With this change, we add the following commands to the dist tool: - `ingest`: verify and accept content into storage - `active`: display active ingest processes - `list`: list content in storage - `path`: provide a path to a blob by digest - `delete`: remove a piece of content from storage We demonstrate the utility with the following shell pipeline: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | \ jq -r '.layers[] | "./dist fetch docker.io/library/redis "+.digest + "| ./dist ingest --expected-digest "+.digest+" --expected-size "+(.size | tostring) +" docker.io/library/redis@"+.digest' | xargs -I{} -P10 -n1 sh -c "{}" ``` The above fetches a manifest, pipes it to jq, which assembles a shell pipeline to ingest each layer into the content store. Because the transactions are keyed by their digest, concurrent downloads and downloads of repeated content are ignored. Each process is then executed parallel using xargs. Put shortly, this is a parallel layer download. In a separate shell session, could monitor the active downloads with the following: ``` $ watch -n0.2 ./dist active ``` For now, the content is downloaded into `.content` in the current working directory. To watch the contents of this directory, you can use the following: ``` $ watch -n0.2 tree .content ``` This will help to understand what is going on internally. To get access to the layers, you can use the path command: ``` $./dist path sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa /home/sjd/go/src/github.com/docker/containerd/.content/blobs/sha256/010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa ``` When you are done, you can clear out the content with the classic xargs pipeline: ``` $ ./dist list -q | xargs ./dist delete ``` Note that this is mostly a POC. Things like failed downloads and abandoned download cleanup aren't quite handled. We'll probably make adjustments around how content store transactions are handled to address this. From here, we'll build out full image pull and create tooling to get runtime bundles from the fetched content. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-26 22:08:56 +00:00
deleteCommand,
listCommand,
dist: provide apply command to build rootfs This changeset adds the simple apply command. It consumes a tar layer and applies that layer to the specified directory. For the most part, it is a direct call into Docker's `pkg/archive.ApplyLayer`. The following demonstrates unpacking the wordpress rootfs into a local directory `wordpress`: ``` $ ./dist fetch docker.io/library/wordpress 4.5 mediatype:application/vnd.docker.distribution.manifest.v2+json | \ jq -r '.layers[] | "sudo ./dist apply ./wordpress < $(./dist path -n "+.digest+")"' | xargs -I{} -n1 sh -c "{}" ``` Note that you should have fetched the layers into the local content store before running the above. Alternatively, you can just read the manifest from the content store, rather than fetching it. We use fetch above to avoid having to lookup the manifest digest for our demo. This tool has a long way to go. We still need to incorporate snapshotting, as well as the ability to calculate the `ChainID` under subsequent unpacking. Once we have some tools to play around with snapshotting, we'll be able to incorporate our `rootfs.ApplyLayer` algorithm that will get us a lot closer to a production worthy system. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 19:00:29 +00:00
applyCommand,
cmd/dist: POC implementation of dist fetch With this changeset we introduce several new things. The first is the top-level dist command. This is a toolkit that implements various distribution primitives, such as fetching, unpacking and ingesting. The first component to this is a simple `fetch` command. It is a low-level command that takes a "remote", identified by a `locator`, and an object identifier. Keyed by the locator, this tool can identify a remote implementation to fetch the content and write it back to standard out. By allowing this to be the unit of pluggability in fetching content, we can have quite a bit of flexibility in how we retrieve images. The current `fetch` implementation provides anonymous access to docker hub images, through the namespace `docker.io`. As an example, one can fetch the manifest for `redis` with the following command: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json ``` Note that we have provided a mediatype "hint", nudging the fetch implementation to grab the correct endpoint. We can hash the output of that to fetch the same content by digest: ``` $ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256) ``` Note that the hint is now elided, since we have affixed the content to a particular hash. If you are not yet entertained, let's bring `jq` and `xargs` into the mix for maximum fun. The following incantation fetches the same manifest and downloads all layers into the convenience of `/dev/null`: ``` $ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null ``` This is just the beginning. We should be able to centralize configuration around fetch to implement a number of distribution methodologies that have been challenging or impossible up to this point. The `locator`, mentioned earlier, is a schemaless URL that provides a host and path that can be used to resolve the remote. By dispatching on this common identifier, we should be able to support almost any protocol and discovery mechanism imaginable. When this is more solidified, we can roll these up into higher-level operations that can be orchestrated through the `dist` tool or via GRPC. What a time to be alive! Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-20 03:03:44 +00:00
}
app.Before = func(context *cli.Context) error {
content: refactor content store for API After iterating on the GRPC API, the changes required for the actual implementation are now included in the content store. The begin change is the move to a single, atomic `Ingester.Writer` method for locking content ingestion on a key. From this, comes several new interface definitions. The main benefit here is the clarification between `Status` and `Info` that came out of the GPRC API. `Status` tells the status of a write, whereas `Info` is for querying metadata about various blobs. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 08:07:02 +00:00
var (
debug = context.GlobalBool("debug")
timeout = context.GlobalDuration("timeout")
)
if debug {
cmd/dist: POC implementation of dist fetch With this changeset we introduce several new things. The first is the top-level dist command. This is a toolkit that implements various distribution primitives, such as fetching, unpacking and ingesting. The first component to this is a simple `fetch` command. It is a low-level command that takes a "remote", identified by a `locator`, and an object identifier. Keyed by the locator, this tool can identify a remote implementation to fetch the content and write it back to standard out. By allowing this to be the unit of pluggability in fetching content, we can have quite a bit of flexibility in how we retrieve images. The current `fetch` implementation provides anonymous access to docker hub images, through the namespace `docker.io`. As an example, one can fetch the manifest for `redis` with the following command: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json ``` Note that we have provided a mediatype "hint", nudging the fetch implementation to grab the correct endpoint. We can hash the output of that to fetch the same content by digest: ``` $ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256) ``` Note that the hint is now elided, since we have affixed the content to a particular hash. If you are not yet entertained, let's bring `jq` and `xargs` into the mix for maximum fun. The following incantation fetches the same manifest and downloads all layers into the convenience of `/dev/null`: ``` $ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null ``` This is just the beginning. We should be able to centralize configuration around fetch to implement a number of distribution methodologies that have been challenging or impossible up to this point. The `locator`, mentioned earlier, is a schemaless URL that provides a host and path that can be used to resolve the remote. By dispatching on this common identifier, we should be able to support almost any protocol and discovery mechanism imaginable. When this is more solidified, we can roll these up into higher-level operations that can be orchestrated through the `dist` tool or via GRPC. What a time to be alive! Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-20 03:03:44 +00:00
logrus.SetLevel(logrus.DebugLevel)
}
content: refactor content store for API After iterating on the GRPC API, the changes required for the actual implementation are now included in the content store. The begin change is the move to a single, atomic `Ingester.Writer` method for locking content ingestion on a key. From this, comes several new interface definitions. The main benefit here is the clarification between `Status` and `Info` that came out of the GPRC API. `Status` tells the status of a write, whereas `Info` is for querying metadata about various blobs. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 08:07:02 +00:00
if timeout > 0 {
background, _ = contextpkg.WithTimeout(background, timeout)
}
cmd/dist: POC implementation of dist fetch With this changeset we introduce several new things. The first is the top-level dist command. This is a toolkit that implements various distribution primitives, such as fetching, unpacking and ingesting. The first component to this is a simple `fetch` command. It is a low-level command that takes a "remote", identified by a `locator`, and an object identifier. Keyed by the locator, this tool can identify a remote implementation to fetch the content and write it back to standard out. By allowing this to be the unit of pluggability in fetching content, we can have quite a bit of flexibility in how we retrieve images. The current `fetch` implementation provides anonymous access to docker hub images, through the namespace `docker.io`. As an example, one can fetch the manifest for `redis` with the following command: ``` $ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json ``` Note that we have provided a mediatype "hint", nudging the fetch implementation to grab the correct endpoint. We can hash the output of that to fetch the same content by digest: ``` $ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256) ``` Note that the hint is now elided, since we have affixed the content to a particular hash. If you are not yet entertained, let's bring `jq` and `xargs` into the mix for maximum fun. The following incantation fetches the same manifest and downloads all layers into the convenience of `/dev/null`: ``` $ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null ``` This is just the beginning. We should be able to centralize configuration around fetch to implement a number of distribution methodologies that have been challenging or impossible up to this point. The `locator`, mentioned earlier, is a schemaless URL that provides a host and path that can be used to resolve the remote. By dispatching on this common identifier, we should be able to support almost any protocol and discovery mechanism imaginable. When this is more solidified, we can roll these up into higher-level operations that can be orchestrated through the `dist` tool or via GRPC. What a time to be alive! Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-20 03:03:44 +00:00
return nil
}
if err := app.Run(os.Args); err != nil {
fmt.Fprintf(os.Stderr, "dist: %s\n", err)
os.Exit(1)
}
}
Copy permalink