We now include btrfs in the snapshot driver test suite. This includes
the addition of parent links and name hashing into the btrfs driver.
We'll probably endup replacing this with a common metadata store, as
these relationships are generally identical between implementations.
A small bug was discovered in the delete implementation in the course
testing, so the btrfs package has been updated with a fix.
The overlay driver was modified accordingly with the btrfs driver to use
`Driver` as the exported type of each driver packge.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Remove change type in favor of explicit change function.
Using change function makes it more difficult to unnecessarily
add to the change interface.
Update test apply functions to use an interface rather
than a function type.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The change computation will be done on the callers thread
and use callbacks rather than running a goroutine and
returning a channel.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Compare is using its own time check comparison and
doing byte comparison when ambiguous rather than
ignoring it like this function does.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Add diff comparison with support for double walking
two trees for comparison or single walking a diff
tree. Single walking requires further implementation
for specific mount types.
Add directory copy function which is intended to provide
fastest possible local copy of file system directories
without hardlinking.
Add test package to make creating filesystems for
test easy and comparisons deep and informative.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
We now define the `snapshot.Driver` interface based on earlier work.
Many details of the model are worked out, such as snapshot lifecycle and
parentage of commits against "Active" snapshots.
The impetus of this change is to provide a snapshot POC that does a
complete push/pull workflow. The beginnings of a test suite for snapshot
drivers is included that we can use to verify the assumptions of
drivers. The intent is to port the existing tests over to this test
suite and start scaling contributions and test to the snapshot driver
subsystem.
There are still some details that need to be worked out, such as listing
and metadata access. We can do this activity as we further integrate
with tooling.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset adds the simple apply command. It consumes a tar layer
and applies that layer to the specified directory. For the most part, it
is a direct call into Docker's `pkg/archive.ApplyLayer`.
The following demonstrates unpacking the wordpress rootfs into a local
directory `wordpress`:
```
$ ./dist fetch docker.io/library/wordpress 4.5 mediatype:application/vnd.docker.distribution.manifest.v2+json | \
jq -r '.layers[] | "sudo ./dist apply ./wordpress < $(./dist path -n "+.digest+")"' | xargs -I{} -n1 sh -c "{}"
```
Note that you should have fetched the layers into the local content
store before running the above. Alternatively, you can just read the
manifest from the content store, rather than fetching it. We use fetch
above to avoid having to lookup the manifest digest for our demo.
This tool has a long way to go. We still need to incorporate
snapshotting, as well as the ability to calculate the `ChainID` under
subsequent unpacking. Once we have some tools to play around with
snapshotting, we'll be able to incorporate our `rootfs.ApplyLayer`
algorithm that will get us a lot closer to a production worthy system.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Derek McGowan will be joining as a maintainer of containerd. He will
bring his extensive experience in graphdrivers and distribution to
ensure success in this project.
Please welcome him!
Signed-off-by: Stephen J Day <stephen.day@docker.com>
With this change, we add the following commands to the dist tool:
- `ingest`: verify and accept content into storage
- `active`: display active ingest processes
- `list`: list content in storage
- `path`: provide a path to a blob by digest
- `delete`: remove a piece of content from storage
We demonstrate the utility with the following shell pipeline:
```
$ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | \
jq -r '.layers[] | "./dist fetch docker.io/library/redis "+.digest + "| ./dist ingest --expected-digest "+.digest+" --expected-size "+(.size | tostring) +" docker.io/library/redis@"+.digest' | xargs -I{} -P10 -n1 sh -c "{}"
```
The above fetches a manifest, pipes it to jq, which assembles a shell
pipeline to ingest each layer into the content store. Because the
transactions are keyed by their digest, concurrent downloads and
downloads of repeated content are ignored. Each process is then executed
parallel using xargs.
Put shortly, this is a parallel layer download.
In a separate shell session, could monitor the active downloads with the
following:
```
$ watch -n0.2 ./dist active
```
For now, the content is downloaded into `.content` in the current
working directory. To watch the contents of this directory, you can use
the following:
```
$ watch -n0.2 tree .content
```
This will help to understand what is going on internally.
To get access to the layers, you can use the path command:
```
$./dist path sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa
sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa /home/sjd/go/src/github.com/docker/containerd/.content/blobs/sha256/010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa
```
When you are done, you can clear out the content with the classic xargs
pipeline:
```
$ ./dist list -q | xargs ./dist delete
```
Note that this is mostly a POC. Things like failed downloads and
abandoned download cleanup aren't quite handled. We'll probably make
adjustments around how content store transactions are handled to address
this.
From here, we'll build out full image pull and create tooling to get
runtime bundles from the fetched content.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
