Commit graph

12 commits

Author SHA1 Message Date
fate-grand-order
d44c8525d7 fix misspell "scrutiny" in content/content.go
Signed-off-by: fate-grand-order <chenjg@harmonycloud.cn>
2017-02-07 15:06:40 +08:00
Akihiro Suda
e9510eb69c content: fix panic during filepath.Walk()
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-02-06 05:16:21 +00:00
Stephen J Day
5ff837e2e7
content: use buffer pool in hash copy
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-02 14:27:52 -08:00
Stephen J Day
f9cd9be61a
dist: expand functionality of the dist tool
With this change, we add the following commands to the dist tool:

- `ingest`: verify and accept content into storage
- `active`: display active ingest processes
- `list`: list content in storage
- `path`: provide a path to a blob by digest
- `delete`: remove a piece of content from storage

We demonstrate the utility with the following shell pipeline:

```
$ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | \
    jq -r '.layers[] | "./dist fetch docker.io/library/redis "+.digest + "| ./dist ingest --expected-digest "+.digest+" --expected-size "+(.size | tostring) +" docker.io/library/redis@"+.digest' | xargs -I{} -P10 -n1 sh -c "{}"
```

The above fetches a manifest, pipes it to jq, which assembles a shell
pipeline to ingest each layer into the content store. Because the
transactions are keyed by their digest, concurrent downloads and
downloads of repeated content are ignored. Each process is then executed
parallel using xargs.

Put shortly, this is a parallel layer download.

In a separate shell session, could monitor the active downloads with the
following:

```
$ watch -n0.2 ./dist active
```

For now, the content is downloaded into `.content` in the current
working directory. To watch the contents of this directory, you can use
the following:

```
$ watch -n0.2 tree .content
```

This will help to understand what is going on internally.

To get access to the layers, you can use the path command:

```
$./dist path sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa
sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa /home/sjd/go/src/github.com/docker/containerd/.content/blobs/sha256/010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa
```

When you are done, you can clear out the content with the classic xargs
pipeline:

```
$ ./dist list -q | xargs ./dist delete
```

Note that this is mostly a POC. Things like failed downloads and
abandoned download cleanup aren't quite handled. We'll probably make
adjustments around how content store transactions are handled to address
this.

From here, we'll build out full image pull and create tooling to get
runtime bundles from the fetched content.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 10:29:10 -08:00
Stephen J Day
f27d43285a
content: cleanup package exports
The package exports are now cleaned up to remove a lot of stuttering in
the API. We also remove direct mapping of refs to the filesystem, opting
for a hash-based approach. This *does* affect insertion performance,
since it requires more individual file ios. A benchmark baseline has
been added and we can fix this later.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-23 20:11:35 -08:00
Gábor Lipták
d8aee18f6c Point digest import to opencontainers/go-digest
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2017-01-09 18:10:52 -05:00
Stephen J Day
ab4bfc79e9
content: provide more context for lockfile error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-07 17:46:01 -08:00
Stephen J Day
47f8b25d25
content: add Walk method to content store
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-01 21:37:58 -08:00
Stephen J Day
9dea9654bd
content: query active downloads and status
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-11-15 20:29:43 -08:00
Stephen J Day
b6e446e7be
content: add cross-process ingest locking
Allow content stores to ingest content without coordination of a daemon
to manage locks. Supports coordinated ingest and cross-process ingest
status.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-11-15 20:29:10 -08:00
Stephen J Day
3469905bbb
content: break up into multiple files
Break up the content store prototype into a few logical files. We have a
file for the store, the writer and helpers.

Also, the writer has been modified to remove write and exec permissions
on blobs in the store.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-11-03 17:18:45 -07:00
Stephen J Day
9008471a84
content: initial implementation of content store
After experimenting with pull, we've defined a transactional content
store for verified storage of fetched items. A base component of
containerkit, this will interact with both the runtime and distribution
sides of the system, avoiding coupling.

Blob access if provided through direct access to readonly files.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-11-01 18:43:15 -07:00