Commit graph

989 commits

Author SHA1 Message Date
Derek McGowan
245495d54e Use full mode for making devices node
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-03 11:28:06 -08:00
Derek McGowan
bf8f37ba78 Remove incorrect and unused timespec check
Compare is using its own time check comparison and
doing byte comparison when ambiguous rather than
ignoring it like this function does.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-03 11:28:06 -08:00
Derek McGowan
f78105d832 Add support for devices
Address code comments from previous commit

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-03 11:28:06 -08:00
Derek McGowan
200cd6e877 Update snapshots to use fs package
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-03 11:28:06 -08:00
Derek McGowan
574862fd89 Add fs package
Add diff comparison with support for double walking
two trees for comparison or single walking a diff
tree. Single walking requires further implementation
for specific mount types.

Add directory copy function which is intended to provide
fastest possible local copy of file system directories
without hardlinking.

Add test package to make creating filesystems for
test easy and comparisons deep and informative.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-03 11:27:40 -08:00
Derek McGowan
5f08e609c0 Vendor continuity
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-03 11:27:40 -08:00
Kenfe-Mickaël Laventure
c6446b93da Merge pull request #484 from stevvooe/snapshot-drivers
[WIP] snapshot: define the snapshot driver interface
2017-02-03 07:56:12 -08:00
Stephen J Day
d0b4ce8d17
snapshot/overlay: add snapshot test suite to overlay
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-02 18:55:53 -08:00
Stephen J Day
68fd25221d
snapshot: define the snapshot driver interface
We now define the `snapshot.Driver` interface based on earlier work.
Many details of the model are worked out, such as snapshot lifecycle and
parentage of commits against "Active" snapshots.

The impetus of this change is to provide a snapshot POC that does a
complete push/pull workflow. The beginnings of a test suite for snapshot
drivers is included that we can use to verify the assumptions of
drivers. The intent is to port the existing tests over to this test
suite and start scaling contributions and test to the snapshot driver
subsystem.

There are still some details that need to be worked out, such as listing
and metadata access. We can do this activity as we further integrate
with tooling.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-02 18:06:02 -08:00
Derek McGowan
29132c4061 Merge pull request #487 from stevvooe/use-pool-in-copy
content: use buffer pool in hash copy
2017-02-02 15:42:46 -08:00
Kenfe-Mickael Laventure
6f9eda1134 api/execution: remove ProcessID from rpc calls
Now that the shim handles all container's processes the system pid is
sufficient.

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-02-02 15:36:10 -08:00
Kenfe-Mickael Laventure
40b0b211b7 api/execution: add Container suffix to relevant rpc calls
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-02-02 14:40:23 -08:00
Stephen J Day
5ff837e2e7
content: use buffer pool in hash copy
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-02 14:27:52 -08:00
Michael Crosby
8c3158bf5f Merge pull request #485 from crosbymichael/supervisor
Port over supervisor to use grpc shim
2017-02-01 16:15:56 -08:00
Michael Crosby
f187da9485 Port over supervisor to use grpc shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-02-01 16:01:53 -08:00
Daniel, Dao Quang Minh
e1eeb40d1d Merge pull request #478 from kunalkushwaha/profiler
Exposing pprof and expvars interfaces on default containerd socket
2017-02-01 18:17:04 +00:00
Kunal Kushwaha
6af4f5fee8 Exposing pprof and expvars interfaces on default containerd socket
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2017-02-01 11:05:55 +09:00
Stephen Day
78b55a1301 Merge pull request #482 from samuelkarp/remove-binary-file
snapshot: Remove mistakenly-added binary file
2017-01-30 14:08:49 -08:00
Samuel Karp
3c37c9e6b1 snapshot: Remove mistakenly-added binary file
Signed-off-by: Samuel Karp <skarp@amazon.com>
2017-01-30 13:57:58 -08:00
Kenfe-Mickaël Laventure
5f28d28738 Merge pull request #477 from teddyking/mkdirall-root-dir
os.MkdirAll the containerd root dir
2017-01-30 07:34:43 -08:00
Ed King
89607a10db os.MkdirAll the containerd root dir
Signed-off-by: Ed King <ed@teddyking.co.uk>
2017-01-28 15:47:51 +00:00
Stephen Day
d7bea77cc3 Merge pull request #473 from stevvooe/dmcgowan-maintainer
MAINTAINERS: add dmcgowan
2017-01-27 13:23:31 -08:00
Michael Crosby
9278a3310f Merge pull request #475 from stevvooe/dev-report-2017-01-27
reports: development report for 2017-01-27
2017-01-27 13:15:18 -08:00
Stephen J Day
ff0ddaa28e
reports: development report for 2017-01-27
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 12:38:40 -08:00
Michael Crosby
b12308a5be Merge pull request #437 from crosbymichael/lifecycle
Add container lifecycle document
2017-01-27 12:03:53 -08:00
Michael Crosby
c159fb22cc Add container lifecycle document
Closes #383

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-27 11:58:07 -08:00
Michael Crosby
dd866d6569 Merge pull request #413 from gliptak/coverage1
Enable codecov.io
2017-01-27 11:52:40 -08:00
Michael Crosby
9aab1cd89e Merge pull request #431 from hqhq/add_logrus_info
Add some logrus info for containerd
2017-01-27 11:50:32 -08:00
Michael Crosby
64a8f196ee Merge pull request #474 from stevvooe/apply-command
dist: provide apply command to build rootfs
2017-01-27 11:12:35 -08:00
Stephen J Day
3e0238612b
dist: provide apply command to build rootfs
This changeset adds the simple apply command. It consumes a tar layer
and applies that layer to the specified directory. For the most part, it
is a direct call into Docker's `pkg/archive.ApplyLayer`.

The following demonstrates unpacking the wordpress rootfs into a local
directory `wordpress`:

```
$ ./dist fetch docker.io/library/wordpress 4.5 mediatype:application/vnd.docker.distribution.manifest.v2+json | \
    jq -r '.layers[] | "sudo ./dist apply ./wordpress < $(./dist path -n "+.digest+")"' | xargs -I{} -n1 sh -c "{}"
```

Note that you should have fetched the layers into the local content
store before running the above. Alternatively, you can just read the
manifest from the content store, rather than fetching it. We use fetch
above to avoid having to lookup the manifest digest for our demo.

This tool has a long way to go. We still need to incorporate
snapshotting, as well as the ability to calculate the `ChainID` under
subsequent unpacking. Once we have some tools to play around with
snapshotting, we'll be able to incorporate our `rootfs.ApplyLayer`
algorithm that will get us a lot closer to a production worthy system.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 11:00:29 -08:00
Stephen J Day
4d51c553c8
MAINTAINERS: add dmcgowan
Derek McGowan will be joining as a maintainer of containerd. He will
bring his extensive experience in graphdrivers and distribution to
ensure success in this project.

Please welcome him!

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 10:43:29 -08:00
Stephen Day
594dca9e31 Merge pull request #472 from stevvooe/expanding-dist-tool
dist: expand functionality of the dist tool
2017-01-27 10:34:54 -08:00
Stephen J Day
f9cd9be61a
dist: expand functionality of the dist tool
With this change, we add the following commands to the dist tool:

- `ingest`: verify and accept content into storage
- `active`: display active ingest processes
- `list`: list content in storage
- `path`: provide a path to a blob by digest
- `delete`: remove a piece of content from storage

We demonstrate the utility with the following shell pipeline:

```
$ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | \
    jq -r '.layers[] | "./dist fetch docker.io/library/redis "+.digest + "| ./dist ingest --expected-digest "+.digest+" --expected-size "+(.size | tostring) +" docker.io/library/redis@"+.digest' | xargs -I{} -P10 -n1 sh -c "{}"
```

The above fetches a manifest, pipes it to jq, which assembles a shell
pipeline to ingest each layer into the content store. Because the
transactions are keyed by their digest, concurrent downloads and
downloads of repeated content are ignored. Each process is then executed
parallel using xargs.

Put shortly, this is a parallel layer download.

In a separate shell session, could monitor the active downloads with the
following:

```
$ watch -n0.2 ./dist active
```

For now, the content is downloaded into `.content` in the current
working directory. To watch the contents of this directory, you can use
the following:

```
$ watch -n0.2 tree .content
```

This will help to understand what is going on internally.

To get access to the layers, you can use the path command:

```
$./dist path sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa
sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa /home/sjd/go/src/github.com/docker/containerd/.content/blobs/sha256/010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa
```

When you are done, you can clear out the content with the classic xargs
pipeline:

```
$ ./dist list -q | xargs ./dist delete
```

Note that this is mostly a POC. Things like failed downloads and
abandoned download cleanup aren't quite handled. We'll probably make
adjustments around how content store transactions are handled to address
this.

From here, we'll build out full image pull and create tooling to get
runtime bundles from the fetched content.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 10:29:10 -08:00
Qiang Huang
7e4a7402f9 Add some logrus info for containerd
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2017-01-27 22:12:52 +08:00
Kenfe-Mickaël Laventure
24c2810899 Merge pull request #462 from crosbymichael/shim
shim: GRPC service
2017-01-26 16:11:45 -08:00
Michael Crosby
10c5a0ab3d Add exec io copy
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 16:07:43 -08:00
Michael Crosby
f70d485d72 Merge pull request #468 from jzwlqx/master
optimal help info for ctr cli.
2017-01-26 15:34:31 -08:00
Michael Crosby
f431bf4ad4 Add state rpc to shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 15:09:59 -08:00
Michael Crosby
ead53658cc bump go-runc to 706de6f422f397fb70b8c98f9b8c8eab2de32ae2
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 14:31:20 -08:00
Michael Crosby
b59bd59d8a Working tty and io support in shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
8e5e9ae70e Add console pkg and update go-runc
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
07c81ccac4 Add events api to shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
462bdd7669 Update go-runc to f36917a18b3d962aee066063cba0bcff44e338ca
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
d619954a2b Move shim service into top lvl package
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
fe280d2df0 Fix logrus import in shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
271cac8634 Update runc to ce450bcc6c135cae93ee2a99d41a308c179ff6dc
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
e09b0b0c35 Add exec functionality to shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
d5d2e586cd Refactor shim terminal and io handling
This also finishes the service implementation of the shim behind GRPC

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
6e9e0a895a Update go-runc to 7b66c5da30493c5eb9c655cab67ba88071891ac5
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
911bc57483 Update go-runc to abd8eada6a98acdf415d47a8d372dc314494c07f
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00