Commit graph

141 commits

Author SHA1 Message Date
Stephen J Day
e6efb397cf
cmd/dist: port commands over to use GRPC content store
Following from the rest of the work in this branch, we now are porting
the dist command to work directly against the containerd content API.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-21 13:10:31 -08:00
Stephen J Day
621164bc84
content: refactor content store for API
After iterating on the GRPC API, the changes required for the actual
implementation are now included in the content store. The begin change
is the move to a single, atomic `Ingester.Writer` method for locking
content ingestion on a key. From this, comes several new interface
definitions.

The main benefit here is the clarification between `Status` and `Info`
that came out of the GPRC API. `Status` tells the status of a write,
whereas `Info` is for querying metadata about various blobs.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-21 13:10:22 -08:00
Michael Crosby
f01434283c Merge pull request #553 from crosbymichael/config
Add config file
2017-02-21 11:52:08 -08:00
Michael Crosby
a9950aedcf Add config file
This adds a config file for containerd configuration.  It is hard to
have structure data on cli flags and the config file should be used for
the majority of fields when configuring containerd.

There are still a few flags on the daemon that override config file
values but flags should take a back seat going forward and should be
kept at a minimum.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-02-21 11:22:08 -08:00
Michael Crosby
1d08c7bc5c Merge pull request #549 from fate-grand-order/typo
correct misspell in cmd/dist/fetch.go and events/transaction.go
2017-02-21 11:14:15 -08:00
Michael Crosby
559a3c1d3c Merge pull request #550 from hqhq/fix_interceptor_ctx
Fix interceptor handler parameter
2017-02-21 11:13:55 -08:00
Qiang Huang
f83652dbf1 Ensure to read all io in error cases
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2017-02-21 08:46:04 -08:00
Qiang Huang
7206427df4 Fix interceptor handler parameter
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2017-02-21 08:41:31 -08:00
fate-grand-order
3626ee7b77 correct misspell in cmd/dist/fetch.go and events/transaction.go
Signed-off-by: fate-grand-order <chenjg@harmonycloud.cn>
2017-02-21 20:24:04 +08:00
Akihiro Suda
5f6f04742c ctr: add --runtime-config
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-02-21 05:04:18 +00:00
Akihiro Suda
2836fbeed1 ctr: fix Sprintf
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-02-20 05:28:09 +00:00
Qiang Huang
d42dfcba23 Delete default log-level value
It's duplicate with --log-level, and since --log-level
takes advantage of --debug and it has default value,
--debug never works now.

Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2017-02-18 21:14:58 +08:00
Derek McGowan
6443891a7d Update log lines to use containerd log package
Removed unused requires root test function and updated
tar requires function to use lookup method.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-17 11:50:49 -08:00
Derek McGowan
f0a43e72cd Update layer apply to use containerd archive
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-17 11:50:49 -08:00
Michael Crosby
47ececd6b8 Change Container interface to include Info
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-02-16 09:59:40 -08:00
Michael Crosby
ab8586b7c5 Remove bundles from API
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-02-15 13:56:41 -08:00
Stephen J Day
a3120172b0
cmd/protoc-gen-gogoctrd: turn off GoString generation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-15 13:37:48 -08:00
Stephen J Day
83e7610194
cmd/ctrd-protobuild: create proper command for building protos
After trying to explain the complexities of developing with protobuf, I
have now created a command that correctly calculates the import paths
for each package and runs the protobuf command.

The Makefile has been updated accordingly, expect we now no longer use
`go generate`. A new target `protos` has been defined. We alias the two,
for the lazy. We leave `go generate` in place for cases where we will
actually use `go generate`.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-15 12:05:27 -08:00
Akihiro Suda
bf8abef70f api: introduce api/{types,services} for deduplication of API definition
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-02-14 02:42:13 +00:00
Michael Crosby
89f62d4fae Merge pull request #515 from fate-grand-order/branch
Use errors.New() directly to output the error message
2017-02-10 13:55:10 -08:00
Phil Estes
43e42b4222 Merge pull request #503 from jmzwcn/master
Allow specifying the log level
2017-02-10 12:35:14 -08:00
Daniel Zhang
ca52365780 Allow specifying the log level
Signed-off-by: Daniel Zhang <jmzwcn@gmail.com>
2017-02-10 17:29:06 +08:00
fate-grand-order
af86cd4d2f Use error.New () directly to output the error message
Signed-off-by: fate-grand-order <chenjg@harmonycloud.cn>
2017-02-10 14:31:49 +08:00
Kunal Kushwaha
a9ea9d80a3 ctr pprof support added
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2017-02-10 11:16:40 +09:00
Michael Crosby
42a17f9391 Merge pull request #501 from mlaventure/new-shim-continued
New shim continued
2017-02-07 15:52:08 -08:00
Phil Estes
eb2191c744 Merge pull request #493 from hqhq/specify_shim_socket
Remove duplicated close
2017-02-07 09:47:14 -08:00
Kenfe-Mickael Laventure
b4a299e61d supervisor/shim: add exec support
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-02-07 09:25:55 -08:00
Kunal Kushwaha
1191b209ae closing http connection
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2017-02-07 14:57:44 +09:00
Qiang Huang
81246555c9 Remove duplicated close
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2017-02-07 10:17:47 +08:00
Kenfe-Mickael Laventure
78d7e8b256 supervisor: implement monitoring
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-02-06 15:16:55 -08:00
Kenfe-Mickael Laventure
31f26fed18 Move to a single Event type
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-02-06 08:46:37 -08:00
Kenfe-Mickael Laventure
6f9eda1134 api/execution: remove ProcessID from rpc calls
Now that the shim handles all container's processes the system pid is
sufficient.

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-02-02 15:36:10 -08:00
Kenfe-Mickael Laventure
40b0b211b7 api/execution: add Container suffix to relevant rpc calls
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-02-02 14:40:23 -08:00
Michael Crosby
f187da9485 Port over supervisor to use grpc shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-02-01 16:01:53 -08:00
Kunal Kushwaha
6af4f5fee8 Exposing pprof and expvars interfaces on default containerd socket
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2017-02-01 11:05:55 +09:00
Ed King
89607a10db os.MkdirAll the containerd root dir
Signed-off-by: Ed King <ed@teddyking.co.uk>
2017-01-28 15:47:51 +00:00
Michael Crosby
9aab1cd89e Merge pull request #431 from hqhq/add_logrus_info
Add some logrus info for containerd
2017-01-27 11:50:32 -08:00
Stephen J Day
3e0238612b
dist: provide apply command to build rootfs
This changeset adds the simple apply command. It consumes a tar layer
and applies that layer to the specified directory. For the most part, it
is a direct call into Docker's `pkg/archive.ApplyLayer`.

The following demonstrates unpacking the wordpress rootfs into a local
directory `wordpress`:

```
$ ./dist fetch docker.io/library/wordpress 4.5 mediatype:application/vnd.docker.distribution.manifest.v2+json | \
    jq -r '.layers[] | "sudo ./dist apply ./wordpress < $(./dist path -n "+.digest+")"' | xargs -I{} -n1 sh -c "{}"
```

Note that you should have fetched the layers into the local content
store before running the above. Alternatively, you can just read the
manifest from the content store, rather than fetching it. We use fetch
above to avoid having to lookup the manifest digest for our demo.

This tool has a long way to go. We still need to incorporate
snapshotting, as well as the ability to calculate the `ChainID` under
subsequent unpacking. Once we have some tools to play around with
snapshotting, we'll be able to incorporate our `rootfs.ApplyLayer`
algorithm that will get us a lot closer to a production worthy system.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 11:00:29 -08:00
Stephen Day
594dca9e31 Merge pull request #472 from stevvooe/expanding-dist-tool
dist: expand functionality of the dist tool
2017-01-27 10:34:54 -08:00
Stephen J Day
f9cd9be61a
dist: expand functionality of the dist tool
With this change, we add the following commands to the dist tool:

- `ingest`: verify and accept content into storage
- `active`: display active ingest processes
- `list`: list content in storage
- `path`: provide a path to a blob by digest
- `delete`: remove a piece of content from storage

We demonstrate the utility with the following shell pipeline:

```
$ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | \
    jq -r '.layers[] | "./dist fetch docker.io/library/redis "+.digest + "| ./dist ingest --expected-digest "+.digest+" --expected-size "+(.size | tostring) +" docker.io/library/redis@"+.digest' | xargs -I{} -P10 -n1 sh -c "{}"
```

The above fetches a manifest, pipes it to jq, which assembles a shell
pipeline to ingest each layer into the content store. Because the
transactions are keyed by their digest, concurrent downloads and
downloads of repeated content are ignored. Each process is then executed
parallel using xargs.

Put shortly, this is a parallel layer download.

In a separate shell session, could monitor the active downloads with the
following:

```
$ watch -n0.2 ./dist active
```

For now, the content is downloaded into `.content` in the current
working directory. To watch the contents of this directory, you can use
the following:

```
$ watch -n0.2 tree .content
```

This will help to understand what is going on internally.

To get access to the layers, you can use the path command:

```
$./dist path sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa
sha256:010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa /home/sjd/go/src/github.com/docker/containerd/.content/blobs/sha256/010c454d55e53059beaba4044116ea4636f8dd8181e975d893931c7e7204fffa
```

When you are done, you can clear out the content with the classic xargs
pipeline:

```
$ ./dist list -q | xargs ./dist delete
```

Note that this is mostly a POC. Things like failed downloads and
abandoned download cleanup aren't quite handled. We'll probably make
adjustments around how content store transactions are handled to address
this.

From here, we'll build out full image pull and create tooling to get
runtime bundles from the fetched content.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 10:29:10 -08:00
Qiang Huang
7e4a7402f9 Add some logrus info for containerd
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2017-01-27 22:12:52 +08:00
Kenfe-Mickaël Laventure
24c2810899 Merge pull request #462 from crosbymichael/shim
shim: GRPC service
2017-01-26 16:11:45 -08:00
Michael Crosby
f70d485d72 Merge pull request #468 from jzwlqx/master
optimal help info for ctr cli.
2017-01-26 15:34:31 -08:00
Michael Crosby
f431bf4ad4 Add state rpc to shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 15:09:59 -08:00
Michael Crosby
b59bd59d8a Working tty and io support in shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
07c81ccac4 Add events api to shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
d619954a2b Move shim service into top lvl package
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
fe280d2df0 Fix logrus import in shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
e09b0b0c35 Add exec functionality to shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
d5d2e586cd Refactor shim terminal and io handling
This also finishes the service implementation of the shim behind GRPC

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
bf036b9d78 Add ctr shim command
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Michael Crosby
c08e0e610c Add grpc service to shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-26 11:31:17 -08:00
Kenfe-Mickaël Laventure
b5ee0b1bd4 Merge pull request #451 from AkihiroSuda/trivial-cleanup
*: clean up
2017-01-25 11:02:38 -08:00
Jizhong Jiang
f5615b0225 optimal help info for ctr cli.
Signed-off-by: Jizhong Jiang <jiangjizhong@gmail.com>
2017-01-25 14:16:13 +08:00
Stephen J Day
19eecaab12
cmd/dist: POC implementation of dist fetch
With this changeset we introduce several new things. The first is the
top-level dist command. This is a toolkit that implements various
distribution primitives, such as fetching, unpacking and ingesting.

The first component to this is a simple `fetch` command. It is a
low-level command that takes a "remote", identified by a `locator`, and
an object identifier. Keyed by the locator, this tool can identify a
remote implementation to fetch the content and write it back to standard
out. By allowing this to be the unit of pluggability in fetching
content, we can have quite a bit of flexibility in how we retrieve
images.

The current `fetch` implementation provides anonymous access to docker
hub images, through the namespace `docker.io`. As an example, one can
fetch the manifest for `redis` with the following command:

```
$ ./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json
```

Note that we have provided a mediatype "hint", nudging the fetch
implementation to grab the correct endpoint. We can hash the output of
that to fetch the same content by digest:

```
$ ./dist fetch docker.io/library/redis sha256:$(./dist fetch docker.io/library/redis latest mediatype:application/vnd.docker.distribution.manifest.v2+json | shasum -a256)
```

Note that the hint is now elided, since we have affixed the content to a
particular hash.

If you are not yet entertained, let's bring `jq` and `xargs` into the
mix for maximum fun. The following incantation fetches the same manifest
and downloads all layers into the convenience of `/dev/null`:

```
$ ./dist fetch docker.io/library/redis sha256:a027a470aa2b9b41cc2539847a97b8a14794ebd0a4c7c5d64e390df6bde56c73 | jq -r '.layers[] | .digest' | xargs -n1 -P10 ./dist fetch docker.io/library/redis > /dev/null
```

This is just the beginning. We should be able to centralize
configuration around fetch to implement a number of distribution
methodologies that have been challenging or impossible up to this point.
The `locator`, mentioned earlier, is a schemaless URL that provides a
host and path that can be used to resolve the remote. By dispatching on
this common identifier, we should be able to support almost any protocol
and discovery mechanism imaginable.

When this is more solidified, we can roll these up into higher-level
operations that can be orchestrated through the `dist` tool or via GRPC.

What a time to be alive!

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-23 13:27:07 -08:00
Akihiro Suda
7fed38881d rootfs: fix compilation error
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-01-23 08:54:57 +00:00
Akihiro Suda
727f0a536e vendor: sirupsen/logrus -> Sirupsen/logrus
github.com/docker/docker/pkg/archive requires Sirupsen/logrus.
So let's remove sirupsen/logrus at the moment.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-01-23 08:50:08 +00:00
Akihiro Suda
e9bfed4cf2 *: clean up
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-01-20 01:50:08 +00:00
Michael Crosby
983d817470 Merge pull request #449 from xulike666/fight-for-readability
Fix all typos I found in this repo.
2017-01-19 10:58:20 -08:00
Aaron.L.Xu
08bcbddb32 fix typo I found in this repo
Signed-off-by: Aaron.L.Xu <likexu@harmonycloud.cn>
2017-01-20 01:18:26 +08:00
Kenfe-Mickael Laventure
3f2d9d19bf execution: remove oci executor
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-01-19 07:57:02 -08:00
Akihiro Suda
8bd1b0d2e4 ctr: add commands: list and inspect
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-01-18 02:47:42 +00:00
Kenfe-Mickaël Laventure
abc1bf4dea Merge pull request #429 from datawolf/io-set-termios-onlcr-for-master
Set -ONLCR on created consoles
2017-01-17 14:07:50 -08:00
Michael Crosby
7d39f2d0e6 Merge pull request #430 from hqhq/fix_check_error
Fix go vet errors
2017-01-17 10:51:35 -08:00
Qiang Huang
9a5b17009c Fix io closer
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2017-01-17 19:04:20 +08:00
Qiang Huang
5d6fa7b886 Fix go vet errors
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2017-01-14 21:43:31 +08:00
Wang Long
7c473041a0 io: stop screwing with \n in console output
The default terminal setting for a new pty on Linux (unix98) has +ONLCR,
resulting in '\n' writes by a container process to be converted to
'\r\n' reads by the managing process. This is quite unexpected, To fix it, make
the terminal sane after opening it by setting -ONLCR.

this patch fix method comes from: eea28f480d
thanks @cyphar Aleksa Sarai <asarai@suse.de>

Signed-off-by: Wang Long <long.wanglong@huawei.com>
2017-01-14 07:00:50 +00:00
Kenfe-Mickael Laventure
a60487b782 ctr: set raw mode when using --tty
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-01-12 11:39:09 -08:00
Kenfe-Mickael Laventure
5b40adf9af execution: add shim runtime
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-01-12 10:45:33 -08:00
Kenfe-Mickael Laventure
3cb57fa2d6 Add containerd-shim binary
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-01-12 10:29:37 -08:00
Kenfe-Mickael Laventure
3234546ee6 ctr: take console flag in account when readying IOs
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-01-12 10:24:22 -08:00
Kenfe-Mickael Laventure
8e603d2e48 ctr: use full path for bundle in run
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-01-12 10:23:29 -08:00
Kenfe-Mickael Laventure
f9aac9d599 ctr: add delete command
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-01-12 10:15:27 -08:00
Michael Crosby
4c848c4654 Merge pull request #382 from runcom/not-only-runc
cmd/containerd: extend to other OCI runtimes
2017-01-10 09:26:21 -08:00
Gábor Lipták
102fb10fde Correct vet warnings
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2016-12-20 17:10:01 -05:00
Kenfe-Mickael Laventure
0fdd2469f6 execution: "restore" container on service creation
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-16 11:40:47 -08:00
Antonio Murdaca
98b39002ba
cmd/containerd: extend to other OCI runtimes
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-12-16 18:35:15 +01:00
Kenfe-Mickael Laventure
13399c1330 ctr: add events command
This simply print all events generated by containerd

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-16 08:49:09 -08:00
Kenfe-Mickael Laventure
aa5ff88bbc Integrate NATS with event subsystem
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-12 14:54:47 -08:00
Kenfe-Mickael Laventure
2ef399b315 Add event support to execution subsystem
The implementation relies on nats.io

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-12 09:21:29 -08:00
Kenfe-Mickael Laventure
2bee4ac5cd Only use IDs for execution RPC requests
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-09 11:32:45 -08:00
Kenfe-Mickael Laventure
0aad42f5cf Add exec and terminal support
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-09 10:00:34 -08:00
Kenfe-Mickael Laventure
5a86eae247 Merge execution and container service
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-07 19:44:22 -08:00
Kenfe-Mickael Laventure
ac3cc32dbc Complete basic support for ctr run
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-07 19:44:22 -08:00
Michael Crosby
df9c09303c Fix ctr build
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2016-12-05 16:36:15 -08:00
Michael Crosby
723a72bdf8 Fix execution build
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2016-12-05 16:18:05 -08:00
Michael Crosby
bde30191f4 Move service to execution package
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2016-12-05 14:33:31 -08:00
Kenfe-Mickael Laventure
c857213b4c move work on execution service
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2016-12-05 14:15:03 -08:00
Michael Crosby
6641888667 Start work on execution server implementation
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2016-12-02 11:33:58 -08:00
Stephen J Day
796b609741
cmd: move cli files to cmd package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-01 12:07:45 -08:00
Stephen J Day
ab088af2a6
cmd, Makefile: add initial build structure
This adds Makefile and cmd/protoc-gen-gogoctrd for generating protobufs.
More adjustments are required and the default target has been stubbed
out for now.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-01 12:04:23 -08:00