cri-o - OCI-based implementation of Kubernetes Container Runtime Interface

Status: Release Candidate 3

What is the scope of this project?

cri-o is meant to provide an integration path between OCI conformant runtimes and the kubelet. Specifically, it implements the Kubelet Container Runtime Interface (CRI) using OCI conformant runtimes. The scope of cri-o is tied to the scope of the CRI.

At a high level, we expect the scope of cri-o to be restricted to the following functionalities:

• Support multiple image formats including the existing Docker image format
• Support for multiple means to download images including trust & image verification
• Container image management (managing image layers, overlay filesystems, etc)
• Container process lifecycle management
• Monitoring and logging required to satisfy the CRI
• Resource isolation as required by the CRI

What is not in scope for this project?

• Building, signing and pushing images to various image storages
• A CLI utility for interacting with cri-o. Any CLIs built as part of this project are only meant for testing this project and there will be no guarantees on the backward compatibility with it.

This is an implementation of the Kubernetes Container Runtime Interface (CRI) that will allow Kubernetes to directly launch and manage Open Container Initiative (OCI) containers.

The plan is to use OCI projects and best of breed libraries for different aspects:

• Runtime: runc (or any OCI runtime-spec implementation) and oci runtime tools
• Images: Image management using containers/image
• Storage: Storage and management of image layers using containers/storage
• Networking: Networking support through use of CNI

It is currently in active development in the Kubernetes community through the design proposal. Questions and issues should be raised in the Kubernetes sig-node Slack channel.

Commands

Command	Description	Demo
crio(8)	OCI Kubernetes Container Runtime daemon
kpod(1)	Simple management tool for pods and images
kpod-attach(1)	Instead of providing a kpod attach command, the man page kpod-attach describes how to use the kpod logs and kpod exec commands to achieve the same goals as kpod attach.
kpod-cp(1)	Instead of providing a kpod cp command, the man page kpod-cp describes how to use the kpod mount command to have even more flexibility and functionality.
kpod-diff(1)	Inspect changes on a container or image's filesystem
kpod-export(1)	Export container's filesystem contents as a tar archive
kpod-history(1)	Shows the history of an image
kpod-images(1)	List images in local storage
kpod-info(1)	Display system information
kpod-inspect(1)	Display the configuration of a container or image
kpod-kill(1)	Kill the main process in one or more running containers
kpod-load(1)	Load an image from docker archive or oci
kpod-login(1)	Login to a container registry
kpod-logout(1)	Logout of a container registry
kpod-logs(1)	Display the logs of a container
kpod-mount(1)	Mount a working container's root filesystem
kpod-pause(1)	Pause one or more running containers
kpod-ps(1)	Prints out information about containers
kpod-pull(1)	Pull an image from a registry
kpod-push(1)	Push an image to a specified destination
kpod-rename(1)	Rename a container
kpod-rm(1)	Removes one or more containers
kpod-rmi(1)	Removes one or more images
kpod-save(1)	Saves an image to an archive
kpod-stats(1)	Display a live stream of one or more containers' resource usage statistics
kpod-stop(1)	Stops one or more running containers
kpod-tag(1)	Add an additional name to a local image
kpod-umount(1)	Unmount a working container's root filesystem
kpod-unpause(1)	Unpause one or more running containers
kpod-version(1)	Display the version information
kpod-wait(1)	Wait on one or more containers to stop and print their exit codes

Configuration

File	Description
crio.conf(5)	CRI-O Configuation file

OCI Hooks Support

CRI-O configures OCI Hooks to run when launching a container

cri-o Usage Transfer

Useful information for ops and dev transfer as it relates to infrastructure that utilizes cri-o

Communication

For async communication and long running discussions please use issues and pull requests on the github repo. This will be the best place to discuss design and implementation.

For sync communication we have an IRC channel #cri-o, on chat.freenode.net, that everyone is welcome to join and chat about development.

Getting started

Prerequisites

Latest version of runc is expected to be installed on the system. It is picked up as the default runtime by crio.

Build and Run Dependencies

Required

Fedora, CentOS, RHEL, and related distributions:

yum install -y \
  btrfs-progs-devel \
  device-mapper-devel \
  git \
  glib2-devel \
  glibc-devel \
  glibc-static \
  go \
  gpgme-devel \
  libassuan-devel \
  libgpg-error-devel \
  libseccomp-devel \
  libselinux-devel \
  ostree-devel \
  pkgconfig \
  runc \
  skopeo-containers

Debian, Ubuntu, and related distributions:

apt-get install -y \
  btrfs-tools \
  git \
  golang-go \
  libassuan-dev \
  libdevmapper-dev \
  libglib2.0-dev \
  libc6-dev \
  libgpgme11-dev \
  libgpg-error-dev \
  libseccomp-dev \
  libselinux1-dev \
  pkg-config \
  runc \
  skopeo-containers

Debian, Ubuntu, and related distributions will also need a copy of the development libraries for ostree, either in the form of the libostree-dev package from the flatpak PPA, or built from source (more on that here).

If using an older release or a long-term support release, be careful to double-check that the version of runc is new enough (running runc --version should produce spec: 1.0.0), or else build your own.

NOTE

Be careful to double-check that the version of golang is new enough, version 1.8.x or higher is required. If needed, golang kits are avaliable at https://golang.org/dl/

Optional

Fedora, CentOS, RHEL, and related distributions:

(no optional packages)

Debian, Ubuntu, and related distributions:

apt-get install -y \
  libapparmor-dev

Get Source Code

As with other Go projects, cri-o must be cloned into a directory structure like:

GOPATH
└── src
    └── github.com
        └── kubernetes-incubator
            └── cri-o

First, configure a GOPATH (if you are using go1.8 or later, this defaults to ~/go).

export GOPATH=~/go
mkdir -p $GOPATH

Next, clone the source code using:

mkdir -p $GOPATH/src/github.com/kubernetes-incubator
cd $_ # or cd $GOPATH/src/github.com/kubernetes-incubator
git clone https://github.com/kubernetes-incubator/cri-o # or your fork
cd cri-o

Build

make install.tools
make
sudo make install

Otherwise, if you do not want to build cri-o with seccomp support you can add BUILDTAGS="" when running make.

make BUILDTAGS=""
sudo make install

Build Tags

cri-o supports optional build tags for compiling support of various features. To add build tags to the make option the BUILDTAGS variable must be set.

make BUILDTAGS='seccomp apparmor'

Build Tag	Feature	Dependency
seccomp	syscall filtering	libseccomp
selinux	selinux process and mount labeling	libselinux
apparmor	apparmor profile support	libapparmor

Running pods and containers

Follow this tutorial to get started with CRI-O.

Setup CNI networking

A proper description of setting up CNI networking is given in the contrib/cni README. But the gist is that you need to have some basic network configurations enabled and CNI plugins installed on your system.

Running with kubernetes

You can run a local version of kubernetes with cri-o using local-up-cluster.sh:

1. Clone the kubernetes repository
2. Start the cri-o daemon (crio)
3. From the kubernetes project directory, run:

CGROUP_DRIVER=systemd \
CONTAINER_RUNTIME=remote \
CONTAINER_RUNTIME_ENDPOINT='/var/run/crio.sock  --runtime-request-timeout=15m' \
./hack/local-up-cluster.sh

To run a full cluster, see the instructions.

Current Roadmap

1. Basic pod/container lifecycle, basic image pull (done)
2. Support for tty handling and state management (done)
3. Basic integration with kubelet once client side changes are ready (done)
4. Support for log management, networking integration using CNI, pluggable image/storage management (done)
5. Support for exec/attach (done)
6. Target fully automated kubernetes testing without failures e2e status
7. Release 1.0
8. Track upstream k8s releases