containerd/reports/2017-03-17.md

Development Report for Mar 17, 2017

Testing Plan

Thanks to @gianarb for starting the discussion around a test and CI plan for containerd. We want to make sure that users of containerd can feel secure depending on containerd; having a solid test plan is a must.

Testing a project like containerd is always a challenge because of the systems that it needs to support. ARM, Windows, Linux, and Power (as well as many more variations) are all examples of platforms that we support today and will need a CI.

You can view the issue and contribute to the testing plan here.

Windows Runtime

Work has started on porting over the Windows execution code. There is still a lot of testing to do after the port but a PR should be coming soon.

Metrics

We started the work to get container level metrics exported over prometheus. You can see the initial output here:

containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Async"} 958464
containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Read"} 958464
containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Sync"} 0
containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Total"} 958464
containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Write"} 0
containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Async"} 17
containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Read"} 17
containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Sync"} 0
containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Total"} 17
containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Write"} 0
containerd_container_cpu_kernel_nanoseconds{id="test"} 1e+07
containerd_container_cpu_throttle_periods_total{id="test"} 0
containerd_container_cpu_throttled_periods_total{id="test"} 0
containerd_container_cpu_throttled_time_nanoseconds{id="test"} 0
containerd_container_cpu_total_nanoseconds{id="test"} 2.1428791e+07
containerd_container_cpu_user_nanoseconds{id="test"} 0
containerd_container_hugetlb_failcnt_total{id="test",page="1GB"} 0
containerd_container_hugetlb_failcnt_total{id="test",page="2MB"} 0
containerd_container_hugetlb_max_bytes{id="test",page="1GB"} 0
containerd_container_hugetlb_max_bytes{id="test",page="2MB"} 0
containerd_container_hugetlb_usage_bytes{id="test",page="1GB"} 0
containerd_container_hugetlb_usage_bytes{id="test",page="2MB"} 0
containerd_container_memory_active_anon_bytes{id="test"} 0
containerd_container_memory_active_file_bytes{id="test"} 659456
containerd_container_memory_cache_bytes{id="test"} 925696
containerd_container_memory_dirty_bytes{id="test"} 0
containerd_container_memory_hierarchical_memory_limit_bytes{id="test"} 9.223372036854772e+18
containerd_container_memory_hierarchical_memsw_limit_bytes{id="test"} 9.223372036854772e+18
containerd_container_memory_inactive_anon_bytes{id="test"} 73728
containerd_container_memory_inactive_file_bytes{id="test"} 266240
containerd_container_memory_kernel_failcnt_total{id="test"} 0
containerd_container_memory_kernel_limit_bytes{id="test"} 9.223372036854772e+18
containerd_container_memory_kernel_max_bytes{id="test"} 0
containerd_container_memory_kernel_usage_bytes{id="test"} 0
containerd_container_memory_kerneltcp_failcnt_total{id="test"} 0
containerd_container_memory_kerneltcp_limit_bytes{id="test"} 9.223372036854772e+18
containerd_container_memory_kerneltcp_max_bytes{id="test"} 0
containerd_container_memory_kerneltcp_usage_bytes{id="test"} 0
containerd_container_memory_mapped_file_bytes{id="test"} 577536
containerd_container_memory_oom_total{id="test"} 0
containerd_container_memory_pgfault_bytes{id="test"} 770
containerd_container_memory_pgmajfault_bytes{id="test"} 6
containerd_container_memory_pgpgin_bytes{id="test"} 651
containerd_container_memory_pgpgout_bytes{id="test"} 407
containerd_container_memory_rss_bytes{id="test"} 73728
containerd_container_memory_rss_huge_bytes{id="test"} 0
containerd_container_memory_swap_failcnt_total{id="test"} 0
containerd_container_memory_swap_limit_bytes{id="test"} 9.223372036854772e+18
containerd_container_memory_swap_max_bytes{id="test"} 1.527808e+06
containerd_container_memory_swap_usage_bytes{id="test"} 999424
containerd_container_memory_total_active_anon_bytes{id="test"} 0
containerd_container_memory_total_active_file_bytes{id="test"} 659456
containerd_container_memory_total_cache_bytes{id="test"} 925696
containerd_container_memory_total_dirty_bytes{id="test"} 0
containerd_container_memory_total_inactive_anon_bytes{id="test"} 73728
containerd_container_memory_total_inactive_file_bytes{id="test"} 266240
containerd_container_memory_total_mapped_file_bytes{id="test"} 577536
containerd_container_memory_total_pgfault_bytes{id="test"} 770
containerd_container_memory_total_pgmajfault_bytes{id="test"} 6
containerd_container_memory_total_pgpgin_bytes{id="test"} 651
containerd_container_memory_total_pgpgout_bytes{id="test"} 407
containerd_container_memory_total_rss_bytes{id="test"} 73728
containerd_container_memory_total_rss_huge_bytes{id="test"} 0
containerd_container_memory_total_unevictable_bytes{id="test"} 0
containerd_container_memory_total_writeback_bytes{id="test"} 0
containerd_container_memory_unevictable_bytes{id="test"} 0
containerd_container_memory_usage_failcnt_total{id="test"} 0
containerd_container_memory_usage_limit_bytes{id="test"} 9.223372036854772e+18
containerd_container_memory_usage_max_bytes{id="test"} 1.527808e+06
containerd_container_memory_usage_usage_bytes{id="test"} 999424
containerd_container_memory_writeback_bytes{id="test"} 0
containerd_container_per_cpu_nanoseconds{cpu="0",id="test"} 7.530139e+06
containerd_container_per_cpu_nanoseconds{cpu="1",id="test"} 4.586408e+06
containerd_container_per_cpu_nanoseconds{cpu="2",id="test"} 5.076059e+06
containerd_container_per_cpu_nanoseconds{cpu="3",id="test"} 4.236185e+06
containerd_container_pids_current{id="test"} 1
containerd_container_pids_limit{id="test"} 0

The id label will be the id of the container so users can filter on the metrics for only the containers that they care about.

The frequency of metric collection is configurable via the prometheus scrape time. Every time the /metrics API is hit, that is when container metrics are collected. There is no internal timer in containerd, you only pay the cost of collecting metrics when you are asking for them. If you never ask for metrics the collection never happens.

There should be a PR up soon so that we can discuss the metrics and label names.