mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-08-22 08:49:45 +00:00
![Linus Torvalds](/assets/img/avatar_default.png)
This KUnit update for Linux 5.14-rc1 consists of fixes and features: -- add support for skipped tests -- introduce kunit_kmalloc_array/kunit_kcalloc() helpers -- add gnu_printf specifiers -- add kunit_shutdown -- add unit test for filtering suites by names -- convert lib/test_list_sort.c to use KUnit -- code organization moving default config to tools/testing/kunit -- refactor of internal parser input handling -- cleanups and updates to documentation -- code cleanup related to casts -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEPZKym/RZuOCGeA/kCwJExA0NQxwFAmDfNHgACgkQCwJExA0N QxzU5g/+ORbcbE5jhumo82xYUCGTScgIqPllNO000Vk9xydMqwx6tpY5puV0ig2z 5X7pxKEQTnype58yzY5mS1p336ryIx3q+AwEownxTW3YurXo2naQ59ZPjEvlAV+E h+DMYLzEjrxzcyJATw02uC9YLUZ3w6FPJfiXViIv93YYrtcnM0u6JpwG0yfBlI4v 4KKB2Xu4K7T90C9/ADFYFKX3mjXQl5fQwvIdtA7wS90Cgq52LKp2mvg1XEiZE5d+ 0dkTZ4Zo8TxxHt665o7vfnUjQQNmh45iGlW65wONxfAPb8BPoneGjVKWQTnN0Hor W+93ZPbMuFMSSKJuoHY9U7sP5VySKvaiIYaGdi6prnZZu0zUabKnLZ6FOy7kEdfs v09ulCBTVLslixVgNcp/kD9T+G/SXwCF5YBMAiMDQ0GNfUqlFtBkEA3gd44KwMI0 KwCcOgUSiaCkqyzOz/VeQsu/nhA5jdMO0KjiAs7Z3e7r7O/qKFs/ll7hZgDNCWSC q8eIrcBkSL0EGgXR1iZ4AtGm8op6KKd4ACBM8NdtTyoGFl1npZOgZnHoIsy35G9K 9mhc7eXSoaDGqy9dONL1Tc8Neg7qLTXQNp2radqsnAAgNPUrJuC7+8YC+DdIsjBH W7OyMjpfbwPws5rP4CS+JdwL+nQprKXZvFIhWGYhkDK44MbOngw= =5QAv -----END PGP SIGNATURE----- Merge tag 'linux-kselftest-kunit-fixes-5.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest Pull KUnit update from Shuah Khan: "Fixes and features: - add support for skipped tests - introduce kunit_kmalloc_array/kunit_kcalloc() helpers - add gnu_printf specifiers - add kunit_shutdown - add unit test for filtering suites by names - convert lib/test_list_sort.c to use KUnit - code organization moving default config to tools/testing/kunit - refactor of internal parser input handling - cleanups and updates to documentation - code cleanup related to casts" * tag 'linux-kselftest-kunit-fixes-5.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest: (29 commits) kunit: add unit test for filtering suites by names kasan: test: make use of kunit_skip() kunit: test: Add example tests which are always skipped kunit: tool: Support skipped tests in kunit_tool kunit: Support skipped tests thunderbolt: test: Reinstate a few casts of bitfields kunit: tool: internal refactor of parser input handling lib/test: convert lib/test_list_sort.c to use KUnit kunit: introduce kunit_kmalloc_array/kunit_kcalloc() helpers kunit: Remove the unused all_tests.config kunit: Move default config from arch/um -> tools/testing/kunit kunit: arch/um/configs: Enable KUNIT_ALL_TESTS by default kunit: Add gnu_printf specifiers lib/cmdline_kunit: Remove a cast which are no-longer required kernel/sysctl-test: Remove some casts which are no-longer required thunderbolt: test: Remove some casts which are no longer required mmc: sdhci-of-aspeed: Remove some unnecessary casts from KUnit tests iio: Remove a cast in iio-test-format which is no longer required device property: Remove some casts in property-entry-test Documentation: kunit: Clean up some string casts in examples ...
774 lines
26 KiB
ReStructuredText
774 lines
26 KiB
ReStructuredText
.. SPDX-License-Identifier: GPL-2.0
|
|
|
|
===========
|
|
Using KUnit
|
|
===========
|
|
|
|
The purpose of this document is to describe what KUnit is, how it works, how it
|
|
is intended to be used, and all the concepts and terminology that are needed to
|
|
understand it. This guide assumes a working knowledge of the Linux kernel and
|
|
some basic knowledge of testing.
|
|
|
|
For a high level introduction to KUnit, including setting up KUnit for your
|
|
project, see Documentation/dev-tools/kunit/start.rst.
|
|
|
|
Organization of this document
|
|
=============================
|
|
|
|
This document is organized into two main sections: Testing and Common Patterns.
|
|
The first covers what unit tests are and how to use KUnit to write them. The
|
|
second covers common testing patterns, e.g. how to isolate code and make it
|
|
possible to unit test code that was otherwise un-unit-testable.
|
|
|
|
Testing
|
|
=======
|
|
|
|
What is KUnit?
|
|
--------------
|
|
|
|
"K" is short for "kernel" so "KUnit" is the "(Linux) Kernel Unit Testing
|
|
Framework." KUnit is intended first and foremost for writing unit tests; it is
|
|
general enough that it can be used to write integration tests; however, this is
|
|
a secondary goal. KUnit has no ambition of being the only testing framework for
|
|
the kernel; for example, it does not intend to be an end-to-end testing
|
|
framework.
|
|
|
|
What is Unit Testing?
|
|
---------------------
|
|
|
|
A `unit test <https://martinfowler.com/bliki/UnitTest.html>`_ is a test that
|
|
tests code at the smallest possible scope, a *unit* of code. In the C
|
|
programming language that's a function.
|
|
|
|
Unit tests should be written for all the publicly exposed functions in a
|
|
compilation unit; so that is all the functions that are exported in either a
|
|
*class* (defined below) or all functions which are **not** static.
|
|
|
|
Writing Tests
|
|
-------------
|
|
|
|
Test Cases
|
|
~~~~~~~~~~
|
|
|
|
The fundamental unit in KUnit is the test case. A test case is a function with
|
|
the signature ``void (*)(struct kunit *test)``. It calls a function to be tested
|
|
and then sets *expectations* for what should happen. For example:
|
|
|
|
.. code-block:: c
|
|
|
|
void example_test_success(struct kunit *test)
|
|
{
|
|
}
|
|
|
|
void example_test_failure(struct kunit *test)
|
|
{
|
|
KUNIT_FAIL(test, "This test never passes.");
|
|
}
|
|
|
|
In the above example ``example_test_success`` always passes because it does
|
|
nothing; no expectations are set, so all expectations pass. On the other hand
|
|
``example_test_failure`` always fails because it calls ``KUNIT_FAIL``, which is
|
|
a special expectation that logs a message and causes the test case to fail.
|
|
|
|
Expectations
|
|
~~~~~~~~~~~~
|
|
An *expectation* is a way to specify that you expect a piece of code to do
|
|
something in a test. An expectation is called like a function. A test is made
|
|
by setting expectations about the behavior of a piece of code under test; when
|
|
one or more of the expectations fail, the test case fails and information about
|
|
the failure is logged. For example:
|
|
|
|
.. code-block:: c
|
|
|
|
void add_test_basic(struct kunit *test)
|
|
{
|
|
KUNIT_EXPECT_EQ(test, 1, add(1, 0));
|
|
KUNIT_EXPECT_EQ(test, 2, add(1, 1));
|
|
}
|
|
|
|
In the above example ``add_test_basic`` makes a number of assertions about the
|
|
behavior of a function called ``add``; the first parameter is always of type
|
|
``struct kunit *``, which contains information about the current test context;
|
|
the second parameter, in this case, is what the value is expected to be; the
|
|
last value is what the value actually is. If ``add`` passes all of these
|
|
expectations, the test case, ``add_test_basic`` will pass; if any one of these
|
|
expectations fails, the test case will fail.
|
|
|
|
It is important to understand that a test case *fails* when any expectation is
|
|
violated; however, the test will continue running, potentially trying other
|
|
expectations until the test case ends or is otherwise terminated. This is as
|
|
opposed to *assertions* which are discussed later.
|
|
|
|
To learn about more expectations supported by KUnit, see
|
|
Documentation/dev-tools/kunit/api/test.rst.
|
|
|
|
.. note::
|
|
A single test case should be pretty short, pretty easy to understand,
|
|
focused on a single behavior.
|
|
|
|
For example, if we wanted to properly test the add function above, we would
|
|
create additional tests cases which would each test a different property that an
|
|
add function should have like this:
|
|
|
|
.. code-block:: c
|
|
|
|
void add_test_basic(struct kunit *test)
|
|
{
|
|
KUNIT_EXPECT_EQ(test, 1, add(1, 0));
|
|
KUNIT_EXPECT_EQ(test, 2, add(1, 1));
|
|
}
|
|
|
|
void add_test_negative(struct kunit *test)
|
|
{
|
|
KUNIT_EXPECT_EQ(test, 0, add(-1, 1));
|
|
}
|
|
|
|
void add_test_max(struct kunit *test)
|
|
{
|
|
KUNIT_EXPECT_EQ(test, INT_MAX, add(0, INT_MAX));
|
|
KUNIT_EXPECT_EQ(test, -1, add(INT_MAX, INT_MIN));
|
|
}
|
|
|
|
void add_test_overflow(struct kunit *test)
|
|
{
|
|
KUNIT_EXPECT_EQ(test, INT_MIN, add(INT_MAX, 1));
|
|
}
|
|
|
|
Notice how it is immediately obvious what all the properties that we are testing
|
|
for are.
|
|
|
|
Assertions
|
|
~~~~~~~~~~
|
|
|
|
KUnit also has the concept of an *assertion*. An assertion is just like an
|
|
expectation except the assertion immediately terminates the test case if it is
|
|
not satisfied.
|
|
|
|
For example:
|
|
|
|
.. code-block:: c
|
|
|
|
static void mock_test_do_expect_default_return(struct kunit *test)
|
|
{
|
|
struct mock_test_context *ctx = test->priv;
|
|
struct mock *mock = ctx->mock;
|
|
int param0 = 5, param1 = -5;
|
|
const char *two_param_types[] = {"int", "int"};
|
|
const void *two_params[] = {¶m0, ¶m1};
|
|
const void *ret;
|
|
|
|
ret = mock->do_expect(mock,
|
|
"test_printk", test_printk,
|
|
two_param_types, two_params,
|
|
ARRAY_SIZE(two_params));
|
|
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ret);
|
|
KUNIT_EXPECT_EQ(test, -4, *((int *) ret));
|
|
}
|
|
|
|
In this example, the method under test should return a pointer to a value, so
|
|
if the pointer returned by the method is null or an errno, we don't want to
|
|
bother continuing the test since the following expectation could crash the test
|
|
case. `ASSERT_NOT_ERR_OR_NULL(...)` allows us to bail out of the test case if
|
|
the appropriate conditions have not been satisfied to complete the test.
|
|
|
|
Test Suites
|
|
~~~~~~~~~~~
|
|
|
|
Now obviously one unit test isn't very helpful; the power comes from having
|
|
many test cases covering all of a unit's behaviors. Consequently it is common
|
|
to have many *similar* tests; in order to reduce duplication in these closely
|
|
related tests most unit testing frameworks - including KUnit - provide the
|
|
concept of a *test suite*. A *test suite* is just a collection of test cases
|
|
for a unit of code with a set up function that gets invoked before every test
|
|
case and then a tear down function that gets invoked after every test case
|
|
completes.
|
|
|
|
Example:
|
|
|
|
.. code-block:: c
|
|
|
|
static struct kunit_case example_test_cases[] = {
|
|
KUNIT_CASE(example_test_foo),
|
|
KUNIT_CASE(example_test_bar),
|
|
KUNIT_CASE(example_test_baz),
|
|
{}
|
|
};
|
|
|
|
static struct kunit_suite example_test_suite = {
|
|
.name = "example",
|
|
.init = example_test_init,
|
|
.exit = example_test_exit,
|
|
.test_cases = example_test_cases,
|
|
};
|
|
kunit_test_suite(example_test_suite);
|
|
|
|
In the above example the test suite, ``example_test_suite``, would run the test
|
|
cases ``example_test_foo``, ``example_test_bar``, and ``example_test_baz``;
|
|
each would have ``example_test_init`` called immediately before it and would
|
|
have ``example_test_exit`` called immediately after it.
|
|
``kunit_test_suite(example_test_suite)`` registers the test suite with the
|
|
KUnit test framework.
|
|
|
|
.. note::
|
|
A test case will only be run if it is associated with a test suite.
|
|
|
|
``kunit_test_suite(...)`` is a macro which tells the linker to put the specified
|
|
test suite in a special linker section so that it can be run by KUnit either
|
|
after late_init, or when the test module is loaded (depending on whether the
|
|
test was built in or not).
|
|
|
|
For more information on these types of things see the
|
|
Documentation/dev-tools/kunit/api/test.rst.
|
|
|
|
Common Patterns
|
|
===============
|
|
|
|
Isolating Behavior
|
|
------------------
|
|
|
|
The most important aspect of unit testing that other forms of testing do not
|
|
provide is the ability to limit the amount of code under test to a single unit.
|
|
In practice, this is only possible by being able to control what code gets run
|
|
when the unit under test calls a function and this is usually accomplished
|
|
through some sort of indirection where a function is exposed as part of an API
|
|
such that the definition of that function can be changed without affecting the
|
|
rest of the code base. In the kernel this primarily comes from two constructs,
|
|
classes, structs that contain function pointers that are provided by the
|
|
implementer, and architecture-specific functions which have definitions selected
|
|
at compile time.
|
|
|
|
Classes
|
|
~~~~~~~
|
|
|
|
Classes are not a construct that is built into the C programming language;
|
|
however, it is an easily derived concept. Accordingly, pretty much every project
|
|
that does not use a standardized object oriented library (like GNOME's GObject)
|
|
has their own slightly different way of doing object oriented programming; the
|
|
Linux kernel is no exception.
|
|
|
|
The central concept in kernel object oriented programming is the class. In the
|
|
kernel, a *class* is a struct that contains function pointers. This creates a
|
|
contract between *implementers* and *users* since it forces them to use the
|
|
same function signature without having to call the function directly. In order
|
|
for it to truly be a class, the function pointers must specify that a pointer
|
|
to the class, known as a *class handle*, be one of the parameters; this makes
|
|
it possible for the member functions (also known as *methods*) to have access
|
|
to member variables (more commonly known as *fields*) allowing the same
|
|
implementation to have multiple *instances*.
|
|
|
|
Typically a class can be *overridden* by *child classes* by embedding the
|
|
*parent class* in the child class. Then when a method provided by the child
|
|
class is called, the child implementation knows that the pointer passed to it is
|
|
of a parent contained within the child; because of this, the child can compute
|
|
the pointer to itself because the pointer to the parent is always a fixed offset
|
|
from the pointer to the child; this offset is the offset of the parent contained
|
|
in the child struct. For example:
|
|
|
|
.. code-block:: c
|
|
|
|
struct shape {
|
|
int (*area)(struct shape *this);
|
|
};
|
|
|
|
struct rectangle {
|
|
struct shape parent;
|
|
int length;
|
|
int width;
|
|
};
|
|
|
|
int rectangle_area(struct shape *this)
|
|
{
|
|
struct rectangle *self = container_of(this, struct shape, parent);
|
|
|
|
return self->length * self->width;
|
|
};
|
|
|
|
void rectangle_new(struct rectangle *self, int length, int width)
|
|
{
|
|
self->parent.area = rectangle_area;
|
|
self->length = length;
|
|
self->width = width;
|
|
}
|
|
|
|
In this example (as in most kernel code) the operation of computing the pointer
|
|
to the child from the pointer to the parent is done by ``container_of``.
|
|
|
|
Faking Classes
|
|
~~~~~~~~~~~~~~
|
|
|
|
In order to unit test a piece of code that calls a method in a class, the
|
|
behavior of the method must be controllable, otherwise the test ceases to be a
|
|
unit test and becomes an integration test.
|
|
|
|
A fake just provides an implementation of a piece of code that is different than
|
|
what runs in a production instance, but behaves identically from the standpoint
|
|
of the callers; this is usually done to replace a dependency that is hard to
|
|
deal with, or is slow.
|
|
|
|
A good example for this might be implementing a fake EEPROM that just stores the
|
|
"contents" in an internal buffer. For example, let's assume we have a class that
|
|
represents an EEPROM:
|
|
|
|
.. code-block:: c
|
|
|
|
struct eeprom {
|
|
ssize_t (*read)(struct eeprom *this, size_t offset, char *buffer, size_t count);
|
|
ssize_t (*write)(struct eeprom *this, size_t offset, const char *buffer, size_t count);
|
|
};
|
|
|
|
And we want to test some code that buffers writes to the EEPROM:
|
|
|
|
.. code-block:: c
|
|
|
|
struct eeprom_buffer {
|
|
ssize_t (*write)(struct eeprom_buffer *this, const char *buffer, size_t count);
|
|
int flush(struct eeprom_buffer *this);
|
|
size_t flush_count; /* Flushes when buffer exceeds flush_count. */
|
|
};
|
|
|
|
struct eeprom_buffer *new_eeprom_buffer(struct eeprom *eeprom);
|
|
void destroy_eeprom_buffer(struct eeprom *eeprom);
|
|
|
|
We can easily test this code by *faking out* the underlying EEPROM:
|
|
|
|
.. code-block:: c
|
|
|
|
struct fake_eeprom {
|
|
struct eeprom parent;
|
|
char contents[FAKE_EEPROM_CONTENTS_SIZE];
|
|
};
|
|
|
|
ssize_t fake_eeprom_read(struct eeprom *parent, size_t offset, char *buffer, size_t count)
|
|
{
|
|
struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent);
|
|
|
|
count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset);
|
|
memcpy(buffer, this->contents + offset, count);
|
|
|
|
return count;
|
|
}
|
|
|
|
ssize_t fake_eeprom_write(struct eeprom *parent, size_t offset, const char *buffer, size_t count)
|
|
{
|
|
struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent);
|
|
|
|
count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset);
|
|
memcpy(this->contents + offset, buffer, count);
|
|
|
|
return count;
|
|
}
|
|
|
|
void fake_eeprom_init(struct fake_eeprom *this)
|
|
{
|
|
this->parent.read = fake_eeprom_read;
|
|
this->parent.write = fake_eeprom_write;
|
|
memset(this->contents, 0, FAKE_EEPROM_CONTENTS_SIZE);
|
|
}
|
|
|
|
We can now use it to test ``struct eeprom_buffer``:
|
|
|
|
.. code-block:: c
|
|
|
|
struct eeprom_buffer_test {
|
|
struct fake_eeprom *fake_eeprom;
|
|
struct eeprom_buffer *eeprom_buffer;
|
|
};
|
|
|
|
static void eeprom_buffer_test_does_not_write_until_flush(struct kunit *test)
|
|
{
|
|
struct eeprom_buffer_test *ctx = test->priv;
|
|
struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer;
|
|
struct fake_eeprom *fake_eeprom = ctx->fake_eeprom;
|
|
char buffer[] = {0xff};
|
|
|
|
eeprom_buffer->flush_count = SIZE_MAX;
|
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0);
|
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0);
|
|
|
|
eeprom_buffer->flush(eeprom_buffer);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff);
|
|
}
|
|
|
|
static void eeprom_buffer_test_flushes_after_flush_count_met(struct kunit *test)
|
|
{
|
|
struct eeprom_buffer_test *ctx = test->priv;
|
|
struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer;
|
|
struct fake_eeprom *fake_eeprom = ctx->fake_eeprom;
|
|
char buffer[] = {0xff};
|
|
|
|
eeprom_buffer->flush_count = 2;
|
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0);
|
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff);
|
|
}
|
|
|
|
static void eeprom_buffer_test_flushes_increments_of_flush_count(struct kunit *test)
|
|
{
|
|
struct eeprom_buffer_test *ctx = test->priv;
|
|
struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer;
|
|
struct fake_eeprom *fake_eeprom = ctx->fake_eeprom;
|
|
char buffer[] = {0xff, 0xff};
|
|
|
|
eeprom_buffer->flush_count = 2;
|
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0);
|
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 2);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff);
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff);
|
|
/* Should have only flushed the first two bytes. */
|
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[2], 0);
|
|
}
|
|
|
|
static int eeprom_buffer_test_init(struct kunit *test)
|
|
{
|
|
struct eeprom_buffer_test *ctx;
|
|
|
|
ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
|
|
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx);
|
|
|
|
ctx->fake_eeprom = kunit_kzalloc(test, sizeof(*ctx->fake_eeprom), GFP_KERNEL);
|
|
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->fake_eeprom);
|
|
fake_eeprom_init(ctx->fake_eeprom);
|
|
|
|
ctx->eeprom_buffer = new_eeprom_buffer(&ctx->fake_eeprom->parent);
|
|
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->eeprom_buffer);
|
|
|
|
test->priv = ctx;
|
|
|
|
return 0;
|
|
}
|
|
|
|
static void eeprom_buffer_test_exit(struct kunit *test)
|
|
{
|
|
struct eeprom_buffer_test *ctx = test->priv;
|
|
|
|
destroy_eeprom_buffer(ctx->eeprom_buffer);
|
|
}
|
|
|
|
Testing against multiple inputs
|
|
-------------------------------
|
|
|
|
Testing just a few inputs might not be enough to have confidence that the code
|
|
works correctly, e.g. for a hash function.
|
|
|
|
In such cases, it can be helpful to have a helper macro or function, e.g. this
|
|
fictitious example for ``sha1sum(1)``
|
|
|
|
.. code-block:: c
|
|
|
|
#define TEST_SHA1(in, want) \
|
|
sha1sum(in, out); \
|
|
KUNIT_EXPECT_STREQ_MSG(test, out, want, "sha1sum(%s)", in);
|
|
|
|
char out[40];
|
|
TEST_SHA1("hello world", "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed");
|
|
TEST_SHA1("hello world!", "430ce34d020724ed75a196dfc2ad67c77772d169");
|
|
|
|
|
|
Note the use of ``KUNIT_EXPECT_STREQ_MSG`` to give more context when it fails
|
|
and make it easier to track down. (Yes, in this example, ``want`` is likely
|
|
going to be unique enough on its own).
|
|
|
|
The ``_MSG`` variants are even more useful when the same expectation is called
|
|
multiple times (in a loop or helper function) and thus the line number isn't
|
|
enough to identify what failed, like below.
|
|
|
|
In some cases, it can be helpful to write a *table-driven test* instead, e.g.
|
|
|
|
.. code-block:: c
|
|
|
|
int i;
|
|
char out[40];
|
|
|
|
struct sha1_test_case {
|
|
const char *str;
|
|
const char *sha1;
|
|
};
|
|
|
|
struct sha1_test_case cases[] = {
|
|
{
|
|
.str = "hello world",
|
|
.sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
|
|
},
|
|
{
|
|
.str = "hello world!",
|
|
.sha1 = "430ce34d020724ed75a196dfc2ad67c77772d169",
|
|
},
|
|
};
|
|
for (i = 0; i < ARRAY_SIZE(cases); ++i) {
|
|
sha1sum(cases[i].str, out);
|
|
KUNIT_EXPECT_STREQ_MSG(test, out, cases[i].sha1,
|
|
"sha1sum(%s)", cases[i].str);
|
|
}
|
|
|
|
|
|
There's more boilerplate involved, but it can:
|
|
|
|
* be more readable when there are multiple inputs/outputs thanks to field names,
|
|
|
|
* E.g. see ``fs/ext4/inode-test.c`` for an example of both.
|
|
* reduce duplication if test cases can be shared across multiple tests.
|
|
|
|
* E.g. if we wanted to also test ``sha256sum``, we could add a ``sha256``
|
|
field and reuse ``cases``.
|
|
|
|
* be converted to a "parameterized test", see below.
|
|
|
|
Parameterized Testing
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The table-driven testing pattern is common enough that KUnit has special
|
|
support for it.
|
|
|
|
Reusing the same ``cases`` array from above, we can write the test as a
|
|
"parameterized test" with the following.
|
|
|
|
.. code-block:: c
|
|
|
|
// This is copy-pasted from above.
|
|
struct sha1_test_case {
|
|
const char *str;
|
|
const char *sha1;
|
|
};
|
|
struct sha1_test_case cases[] = {
|
|
{
|
|
.str = "hello world",
|
|
.sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
|
|
},
|
|
{
|
|
.str = "hello world!",
|
|
.sha1 = "430ce34d020724ed75a196dfc2ad67c77772d169",
|
|
},
|
|
};
|
|
|
|
// Need a helper function to generate a name for each test case.
|
|
static void case_to_desc(const struct sha1_test_case *t, char *desc)
|
|
{
|
|
strcpy(desc, t->str);
|
|
}
|
|
// Creates `sha1_gen_params()` to iterate over `cases`.
|
|
KUNIT_ARRAY_PARAM(sha1, cases, case_to_desc);
|
|
|
|
// Looks no different from a normal test.
|
|
static void sha1_test(struct kunit *test)
|
|
{
|
|
// This function can just contain the body of the for-loop.
|
|
// The former `cases[i]` is accessible under test->param_value.
|
|
char out[40];
|
|
struct sha1_test_case *test_param = (struct sha1_test_case *)(test->param_value);
|
|
|
|
sha1sum(test_param->str, out);
|
|
KUNIT_EXPECT_STREQ_MSG(test, out, test_param->sha1,
|
|
"sha1sum(%s)", test_param->str);
|
|
}
|
|
|
|
// Instead of KUNIT_CASE, we use KUNIT_CASE_PARAM and pass in the
|
|
// function declared by KUNIT_ARRAY_PARAM.
|
|
static struct kunit_case sha1_test_cases[] = {
|
|
KUNIT_CASE_PARAM(sha1_test, sha1_gen_params),
|
|
{}
|
|
};
|
|
|
|
.. _kunit-on-non-uml:
|
|
|
|
KUnit on non-UML architectures
|
|
==============================
|
|
|
|
By default KUnit uses UML as a way to provide dependencies for code under test.
|
|
Under most circumstances KUnit's usage of UML should be treated as an
|
|
implementation detail of how KUnit works under the hood. Nevertheless, there
|
|
are instances where being able to run architecture-specific code or test
|
|
against real hardware is desirable. For these reasons KUnit supports running on
|
|
other architectures.
|
|
|
|
Running existing KUnit tests on non-UML architectures
|
|
-----------------------------------------------------
|
|
|
|
There are some special considerations when running existing KUnit tests on
|
|
non-UML architectures:
|
|
|
|
* Hardware may not be deterministic, so a test that always passes or fails
|
|
when run under UML may not always do so on real hardware.
|
|
* Hardware and VM environments may not be hermetic. KUnit tries its best to
|
|
provide a hermetic environment to run tests; however, it cannot manage state
|
|
that it doesn't know about outside of the kernel. Consequently, tests that
|
|
may be hermetic on UML may not be hermetic on other architectures.
|
|
* Some features and tooling may not be supported outside of UML.
|
|
* Hardware and VMs are slower than UML.
|
|
|
|
None of these are reasons not to run your KUnit tests on real hardware; they are
|
|
only things to be aware of when doing so.
|
|
|
|
Currently, the KUnit Wrapper (``tools/testing/kunit/kunit.py``) (aka
|
|
kunit_tool) only fully supports running tests inside of UML and QEMU; however,
|
|
this is only due to our own time limitations as humans working on KUnit. It is
|
|
entirely possible to support other emulators and even actual hardware, but for
|
|
now QEMU and UML is what is fully supported within the KUnit Wrapper. Again, to
|
|
be clear, this is just the Wrapper. The actualy KUnit tests and the KUnit
|
|
library they are written in is fully architecture agnostic and can be used in
|
|
virtually any setup, you just won't have the benefit of typing a single command
|
|
out of the box and having everything magically work perfectly.
|
|
|
|
Again, all core KUnit framework features are fully supported on all
|
|
architectures, and using them is straightforward: Most popular architectures
|
|
are supported directly in the KUnit Wrapper via QEMU. Currently, supported
|
|
architectures on QEMU include:
|
|
|
|
* i386
|
|
* x86_64
|
|
* arm
|
|
* arm64
|
|
* alpha
|
|
* powerpc
|
|
* riscv
|
|
* s390
|
|
* sparc
|
|
|
|
In order to run KUnit tests on one of these architectures via QEMU with the
|
|
KUnit wrapper, all you need to do is specify the flags ``--arch`` and
|
|
``--cross_compile`` when invoking the KUnit Wrapper. For example, we could run
|
|
the default KUnit tests on ARM in the following manner (assuming we have an ARM
|
|
toolchain installed):
|
|
|
|
.. code-block:: bash
|
|
|
|
tools/testing/kunit/kunit.py run --timeout=60 --jobs=12 --arch=arm --cross_compile=arm-linux-gnueabihf-
|
|
|
|
Alternatively, if you want to run your tests on real hardware or in some other
|
|
emulation environment, all you need to do is to take your kunitconfig, your
|
|
Kconfig options for the tests you would like to run, and merge them into
|
|
whatever config your are using for your platform. That's it!
|
|
|
|
For example, let's say you have the following kunitconfig:
|
|
|
|
.. code-block:: none
|
|
|
|
CONFIG_KUNIT=y
|
|
CONFIG_KUNIT_EXAMPLE_TEST=y
|
|
|
|
If you wanted to run this test on an x86 VM, you might add the following config
|
|
options to your ``.config``:
|
|
|
|
.. code-block:: none
|
|
|
|
CONFIG_KUNIT=y
|
|
CONFIG_KUNIT_EXAMPLE_TEST=y
|
|
CONFIG_SERIAL_8250=y
|
|
CONFIG_SERIAL_8250_CONSOLE=y
|
|
|
|
All these new options do is enable support for a common serial console needed
|
|
for logging.
|
|
|
|
Next, you could build a kernel with these tests as follows:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
make ARCH=x86 olddefconfig
|
|
make ARCH=x86
|
|
|
|
Once you have built a kernel, you could run it on QEMU as follows:
|
|
|
|
.. code-block:: bash
|
|
|
|
qemu-system-x86_64 -enable-kvm \
|
|
-m 1024 \
|
|
-kernel arch/x86_64/boot/bzImage \
|
|
-append 'console=ttyS0' \
|
|
--nographic
|
|
|
|
Interspersed in the kernel logs you might see the following:
|
|
|
|
.. code-block:: none
|
|
|
|
TAP version 14
|
|
# Subtest: example
|
|
1..1
|
|
# example_simple_test: initializing
|
|
ok 1 - example_simple_test
|
|
ok 1 - example
|
|
|
|
Congratulations, you just ran a KUnit test on the x86 architecture!
|
|
|
|
In a similar manner, kunit and kunit tests can also be built as modules,
|
|
so if you wanted to run tests in this way you might add the following config
|
|
options to your ``.config``:
|
|
|
|
.. code-block:: none
|
|
|
|
CONFIG_KUNIT=m
|
|
CONFIG_KUNIT_EXAMPLE_TEST=m
|
|
|
|
Once the kernel is built and installed, a simple
|
|
|
|
.. code-block:: bash
|
|
|
|
modprobe example-test
|
|
|
|
...will run the tests.
|
|
|
|
.. note::
|
|
Note that you should make sure your test depends on ``KUNIT=y`` in Kconfig
|
|
if the test does not support module build. Otherwise, it will trigger
|
|
compile errors if ``CONFIG_KUNIT`` is ``m``.
|
|
|
|
Writing new tests for other architectures
|
|
-----------------------------------------
|
|
|
|
The first thing you must do is ask yourself whether it is necessary to write a
|
|
KUnit test for a specific architecture, and then whether it is necessary to
|
|
write that test for a particular piece of hardware. In general, writing a test
|
|
that depends on having access to a particular piece of hardware or software (not
|
|
included in the Linux source repo) should be avoided at all costs.
|
|
|
|
Even if you only ever plan on running your KUnit test on your hardware
|
|
configuration, other people may want to run your tests and may not have access
|
|
to your hardware. If you write your test to run on UML, then anyone can run your
|
|
tests without knowing anything about your particular setup, and you can still
|
|
run your tests on your hardware setup just by compiling for your architecture.
|
|
|
|
.. important::
|
|
Always prefer tests that run on UML to tests that only run under a particular
|
|
architecture, and always prefer tests that run under QEMU or another easy
|
|
(and monetarily free) to obtain software environment to a specific piece of
|
|
hardware.
|
|
|
|
Nevertheless, there are still valid reasons to write an architecture or hardware
|
|
specific test: for example, you might want to test some code that really belongs
|
|
in ``arch/some-arch/*``. Even so, try your best to write the test so that it
|
|
does not depend on physical hardware: if some of your test cases don't need the
|
|
hardware, only require the hardware for tests that actually need it.
|
|
|
|
Now that you have narrowed down exactly what bits are hardware specific, the
|
|
actual procedure for writing and running the tests is pretty much the same as
|
|
writing normal KUnit tests. One special caveat is that you have to reset
|
|
hardware state in between test cases; if this is not possible, you may only be
|
|
able to run one test case per invocation.
|
|
|
|
.. TODO(brendanhiggins@google.com): Add an actual example of an architecture-
|
|
dependent KUnit test.
|
|
|
|
KUnit debugfs representation
|
|
============================
|
|
When kunit test suites are initialized, they create an associated directory
|
|
in ``/sys/kernel/debug/kunit/<test-suite>``. The directory contains one file
|
|
|
|
- results: "cat results" displays results of each test case and the results
|
|
of the entire suite for the last test run.
|
|
|
|
The debugfs representation is primarily of use when kunit test suites are
|
|
run in a native environment, either as modules or builtin. Having a way
|
|
to display results like this is valuable as otherwise results can be
|
|
intermixed with other events in dmesg output. The maximum size of each
|
|
results file is KUNIT_LOG_SIZE bytes (defined in ``include/kunit/test.h``).
|