From 7b1dd2cf06e1da9a0982937e82736daa6cd400ee Mon Sep 17 00:00:00 2001 From: Mark Rutland Date: Wed, 5 Oct 2022 18:51:49 +0100 Subject: [PATCH 01/29] kunit: log numbers in decimal and hex When KUNIT_EXPECT_EQ() or KUNIT_ASSERT_EQ() log a failure, they log the two values being compared, with numerical values logged in decimal. In some cases, decimal output is painful to consume, and hexadecimal output would be more helpful. For example, this is the case for tests I'm currently developing for the arm64 insn encoding/decoding code, where comparing two 32-bit instruction opcodes results in output such as: | # test_insn_add_shifted_reg: EXPECTATION FAILED at arch/arm64/lib/test_insn.c:2791 | Expected obj_insn == gen_insn, but | obj_insn == 2332164128 | gen_insn == 1258422304 To make this easier to consume, this patch logs the values in both decimal and hexadecimal: | # test_insn_add_shifted_reg: EXPECTATION FAILED at arch/arm64/lib/test_insn.c:2791 | Expected obj_insn == gen_insn, but | obj_insn == 2332164128 (0x8b020020) | gen_insn == 1258422304 (0x4b020020) As can be seen from the example, having hexadecimal makes it significantly easier for a human to spot which specific bits are incorrect. Signed-off-by: Mark Rutland Cc: Brendan Higgins Cc: David Gow Cc: linux-kselftest@vger.kernel.org Cc: kunit-dev@googlegroups.com Acked-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- lib/kunit/assert.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/kunit/assert.c b/lib/kunit/assert.c index d00d6d181ee8..24dec5b48722 100644 --- a/lib/kunit/assert.c +++ b/lib/kunit/assert.c @@ -127,13 +127,15 @@ void kunit_binary_assert_format(const struct kunit_assert *assert, binary_assert->text->right_text); if (!is_literal(stream->test, binary_assert->text->left_text, binary_assert->left_value, stream->gfp)) - string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld\n", + string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld (0x%llx)\n", binary_assert->text->left_text, + binary_assert->left_value, binary_assert->left_value); if (!is_literal(stream->test, binary_assert->text->right_text, binary_assert->right_value, stream->gfp)) - string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld", + string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld (0x%llx)", binary_assert->text->right_text, + binary_assert->right_value, binary_assert->right_value); kunit_assert_print_msg(message, stream); } From f13ecba04babc7b614d8ad896e987549c807e2de Mon Sep 17 00:00:00 2001 From: Sadiya Kazi Date: Tue, 18 Oct 2022 04:03:33 +0000 Subject: [PATCH 02/29] Documentation: Kunit: Update architecture.rst for minor fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated the architecture.rst page with the following changes: -Add missing article _the_ across the document. -Reword content across for style and standard. -Update all occurrences of Command Line to Command-line across the document. -Correct grammatical issues, for example, added _it_wherever missing. -Update all occurrences of “via" to either use “through” or “using”. -Update the text preceding the external links and pushed the full link to a new line for better readability. -Reword content under the config command to make it more clear and concise. Signed-off-by: Sadiya Kazi Reviewed-by: David Gow Signed-off-by: Shuah Khan --- .../dev-tools/kunit/architecture.rst | 115 +++++++++--------- 1 file changed, 58 insertions(+), 57 deletions(-) diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentation/dev-tools/kunit/architecture.rst index 8efe792bdcb9..e95ab05342bb 100644 --- a/Documentation/dev-tools/kunit/architecture.rst +++ b/Documentation/dev-tools/kunit/architecture.rst @@ -4,16 +4,17 @@ KUnit Architecture ================== -The KUnit architecture can be divided into two parts: +The KUnit architecture is divided into two parts: - `In-Kernel Testing Framework`_ -- `kunit_tool (Command Line Test Harness)`_ +- `kunit_tool (Command-line Test Harness)`_ In-Kernel Testing Framework =========================== The kernel testing library supports KUnit tests written in C using -KUnit. KUnit tests are kernel code. KUnit does several things: +KUnit. These KUnit tests are kernel code. KUnit performs the following +tasks: - Organizes tests - Reports test results @@ -22,19 +23,17 @@ KUnit. KUnit tests are kernel code. KUnit does several things: Test Cases ---------- -The fundamental unit in KUnit is the test case. The KUnit test cases are -grouped into KUnit suites. A KUnit test case is a function with type -signature ``void (*)(struct kunit *test)``. -These test case functions are wrapped in a struct called -struct kunit_case. +The test case is the fundamental unit in KUnit. KUnit test cases are organised +into suites. A KUnit test case is a function with type signature +``void (*)(struct kunit *test)``. These test case functions are wrapped in a +struct called struct kunit_case. .. note: ``generate_params`` is optional for non-parameterized tests. -Each KUnit test case gets a ``struct kunit`` context -object passed to it that tracks a running test. The KUnit assertion -macros and other KUnit utilities use the ``struct kunit`` context -object. As an exception, there are two fields: +Each KUnit test case receives a ``struct kunit`` context object that tracks a +running test. The KUnit assertion macros and other KUnit utilities use the +``struct kunit`` context object. As an exception, there are two fields: - ``->priv``: The setup functions can use it to store arbitrary test user data. @@ -77,12 +76,13 @@ Executor The KUnit executor can list and run built-in KUnit tests on boot. The Test suites are stored in a linker section -called ``.kunit_test_suites``. For code, see: -https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/asm-generic/vmlinux.lds.h?h=v5.15#n945. +called ``.kunit_test_suites``. For the code, see ``KUNIT_TABLE()`` macro +definition in +`include/asm-generic/vmlinux.lds.h `_. The linker section consists of an array of pointers to ``struct kunit_suite``, and is populated by the ``kunit_test_suites()`` -macro. To run all tests compiled into the kernel, the KUnit executor -iterates over the linker section array. +macro. The KUnit executor iterates over the linker section array in order to +run all the tests that are compiled into the kernel. .. kernel-figure:: kunit_suitememorydiagram.svg :alt: KUnit Suite Memory @@ -90,17 +90,17 @@ iterates over the linker section array. KUnit Suite Memory Diagram On the kernel boot, the KUnit executor uses the start and end addresses -of this section to iterate over and run all tests. For code, see: -https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/kunit/executor.c - +of this section to iterate over and run all tests. For the implementation of the +executor, see +`lib/kunit/executor.c `_. When built as a module, the ``kunit_test_suites()`` macro defines a ``module_init()`` function, which runs all the tests in the compilation unit instead of utilizing the executor. In KUnit tests, some error classes do not affect other tests or parts of the kernel, each KUnit case executes in a separate thread -context. For code, see: -https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/kunit/try-catch.c?h=v5.15#n58 +context. See the ``kunit_try_catch_run()`` function in +`lib/kunit/try-catch.c `_. Assertion Macros ---------------- @@ -111,37 +111,36 @@ All expectations/assertions are formatted as: - ``{EXPECT|ASSERT}`` determines whether the check is an assertion or an expectation. + In the event of a failure, the testing flow differs as follows: - - For an expectation, if the check fails, marks the test as failed - and logs the failure. + - For expectations, the test is marked as failed and the failure is logged. - - An assertion, on failure, causes the test case to terminate - immediately. + - Failing assertions, on the other hand, result in the test case being + terminated immediately. - - Assertions call function: + - Assertions call the function: ``void __noreturn kunit_abort(struct kunit *)``. - - ``kunit_abort`` calls function: + - ``kunit_abort`` calls the function: ``void __noreturn kunit_try_catch_throw(struct kunit_try_catch *try_catch)``. - - ``kunit_try_catch_throw`` calls function: + - ``kunit_try_catch_throw`` calls the function: ``void kthread_complete_and_exit(struct completion *, long) __noreturn;`` and terminates the special thread context. - ```` denotes a check with options: ``TRUE`` (supplied property - has the boolean value “true”), ``EQ`` (two supplied properties are + has the boolean value "true"), ``EQ`` (two supplied properties are equal), ``NOT_ERR_OR_NULL`` (supplied pointer is not null and does not - contain an “err” value). + contain an "err" value). - ``[_MSG]`` prints a custom message on failure. Test Result Reporting --------------------- -KUnit prints test results in KTAP format. KTAP is based on TAP14, see: -https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md. -KTAP (yet to be standardized format) works with KUnit and Kselftest. -The KUnit executor prints KTAP results to dmesg, and debugfs -(if configured). +KUnit prints the test results in KTAP format. KTAP is based on TAP14, see +Documentation/dev-tools/ktap.rst. +KTAP works with KUnit and Kselftest. The KUnit executor prints KTAP results to +dmesg, and debugfs (if configured). Parameterized Tests ------------------- @@ -150,33 +149,35 @@ Each KUnit parameterized test is associated with a collection of parameters. The test is invoked multiple times, once for each parameter value and the parameter is stored in the ``param_value`` field. The test case includes a KUNIT_CASE_PARAM() macro that accepts a -generator function. -The generator function is passed the previous parameter and returns the next -parameter. It also provides a macro to generate common-case generators based on -arrays. +generator function. The generator function is passed the previous parameter +and returns the next parameter. It also includes a macro for generating +array-based common-case generators. -kunit_tool (Command Line Test Harness) +kunit_tool (Command-line Test Harness) ====================================== -kunit_tool is a Python script ``(tools/testing/kunit/kunit.py)`` -that can be used to configure, build, exec, parse and run (runs other -commands in order) test results. You can either run KUnit tests using -kunit_tool or can include KUnit in kernel and parse manually. +``kunit_tool`` is a Python script, found in ``tools/testing/kunit/kunit.py``. It +is used to configure, build, execute, parse test results and run all of the +previous commands in correct order (i.e., configure, build, execute and parse). +You have two options for running KUnit tests: either build the kernel with KUnit +enabled and manually parse the results (see +Documentation/dev-tools/kunit/run_manual.rst) or use ``kunit_tool`` +(see Documentation/dev-tools/kunit/run_wrapper.rst). - ``configure`` command generates the kernel ``.config`` from a ``.kunitconfig`` file (and any architecture-specific options). - For some architectures, additional config options are specified in the - ``qemu_config`` Python script - (For example: ``tools/testing/kunit/qemu_configs/powerpc.py``). + The Python scripts available in ``qemu_configs`` folder + (for example, ``tools/testing/kunit/qemu configs/powerpc.py``) contains + additional configuration options for specific architectures. It parses both the existing ``.config`` and the ``.kunitconfig`` files - and ensures that ``.config`` is a superset of ``.kunitconfig``. - If this is not the case, it will combine the two and run - ``make olddefconfig`` to regenerate the ``.config`` file. It then - verifies that ``.config`` is now a superset. This checks if all - Kconfig dependencies are correctly specified in ``.kunitconfig``. - ``kunit_config.py`` includes the parsing Kconfigs code. The code which - runs ``make olddefconfig`` is a part of ``kunit_kernel.py``. You can - invoke this command via: ``./tools/testing/kunit/kunit.py config`` and + to ensure that ``.config`` is a superset of ``.kunitconfig``. + If not, it will combine the two and run ``make olddefconfig`` to regenerate + the ``.config`` file. It then checks to see if ``.config`` has become a superset. + This verifies that all the Kconfig dependencies are correctly specified in the + file ``.kunitconfig``. The ``kunit_config.py`` script contains the code for parsing + Kconfigs. The code which runs ``make olddefconfig`` is part of the + ``kunit_kernel.py`` script. You can invoke this command through: + ``./tools/testing/kunit/kunit.py config`` and generate a ``.config`` file. - ``build`` runs ``make`` on the kernel tree with required options (depends on the architecture and some options, for example: build_dir) @@ -184,8 +185,8 @@ kunit_tool or can include KUnit in kernel and parse manually. To build a KUnit kernel from the current ``.config``, you can use the ``build`` argument: ``./tools/testing/kunit/kunit.py build``. - ``exec`` command executes kernel results either directly (using - User-mode Linux configuration), or via an emulator such - as QEMU. It reads results from the log via standard + User-mode Linux configuration), or through an emulator such + as QEMU. It reads results from the log using standard output (stdout), and passes them to ``parse`` to be parsed. If you already have built a kernel with built-in KUnit tests, you can run the kernel and display the test results with the ``exec`` From b8a926bea8b1e790b0afe21359c086e3ee08aee5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ADra=20Canal?= Date: Tue, 25 Oct 2022 20:10:41 -0300 Subject: [PATCH 03/29] kunit: Introduce KUNIT_EXPECT_MEMEQ and KUNIT_EXPECT_MEMNEQ macros MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Currently, in order to compare memory blocks in KUnit, the KUNIT_EXPECT_EQ or KUNIT_EXPECT_FALSE macros are used in conjunction with the memcmp function, such as: KUNIT_EXPECT_EQ(test, memcmp(foo, bar, size), 0); Although this usage produces correct results for the test cases, when the expectation fails, the error message is not very helpful, indicating only the return of the memcmp function. Therefore, create a new set of macros KUNIT_EXPECT_MEMEQ and KUNIT_EXPECT_MEMNEQ that compare memory blocks until a specified size. In case of expectation failure, those macros print the hex dump of the memory blocks, making it easier to debug test failures for memory blocks. That said, the expectation KUNIT_EXPECT_EQ(test, memcmp(foo, bar, size), 0); would translate to the expectation KUNIT_EXPECT_MEMEQ(test, foo, bar, size); Signed-off-by: Maíra Canal Reviewed-by: Daniel Latypov Reviewed-by: Muhammad Usama Anjum Reviewed-by: David Gow Signed-off-by: Shuah Khan --- include/kunit/assert.h | 33 ++++++++++++++++ include/kunit/test.h | 87 ++++++++++++++++++++++++++++++++++++++++++ lib/kunit/assert.c | 56 +++++++++++++++++++++++++++ 3 files changed, 176 insertions(+) diff --git a/include/kunit/assert.h b/include/kunit/assert.h index ace3de8d1ee7..e8a59487fd59 100644 --- a/include/kunit/assert.h +++ b/include/kunit/assert.h @@ -240,4 +240,37 @@ void kunit_binary_str_assert_format(const struct kunit_assert *assert, const struct va_format *message, struct string_stream *stream); +#define KUNIT_INIT_MEM_ASSERT_STRUCT(text_, left_val, right_val, size_) { \ + .text = text_, \ + .left_value = left_val, \ + .right_value = right_val, \ + .size = size_ \ +} + +/** + * struct kunit_mem_assert - An expectation/assertion that compares two + * memory blocks. + * @assert: The parent of this type. + * @text: Holds the textual representations of the operands and comparator. + * @left_value: The actual evaluated value of the expression in the left slot. + * @right_value: The actual evaluated value of the expression in the right slot. + * @size: Size of the memory block analysed in bytes. + * + * Represents an expectation/assertion that compares two memory blocks. For + * example, to expect that the first three bytes of foo is equal to the + * first three bytes of bar, you can use the expectation + * KUNIT_EXPECT_MEMEQ(test, foo, bar, 3); + */ +struct kunit_mem_assert { + struct kunit_assert assert; + const struct kunit_binary_assert_text *text; + const void *left_value; + const void *right_value; + const size_t size; +}; + +void kunit_mem_assert_format(const struct kunit_assert *assert, + const struct va_format *message, + struct string_stream *stream); + #endif /* _KUNIT_ASSERT_H */ diff --git a/include/kunit/test.h b/include/kunit/test.h index b1ab6b32216d..cde97dc4eed5 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -658,6 +658,39 @@ do { \ ##__VA_ARGS__); \ } while (0) +#define KUNIT_MEM_ASSERTION(test, \ + assert_type, \ + left, \ + op, \ + right, \ + size, \ + fmt, \ + ...) \ +do { \ + const void *__left = (left); \ + const void *__right = (right); \ + const size_t __size = (size); \ + static const struct kunit_binary_assert_text __text = { \ + .operation = #op, \ + .left_text = #left, \ + .right_text = #right, \ + }; \ + \ + if (likely(memcmp(__left, __right, __size) op 0)) \ + break; \ + \ + _KUNIT_FAILED(test, \ + assert_type, \ + kunit_mem_assert, \ + kunit_mem_assert_format, \ + KUNIT_INIT_MEM_ASSERT_STRUCT(&__text, \ + __left, \ + __right, \ + __size), \ + fmt, \ + ##__VA_ARGS__); \ +} while (0) + #define KUNIT_PTR_NOT_ERR_OR_NULL_MSG_ASSERTION(test, \ assert_type, \ ptr, \ @@ -928,6 +961,60 @@ do { \ fmt, \ ##__VA_ARGS__) +/** + * KUNIT_EXPECT_MEMEQ() - Expects that the first @size bytes of @left and @right are equal. + * @test: The test context object. + * @left: An arbitrary expression that evaluates to the specified size. + * @right: An arbitrary expression that evaluates to the specified size. + * @size: Number of bytes compared. + * + * Sets an expectation that the values that @left and @right evaluate to are + * equal. This is semantically equivalent to + * KUNIT_EXPECT_TRUE(@test, !memcmp((@left), (@right), (@size))). See + * KUNIT_EXPECT_TRUE() for more information. + * + * Although this expectation works for any memory block, it is not recommended + * for comparing more structured data, such as structs. This expectation is + * recommended for comparing, for example, data arrays. + */ +#define KUNIT_EXPECT_MEMEQ(test, left, right, size) \ + KUNIT_EXPECT_MEMEQ_MSG(test, left, right, size, NULL) + +#define KUNIT_EXPECT_MEMEQ_MSG(test, left, right, size, fmt, ...) \ + KUNIT_MEM_ASSERTION(test, \ + KUNIT_EXPECTATION, \ + left, ==, right, \ + size, \ + fmt, \ + ##__VA_ARGS__) + +/** + * KUNIT_EXPECT_MEMNEQ() - Expects that the first @size bytes of @left and @right are not equal. + * @test: The test context object. + * @left: An arbitrary expression that evaluates to the specified size. + * @right: An arbitrary expression that evaluates to the specified size. + * @size: Number of bytes compared. + * + * Sets an expectation that the values that @left and @right evaluate to are + * not equal. This is semantically equivalent to + * KUNIT_EXPECT_TRUE(@test, memcmp((@left), (@right), (@size))). See + * KUNIT_EXPECT_TRUE() for more information. + * + * Although this expectation works for any memory block, it is not recommended + * for comparing more structured data, such as structs. This expectation is + * recommended for comparing, for example, data arrays. + */ +#define KUNIT_EXPECT_MEMNEQ(test, left, right, size) \ + KUNIT_EXPECT_MEMNEQ_MSG(test, left, right, size, NULL) + +#define KUNIT_EXPECT_MEMNEQ_MSG(test, left, right, size, fmt, ...) \ + KUNIT_MEM_ASSERTION(test, \ + KUNIT_EXPECTATION, \ + left, !=, right, \ + size, \ + fmt, \ + ##__VA_ARGS__) + /** * KUNIT_EXPECT_NULL() - Expects that @ptr is null. * @test: The test context object. diff --git a/lib/kunit/assert.c b/lib/kunit/assert.c index 24dec5b48722..f5b50babe38d 100644 --- a/lib/kunit/assert.c +++ b/lib/kunit/assert.c @@ -206,3 +206,59 @@ void kunit_binary_str_assert_format(const struct kunit_assert *assert, kunit_assert_print_msg(message, stream); } EXPORT_SYMBOL_GPL(kunit_binary_str_assert_format); + +/* Adds a hexdump of a buffer to a string_stream comparing it with + * a second buffer. The different bytes are marked with <>. + */ +static void kunit_assert_hexdump(struct string_stream *stream, + const void *buf, + const void *compared_buf, + const size_t len) +{ + size_t i; + const u8 *buf1 = buf; + const u8 *buf2 = compared_buf; + + string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT); + + for (i = 0; i < len; ++i) { + if (!(i % 16) && i) + string_stream_add(stream, "\n" KUNIT_SUBSUBTEST_INDENT); + + if (buf1[i] != buf2[i]) + string_stream_add(stream, "<%02x>", buf1[i]); + else + string_stream_add(stream, " %02x ", buf1[i]); + } +} + +void kunit_mem_assert_format(const struct kunit_assert *assert, + const struct va_format *message, + struct string_stream *stream) +{ + struct kunit_mem_assert *mem_assert; + + mem_assert = container_of(assert, struct kunit_mem_assert, + assert); + + string_stream_add(stream, + KUNIT_SUBTEST_INDENT "Expected %s %s %s, but\n", + mem_assert->text->left_text, + mem_assert->text->operation, + mem_assert->text->right_text); + + string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s ==\n", + mem_assert->text->left_text); + kunit_assert_hexdump(stream, mem_assert->left_value, + mem_assert->right_value, mem_assert->size); + + string_stream_add(stream, "\n"); + + string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s ==\n", + mem_assert->text->right_text); + kunit_assert_hexdump(stream, mem_assert->right_value, + mem_assert->left_value, mem_assert->size); + + kunit_assert_print_msg(message, stream); +} +EXPORT_SYMBOL_GPL(kunit_mem_assert_format); From 3b30fb62ec23b42be33d94ebf825d27daf508e8e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ADra=20Canal?= Date: Tue, 25 Oct 2022 20:10:42 -0300 Subject: [PATCH 04/29] kunit: Add KUnit memory block assertions to the example_all_expect_macros_test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Augment the example_all_expect_macros_test with the KUNIT_EXPECT_MEMEQ and KUNIT_EXPECT_MEMNEQ macros by creating a test with memory block assertions. Signed-off-by: Maíra Canal Reviewed-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- lib/kunit/kunit-example-test.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c index f8fe582c9e36..66cc4e2365ec 100644 --- a/lib/kunit/kunit-example-test.c +++ b/lib/kunit/kunit-example-test.c @@ -86,6 +86,9 @@ static void example_mark_skipped_test(struct kunit *test) */ static void example_all_expect_macros_test(struct kunit *test) { + const u32 array1[] = { 0x0F, 0xFF }; + const u32 array2[] = { 0x1F, 0xFF }; + /* Boolean assertions */ KUNIT_EXPECT_TRUE(test, true); KUNIT_EXPECT_FALSE(test, false); @@ -109,6 +112,10 @@ static void example_all_expect_macros_test(struct kunit *test) KUNIT_EXPECT_STREQ(test, "hi", "hi"); KUNIT_EXPECT_STRNEQ(test, "hi", "bye"); + /* Memory block assertions */ + KUNIT_EXPECT_MEMEQ(test, array1, array1, sizeof(array1)); + KUNIT_EXPECT_MEMNEQ(test, array1, array2, sizeof(array1)); + /* * There are also ASSERT variants of all of the above that abort test * execution if they fail. Useful for memory allocations, etc. From a52a5451f43bb76743c51dd46788008837243f29 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ADra=20Canal?= Date: Tue, 25 Oct 2022 20:10:43 -0300 Subject: [PATCH 05/29] kunit: Use KUNIT_EXPECT_MEMEQ macro MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use KUNIT_EXPECT_MEMEQ to compare memory blocks in replacement of the KUNIT_EXPECT_EQ macro. Therefor, the statement KUNIT_EXPECT_EQ(test, memcmp(foo, bar, size), 0); is replaced by: KUNIT_EXPECT_MEMEQ(test, foo, bar, size); Signed-off-by: Maíra Canal Acked-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- drivers/gpu/drm/tests/drm_format_helper_test.c | 12 ++++++------ net/core/dev_addr_lists_test.c | 4 ++-- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/drivers/gpu/drm/tests/drm_format_helper_test.c b/drivers/gpu/drm/tests/drm_format_helper_test.c index 2191e57f2297..567c71f95edc 100644 --- a/drivers/gpu/drm/tests/drm_format_helper_test.c +++ b/drivers/gpu/drm/tests/drm_format_helper_test.c @@ -315,7 +315,7 @@ static void drm_test_fb_xrgb8888_to_gray8(struct kunit *test) iosys_map_set_vaddr(&src, xrgb8888); drm_fb_xrgb8888_to_gray8(&dst, &result->dst_pitch, &src, &fb, ¶ms->clip); - KUNIT_EXPECT_EQ(test, memcmp(buf, result->expected, dst_size), 0); + KUNIT_EXPECT_MEMEQ(test, buf, result->expected, dst_size); } static void drm_test_fb_xrgb8888_to_rgb332(struct kunit *test) @@ -345,7 +345,7 @@ static void drm_test_fb_xrgb8888_to_rgb332(struct kunit *test) iosys_map_set_vaddr(&src, xrgb8888); drm_fb_xrgb8888_to_rgb332(&dst, &result->dst_pitch, &src, &fb, ¶ms->clip); - KUNIT_EXPECT_EQ(test, memcmp(buf, result->expected, dst_size), 0); + KUNIT_EXPECT_MEMEQ(test, buf, result->expected, dst_size); } static void drm_test_fb_xrgb8888_to_rgb565(struct kunit *test) @@ -375,10 +375,10 @@ static void drm_test_fb_xrgb8888_to_rgb565(struct kunit *test) iosys_map_set_vaddr(&src, xrgb8888); drm_fb_xrgb8888_to_rgb565(&dst, &result->dst_pitch, &src, &fb, ¶ms->clip, false); - KUNIT_EXPECT_EQ(test, memcmp(buf, result->expected, dst_size), 0); + KUNIT_EXPECT_MEMEQ(test, buf, result->expected, dst_size); drm_fb_xrgb8888_to_rgb565(&dst, &result->dst_pitch, &src, &fb, ¶ms->clip, true); - KUNIT_EXPECT_EQ(test, memcmp(buf, result->expected_swab, dst_size), 0); + KUNIT_EXPECT_MEMEQ(test, buf, result->expected_swab, dst_size); } static void drm_test_fb_xrgb8888_to_rgb888(struct kunit *test) @@ -408,7 +408,7 @@ static void drm_test_fb_xrgb8888_to_rgb888(struct kunit *test) iosys_map_set_vaddr(&src, xrgb8888); drm_fb_xrgb8888_to_rgb888(&dst, &result->dst_pitch, &src, &fb, ¶ms->clip); - KUNIT_EXPECT_EQ(test, memcmp(buf, result->expected, dst_size), 0); + KUNIT_EXPECT_MEMEQ(test, buf, result->expected, dst_size); } static void drm_test_fb_xrgb8888_to_xrgb2101010(struct kunit *test) @@ -439,7 +439,7 @@ static void drm_test_fb_xrgb8888_to_xrgb2101010(struct kunit *test) drm_fb_xrgb8888_to_xrgb2101010(&dst, &result->dst_pitch, &src, &fb, ¶ms->clip); buf = le32buf_to_cpu(test, buf, dst_size / sizeof(u32)); - KUNIT_EXPECT_EQ(test, memcmp(buf, result->expected, dst_size), 0); + KUNIT_EXPECT_MEMEQ(test, buf, result->expected, dst_size); } static struct kunit_case drm_format_helper_test_cases[] = { diff --git a/net/core/dev_addr_lists_test.c b/net/core/dev_addr_lists_test.c index 049cfbc58aa9..90e7e3811ae7 100644 --- a/net/core/dev_addr_lists_test.c +++ b/net/core/dev_addr_lists_test.c @@ -71,11 +71,11 @@ static void dev_addr_test_basic(struct kunit *test) memset(addr, 2, sizeof(addr)); eth_hw_addr_set(netdev, addr); - KUNIT_EXPECT_EQ(test, 0, memcmp(netdev->dev_addr, addr, sizeof(addr))); + KUNIT_EXPECT_MEMEQ(test, netdev->dev_addr, addr, sizeof(addr)); memset(addr, 3, sizeof(addr)); dev_addr_set(netdev, addr); - KUNIT_EXPECT_EQ(test, 0, memcmp(netdev->dev_addr, addr, sizeof(addr))); + KUNIT_EXPECT_MEMEQ(test, netdev->dev_addr, addr, sizeof(addr)); } static void dev_addr_test_sync_one(struct kunit *test) From 8f8b51f7d5c8bd3a89e7ea87aed2cdaa52ca5ba4 Mon Sep 17 00:00:00 2001 From: "YoungJun.park" Date: Mon, 24 Oct 2022 18:59:46 -0700 Subject: [PATCH 06/29] kunit: remove unused structure definition remove unused string_stream_alloc_context structure definition. Signed-off-by: YoungJun.park Reviewed-by: David Gow Signed-off-by: Shuah Khan --- lib/kunit/string-stream.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/lib/kunit/string-stream.c b/lib/kunit/string-stream.c index f5ae79c37400..72659a9773e3 100644 --- a/lib/kunit/string-stream.c +++ b/lib/kunit/string-stream.c @@ -131,11 +131,6 @@ bool string_stream_is_empty(struct string_stream *stream) return list_empty(&stream->fragments); } -struct string_stream_alloc_context { - struct kunit *test; - gfp_t gfp; -}; - struct string_stream *alloc_string_stream(struct kunit *test, gfp_t gfp) { struct string_stream *stream; From 3ffdcf7e3b7dff04b055771c03c9646aa383cc1e Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Fri, 28 Oct 2022 14:02:55 -0700 Subject: [PATCH 07/29] kunit: tool: make unit test not print parsed testdata to stdout Currently, if you run $ ./tools/testing/kunit/kunit_tool_test.py you'll see a lot of output from the parser as we feed it testdata. This makes the output hard to read and fairly confusing, esp. since our testdata includes example failures, which get printed out in red. Silence that output so real failures are easier to see. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit_tool_test.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index e2cd2cc2e98f..a6e53945656e 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -80,6 +80,9 @@ class KconfigTest(unittest.TestCase): self.assertEqual(actual_kconfig, expected_kconfig) class KUnitParserTest(unittest.TestCase): + def setUp(self): + self.print_mock = mock.patch('kunit_printer.Printer.print').start() + self.addCleanup(mock.patch.stopall) def assertContains(self, needle: str, haystack: kunit_parser.LineStream): # Clone the iterator so we can print the contents on failure. @@ -485,6 +488,9 @@ class LinuxSourceTreeTest(unittest.TestCase): class KUnitJsonTest(unittest.TestCase): + def setUp(self): + self.print_mock = mock.patch('kunit_printer.Printer.print').start() + self.addCleanup(mock.patch.stopall) def _json_for(self, log_file): with open(test_data_path(log_file)) as file: From f19dd011d8de6f0c1d20abea5158aa4f5d9cea44 Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Fri, 28 Oct 2022 14:02:56 -0700 Subject: [PATCH 08/29] kunit: tool: print summary of failed tests if a few failed out of a lot E.g. all the hw_breakpoint tests are failing right now. So if I run `kunit.py run --altests --arch=x86_64`, then I see > Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7 Seeing which 9 tests failed out of the hundreds is annoying. If my terminal doesn't have scrollback support, I have to resort to looking at `.kunit/test.log` for the `not ok` lines. Teach kunit.py to print a summarized list of failures if the # of tests reachs an arbitrary threshold (>=100 tests). To try and keep the output from being too long/noisy, this new logic a) just reports "parent_test failed" if every child test failed b) won't print anything if there are >10 failures (also arbitrary). With this patch, we get an extra line of output showing: > Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7 > Failures: hw_breakpoint This also works with parameterized tests, e.g. if I add a fake failure > Failures: kcsan.test_atomic_builtins_missing_barrier.threads=6 Note: we didn't have enough tests for this to be a problem before. But with commit 980ac3ad0512 ("kunit: tool: rename all_test_uml.config, use it for --alltests"), --alltests works and thus running >100 tests will probably become more common. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit_parser.py | 47 ++++++++++++++++++++++++++ tools/testing/kunit/kunit_tool_test.py | 22 ++++++++++++ 2 files changed, 69 insertions(+) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 1ae873e3e341..94dba66feec5 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -58,6 +58,10 @@ class Test: self.counts.errors += 1 stdout.print_with_timestamp(stdout.red('[ERROR]') + f' Test: {self.name}: {error_message}') + def ok_status(self) -> bool: + """Returns true if the status was ok, i.e. passed or skipped.""" + return self.status in (TestStatus.SUCCESS, TestStatus.SKIPPED) + class TestStatus(Enum): """An enumeration class to represent the status of a test.""" SUCCESS = auto() @@ -565,6 +569,40 @@ def print_test_footer(test: Test) -> None: stdout.print_with_timestamp(format_test_divider(message, len(message) - stdout.color_len())) + + +def _summarize_failed_tests(test: Test) -> str: + """Tries to summarize all the failing subtests in `test`.""" + + def failed_names(test: Test, parent_name: str) -> List[str]: + # Note: we use 'main' internally for the top-level test. + if not parent_name or parent_name == 'main': + full_name = test.name + else: + full_name = parent_name + '.' + test.name + + if not test.subtests: # this is a leaf node + return [full_name] + + # If all the children failed, just say this subtest failed. + # Don't summarize it down "the top-level test failed", though. + failed_subtests = [sub for sub in test.subtests if not sub.ok_status()] + if parent_name and len(failed_subtests) == len(test.subtests): + return [full_name] + + all_failures = [] # type: List[str] + for t in failed_subtests: + all_failures.extend(failed_names(t, full_name)) + return all_failures + + failures = failed_names(test, '') + # If there are too many failures, printing them out will just be noisy. + if len(failures) > 10: # this is an arbitrary limit + return '' + + return 'Failures: ' + ', '.join(failures) + + def print_summary_line(test: Test) -> None: """ Prints summary line of test object. Color of line is dependent on @@ -587,6 +625,15 @@ def print_summary_line(test: Test) -> None: color = stdout.red stdout.print_with_timestamp(color(f'Testing complete. {test.counts}')) + # Summarize failures that might have gone off-screen since we had a lot + # of tests (arbitrarily defined as >=100 for now). + if test.ok_status() or test.counts.total() < 100: + return + summarized = _summarize_failed_tests(test) + if not summarized: + return + stdout.print_with_timestamp(color(summarized)) + # Other methods: def bubble_up_test_results(test: Test) -> None: diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index a6e53945656e..7dcd67003b23 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -312,6 +312,28 @@ class KUnitParserTest(unittest.TestCase): result.status) self.assertEqual('kunit-resource-test', result.subtests[0].name) + def test_summarize_failures(self): + output = """ + KTAP version 1 + 1..2 + # Subtest: all_failed_suite + 1..2 + not ok 1 - test1 + not ok 2 - test2 + not ok 1 - all_failed_suite + # Subtest: some_failed_suite + 1..2 + ok 1 - test1 + not ok 2 - test2 + not ok 1 - some_failed_suite + """ + result = kunit_parser.parse_run_tests(output.splitlines()) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) + + self.assertEqual(kunit_parser._summarize_failed_tests(result), + 'Failures: all_failed_suite, some_failed_suite.test2') + + def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: return kunit_parser.LineStream(enumerate(strs, start=1)) From f473dd9488d910aab109e8c6a2e4181125ca322a Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Thu, 3 Nov 2022 10:47:38 -0700 Subject: [PATCH 09/29] kunit: tool: make TestCounts a dataclass Since we're using Python 3.7+, we can use dataclasses to tersen the code. It also lets us create pre-populated TestCounts() objects and compare them in our unit test. (Before, you could only create empty ones). Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit_parser.py | 23 +++++++---------------- tools/testing/kunit/kunit_tool_test.py | 4 +--- 2 files changed, 8 insertions(+), 19 deletions(-) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 94dba66feec5..a56c75a973b5 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -10,6 +10,7 @@ # Author: Rae Moar from __future__ import annotations +from dataclasses import dataclass import re import sys @@ -71,27 +72,17 @@ class TestStatus(Enum): NO_TESTS = auto() FAILURE_TO_PARSE_TESTS = auto() +@dataclass class TestCounts: """ Tracks the counts of statuses of all test cases and any errors within a Test. - - Attributes: - passed : int - the number of tests that have passed - failed : int - the number of tests that have failed - crashed : int - the number of tests that have crashed - skipped : int - the number of tests that have skipped - errors : int - the number of errors in the test and subtests """ - def __init__(self): - """Creates TestCounts object with counts of all test - statuses and test errors set to 0. - """ - self.passed = 0 - self.failed = 0 - self.crashed = 0 - self.skipped = 0 - self.errors = 0 + passed: int = 0 + failed: int = 0 + crashed: int = 0 + skipped: int = 0 + errors: int = 0 def __str__(self) -> str: """Returns the string representation of a TestCounts object.""" diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 7dcd67003b23..440a273f1d21 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -182,9 +182,7 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.extract_tap_lines( file.readlines())) # A missing test plan is not an error. - self.assertEqual(0, result.counts.errors) - # All tests should be accounted for. - self.assertEqual(10, result.counts.total()) + self.assertEqual(result.counts, kunit_parser.TestCounts(passed=10, errors=0)) self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) From 05d9d2c3ee1e4b587f71455f6d3d1493289204ff Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Thu, 3 Nov 2022 10:47:39 -0700 Subject: [PATCH 10/29] kunit: tool: unit tests all check parser errors, standardize formatting a bit Let's verify that the parser isn't reporting any errors for valid inputs. This change also * does result.status checking on one line * makes sure we consistently do it outside of the `with` block Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit_tool_test.py | 93 +++++++++++--------------- 1 file changed, 38 insertions(+), 55 deletions(-) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 440a273f1d21..5e3429a1202b 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -136,33 +136,29 @@ class KUnitParserTest(unittest.TestCase): all_passed_log = test_data_path('test_is_test_passed-all_passed.log') with open(all_passed_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual(result.counts.errors, 0) def test_parse_successful_nested_tests_log(self): all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log') with open(all_passed_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual(result.counts.errors, 0) def test_kselftest_nested(self): kselftest_log = test_data_path('test_is_test_passed-kselftest.log') with open(kselftest_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual(result.counts.errors, 0) def test_parse_failed_test_log(self): failed_log = test_data_path('test_is_test_passed-failure.log') with open(failed_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.FAILURE, - result.status) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) + self.assertEqual(result.counts.errors, 0) def test_no_header(self): empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log') @@ -170,9 +166,8 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) self.assertEqual(0, len(result.subtests)) - self.assertEqual( - kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, - result.status) + self.assertEqual(kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status) + self.assertEqual(result.counts.errors, 1) def test_missing_test_plan(self): missing_plan_log = test_data_path('test_is_test_passed-' @@ -183,9 +178,7 @@ class KUnitParserTest(unittest.TestCase): file.readlines())) # A missing test plan is not an error. self.assertEqual(result.counts, kunit_parser.TestCounts(passed=10, errors=0)) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) def test_no_tests(self): header_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log') @@ -193,9 +186,8 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) self.assertEqual(0, len(result.subtests)) - self.assertEqual( - kunit_parser.TestStatus.NO_TESTS, - result.status) + self.assertEqual(kunit_parser.TestStatus.NO_TESTS, result.status) + self.assertEqual(result.counts.errors, 1) def test_no_tests_no_plan(self): no_plan_log = test_data_path('test_is_test_passed-no_tests_no_plan.log') @@ -206,7 +198,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.subtests[0].subtests[0].status) - self.assertEqual(1, result.counts.errors) + self.assertEqual(result.counts, kunit_parser.TestCounts(passed=1, errors=1)) def test_no_kunit_output(self): @@ -218,6 +210,7 @@ class KUnitParserTest(unittest.TestCase): print_mock.assert_any_call(StrContains('could not find any KTAP output!')) print_mock.stop() self.assertEqual(0, len(result.subtests)) + self.assertEqual(result.counts.errors, 1) def test_skipped_test(self): skipped_log = test_data_path('test_skip_tests.log') @@ -225,18 +218,16 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests(file.readlines()) # A skipped test does not fail the whole suite. - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual(result.counts, kunit_parser.TestCounts(passed=4, skipped=1)) def test_skipped_all_tests(self): skipped_log = test_data_path('test_skip_all_tests.log') with open(skipped_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SKIPPED, - result.status) + self.assertEqual(kunit_parser.TestStatus.SKIPPED, result.status) + self.assertEqual(result.counts, kunit_parser.TestCounts(skipped=5)) def test_ignores_hyphen(self): hyphen_log = test_data_path('test_strip_hyphen.log') @@ -244,9 +235,7 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests(file.readlines()) # A skipped test does not fail the whole suite. - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) self.assertEqual( "sysctl_test", result.subtests[0].name) @@ -260,55 +249,49 @@ class KUnitParserTest(unittest.TestCase): prefix_log = test_data_path('test_config_printk_time.log') with open(prefix_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_ignores_multiple_prefixes(self): prefix_log = test_data_path('test_multiple_prefixes.log') with open(prefix_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_prefix_mixed_kernel_output(self): mixed_prefix_log = test_data_path('test_interrupted_tap_output.log') with open(mixed_prefix_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_prefix_poundsign(self): pound_log = test_data_path('test_pound_sign.log') with open(pound_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_kernel_panic_end(self): panic_log = test_data_path('test_kernel_panic_interrupt.log') with open(panic_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.TEST_CRASHED, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.TEST_CRASHED, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertGreaterEqual(result.counts.errors, 1) def test_pound_no_prefix(self): pound_log = test_data_path('test_pound_no_prefix.log') with open(pound_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_summarize_failures(self): output = """ From 101e32a025da386ba6f6efbfe3e75b6ec5a358aa Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Thu, 3 Nov 2022 10:47:40 -0700 Subject: [PATCH 11/29] kunit: tool: remove redundant file.close() call in unit test We're using a `with` block above, so the file object is already closed. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit_tool_test.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 5e3429a1202b..90c65b072be9 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -242,8 +242,6 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( "example", result.subtests[1].name) - file.close() - def test_ignores_prefix_printk_time(self): prefix_log = test_data_path('test_config_printk_time.log') From 697365c086791372945037557f99bc164e2db855 Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Fri, 30 Sep 2022 17:26:37 -0700 Subject: [PATCH 12/29] kunit: eliminate KUNIT_INIT_*_ASSERT_STRUCT macros These macros exist because passing an initializer list to other macros is hard. The goal of these macros is to generate a line like struct $ASSERT_TYPE __assertion = $APPROPRIATE_INITIALIZER; e.g. struct kunit_unary_assertion __assertion = { .condition = "foo()", .expected_true = true }; But the challenge is you can't pass `{.condition=..., .expect_true=...}` as a macro argument, since the comma means you're actually passing two arguments, `{.condition=...` and `.expect_true=....}`. So we'd made custom macros for each different initializer-list shape. But we can work around this with the following generic macro #define KUNIT_INIT_ASSERT(initializers...) { initializers } Note: this has the downside that we have to rename some macros arguments to not conflict with the struct field names (e.g. `expected_true`). It's a bit gross, but probably worth reducing the # of macros. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- include/kunit/assert.h | 48 ------------------------------------------ include/kunit/test.h | 27 +++++++++++++----------- 2 files changed, 15 insertions(+), 60 deletions(-) diff --git a/include/kunit/assert.h b/include/kunit/assert.h index e8a59487fd59..43144cfddc19 100644 --- a/include/kunit/assert.h +++ b/include/kunit/assert.h @@ -90,19 +90,6 @@ void kunit_unary_assert_format(const struct kunit_assert *assert, const struct va_format *message, struct string_stream *stream); -/** - * KUNIT_INIT_UNARY_ASSERT_STRUCT() - Initializes &struct kunit_unary_assert. - * @cond: A string representation of the expression asserted true or false. - * @expect_true: True if of type KUNIT_{EXPECT|ASSERT}_TRUE, false otherwise. - * - * Initializes a &struct kunit_unary_assert. Intended to be used in - * KUNIT_EXPECT_* and KUNIT_ASSERT_* macros. - */ -#define KUNIT_INIT_UNARY_ASSERT_STRUCT(cond, expect_true) { \ - .condition = cond, \ - .expected_true = expect_true \ -} - /** * struct kunit_ptr_not_err_assert - An expectation/assertion that a pointer is * not NULL and not a -errno. @@ -123,20 +110,6 @@ void kunit_ptr_not_err_assert_format(const struct kunit_assert *assert, const struct va_format *message, struct string_stream *stream); -/** - * KUNIT_INIT_PTR_NOT_ERR_ASSERT_STRUCT() - Initializes a - * &struct kunit_ptr_not_err_assert. - * @txt: A string representation of the expression passed to the expectation. - * @val: The actual evaluated pointer value of the expression. - * - * Initializes a &struct kunit_ptr_not_err_assert. Intended to be used in - * KUNIT_EXPECT_* and KUNIT_ASSERT_* macros. - */ -#define KUNIT_INIT_PTR_NOT_ERR_STRUCT(txt, val) { \ - .text = txt, \ - .value = val \ -} - /** * struct kunit_binary_assert_text - holds strings for &struct * kunit_binary_assert and friends to try and make the structs smaller. @@ -173,27 +146,6 @@ void kunit_binary_assert_format(const struct kunit_assert *assert, const struct va_format *message, struct string_stream *stream); -/** - * KUNIT_INIT_BINARY_ASSERT_STRUCT() - Initializes a binary assert like - * kunit_binary_assert, kunit_binary_ptr_assert, etc. - * - * @text_: Pointer to a kunit_binary_assert_text. - * @left_val: The actual evaluated value of the expression in the left slot. - * @right_val: The actual evaluated value of the expression in the right slot. - * - * Initializes a binary assert like kunit_binary_assert, - * kunit_binary_ptr_assert, etc. This relies on these structs having the same - * fields but with different types for left_val/right_val. - * This is ultimately used by binary assertion macros like KUNIT_EXPECT_EQ, etc. - */ -#define KUNIT_INIT_BINARY_ASSERT_STRUCT(text_, \ - left_val, \ - right_val) { \ - .text = text_, \ - .left_value = left_val, \ - .right_value = right_val \ -} - /** * struct kunit_binary_ptr_assert - An expectation/assertion that compares two * pointer values (for example, KUNIT_EXPECT_PTR_EQ(test, foo, bar)). diff --git a/include/kunit/test.h b/include/kunit/test.h index cde97dc4eed5..d7f60e8aab30 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -515,22 +515,25 @@ void kunit_do_failed_assertion(struct kunit *test, fmt, \ ##__VA_ARGS__) +/* Helper to safely pass around an initializer list to other macros. */ +#define KUNIT_INIT_ASSERT(initializers...) { initializers } + #define KUNIT_UNARY_ASSERTION(test, \ assert_type, \ - condition, \ - expected_true, \ + condition_, \ + expected_true_, \ fmt, \ ...) \ do { \ - if (likely(!!(condition) == !!expected_true)) \ + if (likely(!!(condition_) == !!expected_true_)) \ break; \ \ _KUNIT_FAILED(test, \ assert_type, \ kunit_unary_assert, \ kunit_unary_assert_format, \ - KUNIT_INIT_UNARY_ASSERT_STRUCT(#condition, \ - expected_true), \ + KUNIT_INIT_ASSERT(.condition = #condition_, \ + .expected_true = expected_true_), \ fmt, \ ##__VA_ARGS__); \ } while (0) @@ -590,9 +593,9 @@ do { \ assert_type, \ assert_class, \ format_func, \ - KUNIT_INIT_BINARY_ASSERT_STRUCT(&__text, \ - __left, \ - __right), \ + KUNIT_INIT_ASSERT(.text = &__text, \ + .left_value = __left, \ + .right_value = __right), \ fmt, \ ##__VA_ARGS__); \ } while (0) @@ -651,9 +654,9 @@ do { \ assert_type, \ kunit_binary_str_assert, \ kunit_binary_str_assert_format, \ - KUNIT_INIT_BINARY_ASSERT_STRUCT(&__text, \ - __left, \ - __right), \ + KUNIT_INIT_ASSERT(.text = &__text, \ + .left_value = __left, \ + .right_value = __right), \ fmt, \ ##__VA_ARGS__); \ } while (0) @@ -706,7 +709,7 @@ do { \ assert_type, \ kunit_ptr_not_err_assert, \ kunit_ptr_not_err_assert_format, \ - KUNIT_INIT_PTR_NOT_ERR_STRUCT(#ptr, __ptr), \ + KUNIT_INIT_ASSERT(.text = #ptr, .value = __ptr), \ fmt, \ ##__VA_ARGS__); \ } while (0) From 65c48a48ead042856525b92cedf673d2bf5bdfc9 Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Fri, 11 Nov 2022 10:29:04 -0800 Subject: [PATCH 13/29] Documentation: KUnit: make usage.rst a superset of tips.rst, remove duplication usage.rst had most of the content of the tips.rst page copied over. But it's missing https://www.kernel.org/doc/html/v6.0/dev-tools/kunit/tips.html#customizing-error-messages Copy it over so we can retire tips.rst w/o losing content. And in that process, it also gained a duplicate section about how KUNIT_ASSERT_*() exit the test case early. Remove that. Signed-off-by: Daniel Latypov Reviewed-by: Sadiya Kazi Reviewed-by: David Gow Signed-off-by: Shuah Khan --- Documentation/dev-tools/kunit/usage.rst | 49 ++++++++++++++++--------- 1 file changed, 31 insertions(+), 18 deletions(-) diff --git a/Documentation/dev-tools/kunit/usage.rst b/Documentation/dev-tools/kunit/usage.rst index 2737863ef365..b0a6c3bc0eeb 100644 --- a/Documentation/dev-tools/kunit/usage.rst +++ b/Documentation/dev-tools/kunit/usage.rst @@ -118,6 +118,37 @@ expectation could crash the test case. `ASSERT_NOT_ERR_OR_NULL(...)` allows us to bail out of the test case if the appropriate conditions are not satisfied to complete the test. +Customizing error messages +-------------------------- + +Each of the ``KUNIT_EXPECT`` and ``KUNIT_ASSERT`` macros have a ``_MSG`` +variant. These take a format string and arguments to provide additional +context to the automatically generated error messages. + +.. code-block:: c + + char some_str[41]; + generate_sha1_hex_string(some_str); + + /* Before. Not easy to tell why the test failed. */ + KUNIT_EXPECT_EQ(test, strlen(some_str), 40); + + /* After. Now we see the offending string. */ + KUNIT_EXPECT_EQ_MSG(test, strlen(some_str), 40, "some_str='%s'", some_str); + +Alternatively, one can take full control over the error message by using +``KUNIT_FAIL()``, e.g. + +.. code-block:: c + + /* Before */ + KUNIT_EXPECT_EQ(test, some_setup_function(), 0); + + /* After: full control over the failure message. */ + if (some_setup_function()) + KUNIT_FAIL(test, "Failed to setup thing for testing"); + + Test Suites ~~~~~~~~~~~ @@ -546,24 +577,6 @@ By reusing the same ``cases`` array from above, we can write the test as a {} }; -Exiting Early on Failed Expectations ------------------------------------- - -We can use ``KUNIT_EXPECT_EQ`` to mark the test as failed and continue -execution. In some cases, it is unsafe to continue. We can use the -``KUNIT_ASSERT`` variant to exit on failure. - -.. code-block:: c - - void example_test_user_alloc_function(struct kunit *test) - { - void *object = alloc_some_object_for_me(); - - /* Make sure we got a valid pointer back. */ - KUNIT_ASSERT_NOT_ERR_OR_NULL(test, object); - do_something_with_object(object); - } - Allocating Memory ----------------- From ec0a42a17e463ee5b1ebd2d60337e8ae8e5ace2b Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Fri, 11 Nov 2022 10:29:05 -0800 Subject: [PATCH 14/29] Documentation: KUnit: reword description of assertions The existing wording implies that kunit_kmalloc_array() is "the method under test". We're actually testing the sort() function in that example. This is because the example was changed in commit 953574390634 ("Documentation: KUnit: Rework writing page to focus on writing tests"), but the wording was not. Also add a `note` telling people they can use the KUNIT_ASSERT_EQ() macros from any function. Some users might be coming from a framework like gUnit where that'll compile but silently do the wrong thing. Signed-off-by: Daniel Latypov Reviewed-by: Sadiya Kazi Reviewed-by: David Gow Signed-off-by: Shuah Khan --- Documentation/dev-tools/kunit/usage.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/Documentation/dev-tools/kunit/usage.rst b/Documentation/dev-tools/kunit/usage.rst index b0a6c3bc0eeb..22416ebb94ab 100644 --- a/Documentation/dev-tools/kunit/usage.rst +++ b/Documentation/dev-tools/kunit/usage.rst @@ -112,11 +112,14 @@ terminates the test case if the condition is not satisfied. For example: KUNIT_EXPECT_LE(test, a[i], a[i + 1]); } -In this example, the method under test should return pointer to a value. If the -pointer returns null or an errno, we want to stop the test since the following -expectation could crash the test case. `ASSERT_NOT_ERR_OR_NULL(...)` allows us -to bail out of the test case if the appropriate conditions are not satisfied to -complete the test. +In this example, we need to be able to allocate an array to test the ``sort()`` +function. So we use ``KUNIT_ASSERT_NOT_ERR_OR_NULL()`` to abort the test if +there's an allocation error. + +.. note:: + In other test frameworks, ``ASSERT`` macros are often implemented by calling + ``return`` so they only work from the test function. In KUnit, we stop the + current kthread on failure, so you can call them from anywhere. Customizing error messages -------------------------- From a5b9abaa6049340a93a852b1d6d069064fddb624 Mon Sep 17 00:00:00 2001 From: David Gow Date: Fri, 11 Nov 2022 10:29:06 -0800 Subject: [PATCH 15/29] Documentation: kunit: Remove redundant 'tips.rst' page The contents of 'tips.rst' was mostly included in 'usage.rst' way back in commit 953574390634 ("Documentation: KUnit: Rework writing page to focus on writing tests"), but the tips page remained behind as well. The parent patches in this series fill in the gaps, so now 'tips.rst' is redundant. Therefore, delete 'tips.rst'. While I regret breaking any links to 'tips' which might exist externally, it's confusing to have two subtly different versions of the same content around. Signed-off-by: David Gow Signed-off-by: Daniel Latypov Reviewed-by: Sadiya Kazi Signed-off-by: Shuah Khan --- Documentation/dev-tools/kunit/index.rst | 1 - Documentation/dev-tools/kunit/tips.rst | 190 ------------------------ 2 files changed, 191 deletions(-) delete mode 100644 Documentation/dev-tools/kunit/tips.rst diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst index f5d13f1d37be..d5629817cd72 100644 --- a/Documentation/dev-tools/kunit/index.rst +++ b/Documentation/dev-tools/kunit/index.rst @@ -16,7 +16,6 @@ KUnit - Linux Kernel Unit Testing api/index style faq - tips running_tips This section details the kernel unit testing framework. diff --git a/Documentation/dev-tools/kunit/tips.rst b/Documentation/dev-tools/kunit/tips.rst deleted file mode 100644 index 492d2ded2f5a..000000000000 --- a/Documentation/dev-tools/kunit/tips.rst +++ /dev/null @@ -1,190 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -============================ -Tips For Writing KUnit Tests -============================ - -Exiting early on failed expectations ------------------------------------- - -``KUNIT_EXPECT_EQ`` and friends will mark the test as failed and continue -execution. In some cases, it's unsafe to continue and you can use the -``KUNIT_ASSERT`` variant to exit on failure. - -.. code-block:: c - - void example_test_user_alloc_function(struct kunit *test) - { - void *object = alloc_some_object_for_me(); - - /* Make sure we got a valid pointer back. */ - KUNIT_ASSERT_NOT_ERR_OR_NULL(test, object); - do_something_with_object(object); - } - -Allocating memory ------------------ - -Where you would use ``kzalloc``, you should prefer ``kunit_kzalloc`` instead. -KUnit will ensure the memory is freed once the test completes. - -This is particularly useful since it lets you use the ``KUNIT_ASSERT_EQ`` -macros to exit early from a test without having to worry about remembering to -call ``kfree``. - -Example: - -.. code-block:: c - - void example_test_allocation(struct kunit *test) - { - char *buffer = kunit_kzalloc(test, 16, GFP_KERNEL); - /* Ensure allocation succeeded. */ - KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buffer); - - KUNIT_ASSERT_STREQ(test, buffer, ""); - } - - -Testing static functions ------------------------- - -If you don't want to expose functions or variables just for testing, one option -is to conditionally ``#include`` the test file at the end of your .c file, e.g. - -.. code-block:: c - - /* In my_file.c */ - - static int do_interesting_thing(); - - #ifdef CONFIG_MY_KUNIT_TEST - #include "my_kunit_test.c" - #endif - -Injecting test-only code ------------------------- - -Similarly to the above, it can be useful to add test-specific logic. - -.. code-block:: c - - /* In my_file.h */ - - #ifdef CONFIG_MY_KUNIT_TEST - /* Defined in my_kunit_test.c */ - void test_only_hook(void); - #else - void test_only_hook(void) { } - #endif - -This test-only code can be made more useful by accessing the current kunit -test, see below. - -Accessing the current test --------------------------- - -In some cases, you need to call test-only code from outside the test file, e.g. -like in the example above or if you're providing a fake implementation of an -ops struct. -There is a ``kunit_test`` field in ``task_struct``, so you can access it via -``current->kunit_test``. - -Here's a slightly in-depth example of how one could implement "mocking": - -.. code-block:: c - - #include /* for current */ - - struct test_data { - int foo_result; - int want_foo_called_with; - }; - - static int fake_foo(int arg) - { - struct kunit *test = current->kunit_test; - struct test_data *test_data = test->priv; - - KUNIT_EXPECT_EQ(test, test_data->want_foo_called_with, arg); - return test_data->foo_result; - } - - static void example_simple_test(struct kunit *test) - { - /* Assume priv is allocated in the suite's .init */ - struct test_data *test_data = test->priv; - - test_data->foo_result = 42; - test_data->want_foo_called_with = 1; - - /* In a real test, we'd probably pass a pointer to fake_foo somewhere - * like an ops struct, etc. instead of calling it directly. */ - KUNIT_EXPECT_EQ(test, fake_foo(1), 42); - } - - -Note: here we're able to get away with using ``test->priv``, but if you wanted -something more flexible you could use a named ``kunit_resource``, see -Documentation/dev-tools/kunit/api/test.rst. - -Failing the current test ------------------------- - -But sometimes, you might just want to fail the current test. In that case, we -have ``kunit_fail_current_test(fmt, args...)`` which is defined in ```` and -doesn't require pulling in ````. - -E.g. say we had an option to enable some extra debug checks on some data structure: - -.. code-block:: c - - #include - - #ifdef CONFIG_EXTRA_DEBUG_CHECKS - static void validate_my_data(struct data *data) - { - if (is_valid(data)) - return; - - kunit_fail_current_test("data %p is invalid", data); - - /* Normal, non-KUnit, error reporting code here. */ - } - #else - static void my_debug_function(void) { } - #endif - - -Customizing error messages --------------------------- - -Each of the ``KUNIT_EXPECT`` and ``KUNIT_ASSERT`` macros have a ``_MSG`` variant. -These take a format string and arguments to provide additional context to the automatically generated error messages. - -.. code-block:: c - - char some_str[41]; - generate_sha1_hex_string(some_str); - - /* Before. Not easy to tell why the test failed. */ - KUNIT_EXPECT_EQ(test, strlen(some_str), 40); - - /* After. Now we see the offending string. */ - KUNIT_EXPECT_EQ_MSG(test, strlen(some_str), 40, "some_str='%s'", some_str); - -Alternatively, one can take full control over the error message by using ``KUNIT_FAIL()``, e.g. - -.. code-block:: c - - /* Before */ - KUNIT_EXPECT_EQ(test, some_setup_function(), 0); - - /* After: full control over the failure message. */ - if (some_setup_function()) - KUNIT_FAIL(test, "Failed to setup thing for testing"); - -Next Steps -========== -* Optional: see the Documentation/dev-tools/kunit/usage.rst page for a more - in-depth explanation of KUnit. From 34c68f432c67f0d9bd4e64cf0929f399c6a4e1b0 Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Wed, 9 Nov 2022 13:20:32 -0800 Subject: [PATCH 16/29] kunit: remove KUNIT_INIT_MEM_ASSERTION macro Commit 870f63b7cd78 ("kunit: eliminate KUNIT_INIT_*_ASSERT_STRUCT macros") removed all the other macros of this type. But it raced with commit b8a926bea8b1 ("kunit: Introduce KUNIT_EXPECT_MEMEQ and KUNIT_EXPECT_MEMNEQ macros"), which added another instance. Remove KUNIT_INIT_MEM_ASSERTION and just use the generic KUNIT_INIT_ASSERT macro instead. Rename the `size` arg to avoid conflicts by appending a "_" (like we did in the previous commit). Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- include/kunit/assert.h | 7 ------- include/kunit/test.h | 12 ++++++------ 2 files changed, 6 insertions(+), 13 deletions(-) diff --git a/include/kunit/assert.h b/include/kunit/assert.h index 43144cfddc19..24c2b9fa61e8 100644 --- a/include/kunit/assert.h +++ b/include/kunit/assert.h @@ -192,13 +192,6 @@ void kunit_binary_str_assert_format(const struct kunit_assert *assert, const struct va_format *message, struct string_stream *stream); -#define KUNIT_INIT_MEM_ASSERT_STRUCT(text_, left_val, right_val, size_) { \ - .text = text_, \ - .left_value = left_val, \ - .right_value = right_val, \ - .size = size_ \ -} - /** * struct kunit_mem_assert - An expectation/assertion that compares two * memory blocks. diff --git a/include/kunit/test.h b/include/kunit/test.h index d7f60e8aab30..4666a4d199ea 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -666,13 +666,13 @@ do { \ left, \ op, \ right, \ - size, \ + size_, \ fmt, \ ...) \ do { \ const void *__left = (left); \ const void *__right = (right); \ - const size_t __size = (size); \ + const size_t __size = (size_); \ static const struct kunit_binary_assert_text __text = { \ .operation = #op, \ .left_text = #left, \ @@ -686,10 +686,10 @@ do { \ assert_type, \ kunit_mem_assert, \ kunit_mem_assert_format, \ - KUNIT_INIT_MEM_ASSERT_STRUCT(&__text, \ - __left, \ - __right, \ - __size), \ + KUNIT_INIT_ASSERT(.text = &__text, \ + .left_value = __left, \ + .right_value = __right, \ + .size = __size), \ fmt, \ ##__VA_ARGS__); \ } while (0) From 0a7d5c30b7f02887319a1382fbb8dc1c8250fe2c Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Thu, 10 Nov 2022 19:18:55 -0800 Subject: [PATCH 17/29] kunit: tool: tweak error message when no KTAP found We currently tell people we "couldn't find any KTAP output" with no indication as to what this might mean. After this patch, we get: $ ./tools/testing/kunit/kunit.py parse /dev/null ============================================================ [ERROR] Test: : Could not find any KTAP output. Did any KUnit tests run? ============================================================ Testing complete. Ran 0 tests: errors: 1 Note: we could try and generate a more verbose message like > Please check .kunit/test.log to see the raw kernel output. or the like, but we'd need to know what the build dir was to know where test.log actually lives. This patch tries to make a more minimal improvement. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit_parser.py | 2 +- tools/testing/kunit/kunit_tool_test.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index a56c75a973b5..d0ed5dd5cfc4 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -782,7 +782,7 @@ def parse_run_tests(kernel_output: Iterable[str]) -> Test: test = Test() if not lines: test.name = '' - test.add_error('could not find any KTAP output!') + test.add_error('Could not find any KTAP output. Did any KUnit tests run?') test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: test = parse_test(lines, 0, []) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 90c65b072be9..84a08cf07242 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -207,7 +207,7 @@ class KUnitParserTest(unittest.TestCase): with open(crash_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - print_mock.assert_any_call(StrContains('could not find any KTAP output!')) + print_mock.assert_any_call(StrContains('Could not find any KTAP output.')) print_mock.stop() self.assertEqual(0, len(result.subtests)) self.assertEqual(result.counts.errors, 1) @@ -588,7 +588,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(e.exception.code, 1) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) - self.print_mock.assert_any_call(StrContains('could not find any KTAP output!')) + self.print_mock.assert_any_call(StrContains('Could not find any KTAP output.')) def test_exec_no_tests(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=['TAP version 14', '1..0']) From 309e22effb741a8c65131a2694a49839fd685a27 Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Mon, 21 Nov 2022 11:55:26 -0800 Subject: [PATCH 18/29] kunit: tool: make --json do nothing if --raw_ouput is set When --raw_output is set (to any value), we don't actually parse the test results. So asking to print the test results as json doesn't make sense. We internally create a fake test with one passing subtest, so --json would actually print out something misleading. This patch: * Rewords the flag descriptions so hopefully this is more obvious. * Also updates --raw_output's description to note the default behavior is to print out only "KUnit" results (actually any KTAP results) * also renames and refactors some related logic for clarity (e.g. test_result => test, it's a kunit_parser.Test object). Notably, this patch does not make it an error to specify --json and --raw_output together. This is an edge case, but I know of at least one wrapper around kunit.py that always sets --json. You'd never be able to use --raw_output with that wrapper. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit.py | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 4d4663fb578b..e7b6549712d6 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -192,12 +192,11 @@ def _map_to_overall_status(test_status: kunit_parser.TestStatus) -> KunitStatus: def parse_tests(request: KunitParseRequest, metadata: kunit_json.Metadata, input_data: Iterable[str]) -> Tuple[KunitResult, kunit_parser.Test]: parse_start = time.time() - test_result = kunit_parser.Test() - if request.raw_output: # Treat unparsed results as one passing test. - test_result.status = kunit_parser.TestStatus.SUCCESS - test_result.counts.passed = 1 + fake_test = kunit_parser.Test() + fake_test.status = kunit_parser.TestStatus.SUCCESS + fake_test.counts.passed = 1 output: Iterable[str] = input_data if request.raw_output == 'all': @@ -206,14 +205,17 @@ def parse_tests(request: KunitParseRequest, metadata: kunit_json.Metadata, input output = kunit_parser.extract_tap_lines(output, lstrip=False) for line in output: print(line.rstrip()) + parse_time = time.time() - parse_start + return KunitResult(KunitStatus.SUCCESS, parse_time), fake_test - else: - test_result = kunit_parser.parse_run_tests(input_data) - parse_end = time.time() + + # Actually parse the test results. + test = kunit_parser.parse_run_tests(input_data) + parse_time = time.time() - parse_start if request.json: json_str = kunit_json.get_json_result( - test=test_result, + test=test, metadata=metadata) if request.json == 'stdout': print(json_str) @@ -223,10 +225,10 @@ def parse_tests(request: KunitParseRequest, metadata: kunit_json.Metadata, input stdout.print_with_timestamp("Test results stored in %s" % os.path.abspath(request.json)) - if test_result.status != kunit_parser.TestStatus.SUCCESS: - return KunitResult(KunitStatus.TEST_FAILURE, parse_end - parse_start), test_result + if test.status != kunit_parser.TestStatus.SUCCESS: + return KunitResult(KunitStatus.TEST_FAILURE, parse_time), test - return KunitResult(KunitStatus.SUCCESS, parse_end - parse_start), test_result + return KunitResult(KunitStatus.SUCCESS, parse_time), test def run_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitRequest) -> KunitResult: @@ -359,14 +361,14 @@ def add_exec_opts(parser) -> None: choices=['suite', 'test']) def add_parse_opts(parser) -> None: - parser.add_argument('--raw_output', help='If set don\'t format output from kernel. ' - 'If set to --raw_output=kunit, filters to just KUnit output.', + parser.add_argument('--raw_output', help='If set don\'t parse output from kernel. ' + 'By default, filters to just KUnit output. Use ' + '--raw_output=all to show everything', type=str, nargs='?', const='all', default=None, choices=['all', 'kunit']) parser.add_argument('--json', nargs='?', - help='Stores test results in a JSON, and either ' - 'prints to stdout or saves to file if a ' - 'filename is specified', + help='Prints parsed test results as JSON to stdout or a file if ' + 'a filename is specified. Does nothing if --raw_output is set.', type=str, const='stdout', default=None, metavar='FILE') From 908d0c177bbc7c34ab9129c6f2bcd87487115632 Mon Sep 17 00:00:00 2001 From: David Gow Date: Fri, 25 Nov 2022 16:43:04 +0800 Subject: [PATCH 19/29] kunit: Provide a static key to check if KUnit is actively running tests KUnit does a few expensive things when enabled. This hasn't been a problem because KUnit was only enabled on test kernels, but with a few people enabling (but not _using_) KUnit on production systems, we need a runtime way of handling this. Provide a 'kunit_running' static key (defaulting to false), which allows us to hide any KUnit code behind a static branch. This should reduce the performance impact (on other code) of having KUnit enabled to a single NOP when no tests are running. Note that, while it looks unintuitive, tests always run entirely within __kunit_test_suites_init(), so it's safe to decrement the static key at the end of this function, rather than in __kunit_test_suites_exit(), which is only there to clean up results in debugfs. Signed-off-by: David Gow Reviewed-by: Daniel Latypov Signed-off-by: Shuah Khan --- include/kunit/test.h | 4 ++++ lib/kunit/test.c | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/include/kunit/test.h b/include/kunit/test.h index 4666a4d199ea..87ea90576b50 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -27,6 +28,9 @@ #include +/* Static key: true if any KUnit tests are currently running */ +DECLARE_STATIC_KEY_FALSE(kunit_running); + struct kunit; /* Size of log associated with test. */ diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 90640a43cf62..314717b63080 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -20,6 +20,8 @@ #include "string-stream.h" #include "try-catch-impl.h" +DEFINE_STATIC_KEY_FALSE(kunit_running); + #if IS_BUILTIN(CONFIG_KUNIT) /* * Fail the current test and print an error message to the log. @@ -612,10 +614,14 @@ int __kunit_test_suites_init(struct kunit_suite * const * const suites, int num_ return 0; } + static_branch_inc(&kunit_running); + for (i = 0; i < num_suites; i++) { kunit_init_suite(suites[i]); kunit_run_tests(suites[i]); } + + static_branch_dec(&kunit_running); return 0; } EXPORT_SYMBOL_GPL(__kunit_test_suites_init); From 91e93592219f74c4d5cd4f27006d726ac86ae15d Mon Sep 17 00:00:00 2001 From: David Gow Date: Fri, 25 Nov 2022 16:43:05 +0800 Subject: [PATCH 20/29] kunit: Use the static key when retrieving the current test In order to detect if a KUnit test is running, and to access its context, the 'kunit_test' member of the current task_struct is used. Usually, this is accessed directly or via the kunit_fail_current_task() function. In order to speed up the case where no test is running, add a wrapper, kunit_get_current_test(), which uses the static key to fail early. Equally, Speed up kunit_fail_current_test() by using the static key. This should make it convenient for code to call this unconditionally in fakes or error paths, without worrying that this will slow the code down significantly. If CONFIG_KUNIT=n (or m), this compiles away to nothing. If CONFIG_KUNIT=y, it will compile down to a NOP (on most architectures) if no KUnit test is currently running. Note that kunit_get_current_test() does not work if KUnit is built as a module. This mirrors the existing restriction on kunit_fail_current_test(). Note that the definition of kunit_fail_current_test() still wraps an empty, inline function if KUnit is not built-in. This is to ensure that the printf format string __attribute__ will still work. Also update the documentation to suggest users use the new kunit_get_current_test() function, update the example, and to describe the behaviour when KUnit is disabled better. Cc: Jonathan Corbet Cc: Sadiya Kazi Signed-off-by: David Gow Reviewed-by: Daniel Latypov Signed-off-by: Shuah Khan --- Documentation/dev-tools/kunit/usage.rst | 30 +++++++++----- include/kunit/test-bug.h | 53 +++++++++++++++++++++++-- 2 files changed, 71 insertions(+), 12 deletions(-) diff --git a/Documentation/dev-tools/kunit/usage.rst b/Documentation/dev-tools/kunit/usage.rst index 22416ebb94ab..48f8196d5aad 100644 --- a/Documentation/dev-tools/kunit/usage.rst +++ b/Documentation/dev-tools/kunit/usage.rst @@ -641,17 +641,23 @@ as shown in next section: *Accessing The Current Test*. Accessing The Current Test -------------------------- -In some cases, we need to call test-only code from outside the test file. -For example, see example in section *Injecting Test-Only Code* or if -we are providing a fake implementation of an ops struct. Using -``kunit_test`` field in ``task_struct``, we can access it via -``current->kunit_test``. +In some cases, we need to call test-only code from outside the test file. This +is helpful, for example, when providing a fake implementation of a function, or +to fail any current test from within an error handler. +We can do this via the ``kunit_test`` field in ``task_struct``, which we can +access using the ``kunit_get_current_test()`` function in ``kunit/test-bug.h``. -The example below includes how to implement "mocking": +``kunit_get_current_test()`` is safe to call even if KUnit is not enabled. If +KUnit is not enabled, was built as a module (``CONFIG_KUNIT=m``), or no test is +running in the current task, it will return ``NULL``. This compiles down to +either a no-op or a static key check, so will have a negligible performance +impact when no test is running. + +The example below uses this to implement a "mock" implementation of a function, ``foo``: .. code-block:: c - #include /* for current */ + #include /* for kunit_get_current_test */ struct test_data { int foo_result; @@ -660,7 +666,7 @@ The example below includes how to implement "mocking": static int fake_foo(int arg) { - struct kunit *test = current->kunit_test; + struct kunit *test = kunit_get_current_test(); struct test_data *test_data = test->priv; KUNIT_EXPECT_EQ(test, test_data->want_foo_called_with, arg); @@ -691,7 +697,7 @@ Each test can have multiple resources which have string names providing the same flexibility as a ``priv`` member, but also, for example, allowing helper functions to create resources without conflicting with each other. It is also possible to define a clean up function for each resource, making it easy to -avoid resource leaks. For more information, see Documentation/dev-tools/kunit/api/test.rst. +avoid resource leaks. For more information, see Documentation/dev-tools/kunit/api/resource.rst. Failing The Current Test ------------------------ @@ -719,3 +725,9 @@ structures as shown below: static void my_debug_function(void) { } #endif +``kunit_fail_current_test()`` is safe to call even if KUnit is not enabled. If +KUnit is not enabled, was built as a module (``CONFIG_KUNIT=m``), or no test is +running in the current task, it will do nothing. This compiles down to either a +no-op or a static key check, so will have a negligible performance impact when +no test is running. + diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h index 5fc58081d511..c1b2e14eab64 100644 --- a/include/kunit/test-bug.h +++ b/include/kunit/test-bug.h @@ -9,16 +9,63 @@ #ifndef _KUNIT_TEST_BUG_H #define _KUNIT_TEST_BUG_H -#define kunit_fail_current_test(fmt, ...) \ - __kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__) - #if IS_BUILTIN(CONFIG_KUNIT) +#include /* For static branch */ +#include + +/* Static key if KUnit is running any tests. */ +DECLARE_STATIC_KEY_FALSE(kunit_running); + +/** + * kunit_get_current_test() - Return a pointer to the currently running + * KUnit test. + * + * If a KUnit test is running in the current task, returns a pointer to its + * associated struct kunit. This pointer can then be passed to any KUnit + * function or assertion. If no test is running (or a test is running in a + * different task), returns NULL. + * + * This function is safe to call even when KUnit is disabled. If CONFIG_KUNIT + * is not enabled, it will compile down to nothing and will return quickly no + * test is running. + */ +static inline struct kunit *kunit_get_current_test(void) +{ + if (!static_branch_unlikely(&kunit_running)) + return NULL; + + return current->kunit_test; +} + + +/** + * kunit_fail_current_test() - If a KUnit test is running, fail it. + * + * If a KUnit test is running in the current task, mark that test as failed. + * + * This macro will only work if KUnit is built-in (though the tests + * themselves can be modules). Otherwise, it compiles down to nothing. + */ +#define kunit_fail_current_test(fmt, ...) do { \ + if (static_branch_unlikely(&kunit_running)) { \ + __kunit_fail_current_test(__FILE__, __LINE__, \ + fmt, ##__VA_ARGS__); \ + } \ + } while (0) + + extern __printf(3, 4) void __kunit_fail_current_test(const char *file, int line, const char *fmt, ...); #else +static inline struct kunit *kunit_get_current_test(void) { return NULL; } + +/* We define this with an empty helper function so format string warnings work */ +#define kunit_fail_current_test(fmt, ...) \ + __kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__) + static inline __printf(3, 4) void __kunit_fail_current_test(const char *file, int line, const char *fmt, ...) { From 909c6475d568826be377893cf5abb7cde5877230 Mon Sep 17 00:00:00 2001 From: David Gow Date: Fri, 25 Nov 2022 16:43:06 +0800 Subject: [PATCH 21/29] mm: slub: test: Use the kunit_get_current_test() function Use the newly-added function kunit_get_current_test() instead of accessing current->kunit_test directly. This function uses a static key to return more quickly when KUnit is enabled, but no tests are actively running. There should therefore be a negligible performance impact to enabling the slub KUnit tests. Other than the performance improvement, this should be a no-op. Cc: Oliver Glitta Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: Christoph Lameter Cc: Vlastimil Babka Cc: David Rientjes Cc: Andrew Morton Signed-off-by: David Gow Acked-by: Vlastimil Babka Acked-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Shuah Khan --- lib/slub_kunit.c | 1 + mm/slub.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/slub_kunit.c b/lib/slub_kunit.c index 7a0564d7cb7a..8fd19c8301ad 100644 --- a/lib/slub_kunit.c +++ b/lib/slub_kunit.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include #include #include diff --git a/mm/slub.c b/mm/slub.c index 157527d7101b..1887996cb703 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -39,6 +39,7 @@ #include #include #include +#include #include #include @@ -603,7 +604,7 @@ static bool slab_add_kunit_errors(void) { struct kunit_resource *resource; - if (likely(!current->kunit_test)) + if (!kunit_get_current_test()) return false; resource = kunit_find_named_resource(current->kunit_test, "slab_errors"); From 434498a6bee3db729dbdb7f131f3506f4dca85e8 Mon Sep 17 00:00:00 2001 From: Rae Moar Date: Wed, 23 Nov 2022 18:25:57 +0000 Subject: [PATCH 22/29] kunit: tool: parse KTAP compliant test output Change the KUnit parser to be able to parse test output that complies with the KTAP version 1 specification format found here: https://kernel.org/doc/html/latest/dev-tools/ktap.html. Ensure the parser is able to parse tests with the original KUnit test output format as well. KUnit parser now accepts any of the following test output formats: Original KUnit test output format: TAP version 14 1..1 # Subtest: kunit-test-suite 1..3 ok 1 - kunit_test_1 ok 2 - kunit_test_2 ok 3 - kunit_test_3 # kunit-test-suite: pass:3 fail:0 skip:0 total:3 # Totals: pass:3 fail:0 skip:0 total:3 ok 1 - kunit-test-suite KTAP version 1 test output format: KTAP version 1 1..1 KTAP version 1 1..3 ok 1 kunit_test_1 ok 2 kunit_test_2 ok 3 kunit_test_3 ok 1 kunit-test-suite New KUnit test output format (changes made in the next patch of this series): KTAP version 1 1..1 KTAP version 1 # Subtest: kunit-test-suite 1..3 ok 1 kunit_test_1 ok 2 kunit_test_2 ok 3 kunit_test_3 # kunit-test-suite: pass:3 fail:0 skip:0 total:3 # Totals: pass:3 fail:0 skip:0 total:3 ok 1 kunit-test-suite Signed-off-by: Rae Moar Reviewed-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit_parser.py | 79 ++++++++++++------- tools/testing/kunit/kunit_tool_test.py | 14 ++++ .../test_data/test_parse_ktap_output.log | 8 ++ .../test_data/test_parse_subtest_header.log | 7 ++ 4 files changed, 80 insertions(+), 28 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_parse_ktap_output.log create mode 100644 tools/testing/kunit/test_data/test_parse_subtest_header.log diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index d0ed5dd5cfc4..4cc2f8b7ecd0 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -441,6 +441,7 @@ def parse_diagnostic(lines: LineStream) -> List[str]: - '# Subtest: [test name]' - '[ok|not ok] [test number] [-] [test name] [optional skip directive]' + - 'KTAP version [version number]' Parameters: lines - LineStream of KTAP output to parse @@ -449,8 +450,9 @@ def parse_diagnostic(lines: LineStream) -> List[str]: Log of diagnostic lines """ log = [] # type: List[str] - while lines and not TEST_RESULT.match(lines.peek()) and not \ - TEST_HEADER.match(lines.peek()): + non_diagnostic_lines = [TEST_RESULT, TEST_HEADER, KTAP_START] + while lines and not any(re.match(lines.peek()) + for re in non_diagnostic_lines): log.append(lines.pop()) return log @@ -496,11 +498,15 @@ def print_test_header(test: Test) -> None: test - Test object representing current test being printed """ message = test.name + if message != "": + # Add a leading space before the subtest counts only if a test name + # is provided using a "# Subtest" header line. + message += " " if test.expected_count: if test.expected_count == 1: - message += ' (1 subtest)' + message += '(1 subtest)' else: - message += f' ({test.expected_count} subtests)' + message += f'({test.expected_count} subtests)' stdout.print_with_timestamp(format_test_divider(message, len(message))) def print_log(log: Iterable[str]) -> None: @@ -647,7 +653,7 @@ def bubble_up_test_results(test: Test) -> None: elif test.counts.get_status() == TestStatus.TEST_CRASHED: test.status = TestStatus.TEST_CRASHED -def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: +def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool) -> Test: """ Finds next test to parse in LineStream, creates new Test object, parses any subtests of the test, populates Test object with all @@ -665,15 +671,32 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: 1..4 [subtests] - - Subtest header line + - Subtest header (must include either the KTAP version line or + "# Subtest" header line) - Example: + Example (preferred format with both KTAP version line and + "# Subtest" line): + + KTAP version 1 + # Subtest: name + 1..3 + [subtests] + ok 1 name + + Example (only "# Subtest" line): # Subtest: name 1..3 [subtests] ok 1 name + Example (only KTAP version line, compliant with KTAP v1 spec): + + KTAP version 1 + 1..3 + [subtests] + ok 1 name + - Test result line Example: @@ -685,28 +708,29 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: expected_num - expected test number for test to be parsed log - list of strings containing any preceding diagnostic lines corresponding to the current test + is_subtest - boolean indicating whether test is a subtest Return: Test object populated with characteristics and any subtests """ test = Test() test.log.extend(log) - parent_test = False - main = parse_ktap_header(lines, test) - if main: - # If KTAP/TAP header is found, attempt to parse + if not is_subtest: + # If parsing the main/top-level test, parse KTAP version line and # test plan test.name = "main" + ktap_line = parse_ktap_header(lines, test) parse_test_plan(lines, test) parent_test = True else: - # If KTAP/TAP header is not found, test must be subtest - # header or test result line so parse attempt to parser - # subtest header - parent_test = parse_test_header(lines, test) + # If not the main test, attempt to parse a test header containing + # the KTAP version line and/or subtest header line + ktap_line = parse_ktap_header(lines, test) + subtest_line = parse_test_header(lines, test) + parent_test = (ktap_line or subtest_line) if parent_test: - # If subtest header is found, attempt to parse - # test plan and print header + # If KTAP version line and/or subtest header is found, attempt + # to parse test plan and print test header parse_test_plan(lines, test) print_test_header(test) expected_count = test.expected_count @@ -721,7 +745,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: sub_log = parse_diagnostic(lines) sub_test = Test() if not lines or (peek_test_name_match(lines, test) and - not main): + is_subtest): if expected_count and test_num <= expected_count: # If parser reaches end of test before # parsing expected number of subtests, print @@ -735,20 +759,19 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: test.log.extend(sub_log) break else: - sub_test = parse_test(lines, test_num, sub_log) + sub_test = parse_test(lines, test_num, sub_log, True) subtests.append(sub_test) test_num += 1 test.subtests = subtests - if not main: + if is_subtest: # If not main test, look for test result line test.log.extend(parse_diagnostic(lines)) - if (parent_test and peek_test_name_match(lines, test)) or \ - not parent_test: - parse_test_result(lines, test, expected_num) - else: + if test.name != "" and not peek_test_name_match(lines, test): test.add_error('missing subtest result line!') + else: + parse_test_result(lines, test, expected_num) - # Check for there being no tests + # Check for there being no subtests within parent test if parent_test and len(subtests) == 0: # Don't override a bad status if this test had one reported. # Assumption: no subtests means CRASHED is from Test.__init__() @@ -758,11 +781,11 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: # Add statuses to TestCounts attribute in Test object bubble_up_test_results(test) - if parent_test and not main: + if parent_test and is_subtest: # If test has subtests and is not the main test object, print # footer. print_test_footer(test) - elif not main: + elif is_subtest: print_test_result(test) return test @@ -785,7 +808,7 @@ def parse_run_tests(kernel_output: Iterable[str]) -> Test: test.add_error('Could not find any KTAP output. Did any KUnit tests run?') test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: - test = parse_test(lines, 0, []) + test = parse_test(lines, 0, [], False) if test.status != TestStatus.NO_TESTS: test.status = test.counts.get_status() stdout.print_with_timestamp(DIVIDER) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 84a08cf07242..d7f669cbf2a8 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -312,6 +312,20 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual(kunit_parser._summarize_failed_tests(result), 'Failures: all_failed_suite, some_failed_suite.test2') + def test_ktap_format(self): + ktap_log = test_data_path('test_parse_ktap_output.log') + with open(ktap_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual(result.counts, kunit_parser.TestCounts(passed=3)) + self.assertEqual('suite', result.subtests[0].name) + self.assertEqual('case_1', result.subtests[0].subtests[0].name) + self.assertEqual('case_2', result.subtests[0].subtests[1].name) + + def test_parse_subtest_header(self): + ktap_log = test_data_path('test_parse_subtest_header.log') + with open(ktap_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.print_mock.assert_any_call(StrContains('suite (1 subtest)')) def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: return kunit_parser.LineStream(enumerate(strs, start=1)) diff --git a/tools/testing/kunit/test_data/test_parse_ktap_output.log b/tools/testing/kunit/test_data/test_parse_ktap_output.log new file mode 100644 index 000000000000..ccdf244e5303 --- /dev/null +++ b/tools/testing/kunit/test_data/test_parse_ktap_output.log @@ -0,0 +1,8 @@ +KTAP version 1 +1..1 + KTAP version 1 + 1..3 + ok 1 case_1 + ok 2 case_2 + ok 3 case_3 +ok 1 suite diff --git a/tools/testing/kunit/test_data/test_parse_subtest_header.log b/tools/testing/kunit/test_data/test_parse_subtest_header.log new file mode 100644 index 000000000000..216631092e7b --- /dev/null +++ b/tools/testing/kunit/test_data/test_parse_subtest_header.log @@ -0,0 +1,7 @@ +KTAP version 1 +1..1 + KTAP version 1 + # Subtest: suite + 1..1 + ok 1 test +ok 1 suite \ No newline at end of file From 6c738b52316c58ae8a87abf0907f87a7b5e7a109 Mon Sep 17 00:00:00 2001 From: Rae Moar Date: Wed, 23 Nov 2022 18:25:58 +0000 Subject: [PATCH 23/29] kunit: improve KTAP compliance of KUnit test output MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Change KUnit test output to better comply with KTAP v1 specifications found here: https://kernel.org/doc/html/latest/dev-tools/ktap.html. 1) Use "KTAP version 1" instead of "TAP version 14" as test output header 2) Remove '-' between test number and test name on test result lines 2) Add KTAP version lines to each subtest header as well Note that the new KUnit output still includes the “# Subtest” line now located after the KTAP version line. This does not completely match the KTAP v1 spec but since it is classified as a diagnostic line, it is not expected to be disruptive or break any existing parsers. This “# Subtest” line comes from the TAP 14 spec (https://testanything.org/tap-version-14-specification.html) and it is used to define the test name before the results. Original output: TAP version 14 1..1 # Subtest: kunit-test-suite 1..3 ok 1 - kunit_test_1 ok 2 - kunit_test_2 ok 3 - kunit_test_3 # kunit-test-suite: pass:3 fail:0 skip:0 total:3 # Totals: pass:3 fail:0 skip:0 total:3 ok 1 - kunit-test-suite New output: KTAP version 1 1..1 KTAP version 1 # Subtest: kunit-test-suite 1..3 ok 1 kunit_test_1 ok 2 kunit_test_2 ok 3 kunit_test_3 # kunit-test-suite: pass:3 fail:0 skip:0 total:3 # Totals: pass:3 fail:0 skip:0 total:3 ok 1 kunit-test-suite Signed-off-by: Rae Moar Reviewed-by: Daniel Latypov Reviewed-by: David Gow Tested-by: Anders Roxell Signed-off-by: Shuah Khan --- lib/kunit/debugfs.c | 2 +- lib/kunit/executor.c | 6 +++--- lib/kunit/test.c | 9 ++++++--- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/lib/kunit/debugfs.c b/lib/kunit/debugfs.c index 1048ef1b8d6e..de0ee2e03ed6 100644 --- a/lib/kunit/debugfs.c +++ b/lib/kunit/debugfs.c @@ -63,7 +63,7 @@ static int debugfs_print_results(struct seq_file *seq, void *v) kunit_suite_for_each_test_case(suite, test_case) debugfs_print_result(seq, suite, test_case); - seq_printf(seq, "%s %d - %s\n", + seq_printf(seq, "%s %d %s\n", kunit_status_to_ok_not_ok(success), 1, suite->name); return 0; } diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index 9bbc422c284b..74982b83707c 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -166,7 +166,7 @@ static void kunit_exec_run_tests(struct suite_set *suite_set) { size_t num_suites = suite_set->end - suite_set->start; - pr_info("TAP version 14\n"); + pr_info("KTAP version 1\n"); pr_info("1..%zu\n", num_suites); __kunit_test_suites_init(suite_set->start, num_suites); @@ -177,8 +177,8 @@ static void kunit_exec_list_tests(struct suite_set *suite_set) struct kunit_suite * const *suites; struct kunit_case *test_case; - /* Hack: print a tap header so kunit.py can find the start of KUnit output. */ - pr_info("TAP version 14\n"); + /* Hack: print a ktap header so kunit.py can find the start of KUnit output. */ + pr_info("KTAP version 1\n"); for (suites = suite_set->start; suites < suite_set->end; suites++) kunit_suite_for_each_test_case((*suites), test_case) { diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 314717b63080..87a5d795843b 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -151,6 +151,7 @@ EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases); static void kunit_print_suite_start(struct kunit_suite *suite) { + kunit_log(KERN_INFO, suite, KUNIT_SUBTEST_INDENT "KTAP version 1\n"); kunit_log(KERN_INFO, suite, KUNIT_SUBTEST_INDENT "# Subtest: %s", suite->name); kunit_log(KERN_INFO, suite, KUNIT_SUBTEST_INDENT "1..%zd", @@ -177,13 +178,13 @@ static void kunit_print_ok_not_ok(void *test_or_suite, * representation. */ if (suite) - pr_info("%s %zd - %s%s%s\n", + pr_info("%s %zd %s%s%s\n", kunit_status_to_ok_not_ok(status), test_number, description, directive_header, (status == KUNIT_SKIPPED) ? directive : ""); else kunit_log(KERN_INFO, test, - KUNIT_SUBTEST_INDENT "%s %zd - %s%s%s", + KUNIT_SUBTEST_INDENT "%s %zd %s%s%s", kunit_status_to_ok_not_ok(status), test_number, description, directive_header, (status == KUNIT_SKIPPED) ? directive : ""); @@ -544,6 +545,8 @@ int kunit_run_tests(struct kunit_suite *suite) /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); + kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "KTAP version 1\n"); kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT "# Subtest: %s", test_case->name); @@ -557,7 +560,7 @@ int kunit_run_tests(struct kunit_suite *suite) kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT - "%s %d - %s", + "%s %d %s", kunit_status_to_ok_not_ok(test.status), test.param_index + 1, param_desc); From 5937e0c04afc7d4b7b737fda93316ba4b74183c0 Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Mon, 28 Nov 2022 16:12:34 -0800 Subject: [PATCH 24/29] kunit: tool: don't include KTAP headers and the like in the test log We print the "test log" on failure. This is meant to be all the kernel output that happened during the test. But we also include the special KTAP lines in it, which are often redundant. E.g. we include the "not ok" line in the log, right before we print that the test case failed... [13:51:48] Expected 2 + 1 == 2, but [13:51:48] 2 + 1 == 3 (0x3) [13:51:48] not ok 1 example_simple_test [13:51:48] [FAILED] example_simple_test More full example after this patch: [13:51:48] =================== example (4 subtests) =================== [13:51:48] # example_simple_test: initializing [13:51:48] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 [13:51:48] Expected 2 + 1 == 2, but [13:51:48] 2 + 1 == 3 (0x3) [13:51:48] [FAILED] example_simple_test Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit_parser.py | 8 ++++---- tools/testing/kunit/kunit_tool_test.py | 17 +++++++++++++++++ 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 4cc2f8b7ecd0..99b8f058db40 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -295,7 +295,7 @@ def parse_ktap_header(lines: LineStream, test: Test) -> bool: check_version(version_num, TAP_VERSIONS, 'TAP', test) else: return False - test.log.append(lines.pop()) + lines.pop() return True TEST_HEADER = re.compile(r'^# Subtest: (.*)$') @@ -318,8 +318,8 @@ def parse_test_header(lines: LineStream, test: Test) -> bool: match = TEST_HEADER.match(lines.peek()) if not match: return False - test.log.append(lines.pop()) test.name = match.group(1) + lines.pop() return True TEST_PLAN = re.compile(r'1\.\.([0-9]+)') @@ -345,9 +345,9 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: if not match: test.expected_count = None return False - test.log.append(lines.pop()) expected_count = int(match.group(1)) test.expected_count = expected_count + lines.pop() return True TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') @@ -409,7 +409,7 @@ def parse_test_result(lines: LineStream, test: Test, # Check if line matches test result line format if not match: return False - test.log.append(lines.pop()) + lines.pop() # Set name of test object if skip_match: diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index d7f669cbf2a8..1ef921ac4331 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -84,6 +84,10 @@ class KUnitParserTest(unittest.TestCase): self.print_mock = mock.patch('kunit_printer.Printer.print').start() self.addCleanup(mock.patch.stopall) + def noPrintCallContains(self, substr: str): + for call in self.print_mock.mock_calls: + self.assertNotIn(substr, call.args[0]) + def assertContains(self, needle: str, haystack: kunit_parser.LineStream): # Clone the iterator so we can print the contents on failure. copy, backup = itertools.tee(haystack) @@ -327,6 +331,19 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests(file.readlines()) self.print_mock.assert_any_call(StrContains('suite (1 subtest)')) + def test_show_test_output_on_failure(self): + output = """ + KTAP version 1 + 1..1 + Test output. + not ok 1 test1 + """ + result = kunit_parser.parse_run_tests(output.splitlines()) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) + + self.print_mock.assert_any_call(StrContains('Test output.')) + self.noPrintCallContains('not ok 1 test1') + def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: return kunit_parser.LineStream(enumerate(strs, start=1)) From a81fe7ecf717e5ae16892c282378ca380d4e99c9 Mon Sep 17 00:00:00 2001 From: David Gow Date: Wed, 7 Dec 2022 12:33:19 +0800 Subject: [PATCH 25/29] Documentation: kunit: Fix "How Do I Use This" / "Next Steps" sections The "How Do I Use This" section of index.rst and "Next Steps" section of start.rst were just copies of the table of contents, and therefore weren't really useful either when looking a sphinx generated output (which already had the TOC visible) or when reading the source (where it's just a list of files that ls could give you). Instead, provide a small number of concrete next steps, and a bit more description about what the pages contain. This also removes the broken reference to 'tips.rst', which was previously removed. Fixed git am whitespace complaints during commit: Shuah Khan Fixes: 4399c737a97d ("Documentation: kunit: Remove redundant 'tips.rst' page") Signed-off-by: David Gow Reviewed-by: Sadiya Kazi Reviewed-by: Bagas Sanjaya Signed-off-by: Shuah Khan --- Documentation/dev-tools/kunit/index.rst | 19 ++++++++----------- Documentation/dev-tools/kunit/start.rst | 18 ++++++++---------- 2 files changed, 16 insertions(+), 21 deletions(-) diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst index d5629817cd72..b3593ae29ace 100644 --- a/Documentation/dev-tools/kunit/index.rst +++ b/Documentation/dev-tools/kunit/index.rst @@ -99,14 +99,11 @@ Read also :ref:`kinds-of-tests`. How do I use it? ================ -* Documentation/dev-tools/kunit/start.rst - for KUnit new users. -* Documentation/dev-tools/kunit/architecture.rst - KUnit architecture. -* Documentation/dev-tools/kunit/run_wrapper.rst - run kunit_tool. -* Documentation/dev-tools/kunit/run_manual.rst - run tests without kunit_tool. -* Documentation/dev-tools/kunit/usage.rst - write tests. -* Documentation/dev-tools/kunit/tips.rst - best practices with - examples. -* Documentation/dev-tools/kunit/api/index.rst - KUnit APIs - used for testing. -* Documentation/dev-tools/kunit/faq.rst - KUnit common questions and - answers. +You can find a step-by-step guide to writing and running KUnit tests in +Documentation/dev-tools/kunit/start.rst + +Alternatively, feel free to look through the rest of the KUnit documentation, +or to experiment with tools/testing/kunit/kunit.py and the example test under +lib/kunit/kunit-example-test.c + +Happy testing! diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-tools/kunit/start.rst index f4f504f1fb15..c736613c9b19 100644 --- a/Documentation/dev-tools/kunit/start.rst +++ b/Documentation/dev-tools/kunit/start.rst @@ -294,13 +294,11 @@ Congrats! You just wrote your first KUnit test. Next Steps ========== -* Documentation/dev-tools/kunit/architecture.rst - KUnit architecture. -* Documentation/dev-tools/kunit/run_wrapper.rst - run kunit_tool. -* Documentation/dev-tools/kunit/run_manual.rst - run tests without kunit_tool. -* Documentation/dev-tools/kunit/usage.rst - write tests. -* Documentation/dev-tools/kunit/tips.rst - best practices with - examples. -* Documentation/dev-tools/kunit/api/index.rst - KUnit APIs - used for testing. -* Documentation/dev-tools/kunit/faq.rst - KUnit common questions and - answers. +If you're interested in using some of the more advanced features of kunit.py, +take a look at Documentation/dev-tools/kunit/run_wrapper.rst + +If you'd like to run tests without using kunit.py, check out +Documentation/dev-tools/kunit/run_manual.rst + +For more information on writing KUnit tests (including some common techniques +for testing different things), see Documentation/dev-tools/kunit/usage.rst From c2bb92bc4ea13842fdd27819c0d5b48df2b86ea5 Mon Sep 17 00:00:00 2001 From: Daniel Latypov Date: Wed, 30 Nov 2022 10:54:19 -0800 Subject: [PATCH 26/29] kunit: tool: make parser preserve whitespace when printing test log Currently, kunit_parser.py is stripping all leading whitespace to make parsing easier. But this means we can't accurately show kernel output for failing tests or when the kernel crashes. Embarassingly, this affects even KUnit's own output, e.g. [13:40:46] Expected 2 + 1 == 2, but [13:40:46] 2 + 1 == 3 (0x3) [13:40:46] not ok 1 example_simple_test [13:40:46] [FAILED] example_simple_test After this change, here's what the output in context would look like [13:40:46] =================== example (4 subtests) =================== [13:40:46] # example_simple_test: initializing [13:40:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 [13:40:46] Expected 2 + 1 == 2, but [13:40:46] 2 + 1 == 3 (0x3) [13:40:46] [FAILED] example_simple_test [13:40:46] [SKIPPED] example_skip_test [13:40:46] [SKIPPED] example_mark_skipped_test [13:40:46] [PASSED] example_all_expect_macros_test [13:40:46] # example: initializing suite [13:40:46] # example: pass:1 fail:1 skip:2 total:4 [13:40:46] # Totals: pass:1 fail:1 skip:2 total:4 [13:40:46] ===================== [FAILED] example ===================== This example shows one minor cosmetic defect this approach has. The test counts lines prevent us from dedenting the suite-level output. But at the same time, any form of non-KUnit output would do the same unless it happened to be indented as well. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan --- tools/testing/kunit/kunit.py | 2 +- tools/testing/kunit/kunit_parser.py | 27 +++++++++++++------------- tools/testing/kunit/kunit_tool_test.py | 2 ++ 3 files changed, 16 insertions(+), 15 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index e7b6549712d6..43fbe96318fe 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -202,7 +202,7 @@ def parse_tests(request: KunitParseRequest, metadata: kunit_json.Metadata, input if request.raw_output == 'all': pass elif request.raw_output == 'kunit': - output = kunit_parser.extract_tap_lines(output, lstrip=False) + output = kunit_parser.extract_tap_lines(output) for line in output: print(line.rstrip()) parse_time = time.time() - parse_start diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 99b8f058db40..a225799f6b1b 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -13,6 +13,7 @@ from __future__ import annotations from dataclasses import dataclass import re import sys +import textwrap from enum import Enum, auto from typing import Iterable, Iterator, List, Optional, Tuple @@ -208,12 +209,12 @@ class LineStream: # Parsing helper methods: -KTAP_START = re.compile(r'KTAP version ([0-9]+)$') -TAP_START = re.compile(r'TAP version ([0-9]+)$') -KTAP_END = re.compile('(List of all partitions:|' +KTAP_START = re.compile(r'\s*KTAP version ([0-9]+)$') +TAP_START = re.compile(r'\s*TAP version ([0-9]+)$') +KTAP_END = re.compile(r'\s*(List of all partitions:|' 'Kernel panic - not syncing: VFS:|reboot: System halted)') -def extract_tap_lines(kernel_output: Iterable[str], lstrip=True) -> LineStream: +def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: """Extracts KTAP lines from the kernel output.""" def isolate_ktap_output(kernel_output: Iterable[str]) \ -> Iterator[Tuple[int, str]]: @@ -239,11 +240,8 @@ def extract_tap_lines(kernel_output: Iterable[str], lstrip=True) -> LineStream: # stop extracting KTAP lines break elif started: - # remove the prefix and optionally any leading - # whitespace. Our parsing logic relies on this. + # remove the prefix, if any. line = line[prefix_len:] - if lstrip: - line = line.lstrip() yield line_num, line return LineStream(lines=isolate_ktap_output(kernel_output)) @@ -298,7 +296,7 @@ def parse_ktap_header(lines: LineStream, test: Test) -> bool: lines.pop() return True -TEST_HEADER = re.compile(r'^# Subtest: (.*)$') +TEST_HEADER = re.compile(r'^\s*# Subtest: (.*)$') def parse_test_header(lines: LineStream, test: Test) -> bool: """ @@ -322,7 +320,7 @@ def parse_test_header(lines: LineStream, test: Test) -> bool: lines.pop() return True -TEST_PLAN = re.compile(r'1\.\.([0-9]+)') +TEST_PLAN = re.compile(r'^\s*1\.\.([0-9]+)') def parse_test_plan(lines: LineStream, test: Test) -> bool: """ @@ -350,9 +348,9 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: lines.pop() return True -TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') +TEST_RESULT = re.compile(r'^\s*(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') -TEST_RESULT_SKIP = re.compile(r'^(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') +TEST_RESULT_SKIP = re.compile(r'^\s*(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') def peek_test_name_match(lines: LineStream, test: Test) -> bool: """ @@ -511,8 +509,9 @@ def print_test_header(test: Test) -> None: def print_log(log: Iterable[str]) -> None: """Prints all strings in saved log for test in yellow.""" - for m in log: - stdout.print_with_timestamp(stdout.yellow(m)) + formatted = textwrap.dedent('\n'.join(log)) + for line in formatted.splitlines(): + stdout.print_with_timestamp(stdout.yellow(line)) def format_test_result(test: Test) -> str: """ diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 1ef921ac4331..0c2190514103 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -336,12 +336,14 @@ class KUnitParserTest(unittest.TestCase): KTAP version 1 1..1 Test output. + Indented more. not ok 1 test1 """ result = kunit_parser.parse_run_tests(output.splitlines()) self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) self.print_mock.assert_any_call(StrContains('Test output.')) + self.print_mock.assert_any_call(StrContains(' Indented more.')) self.noPrintCallContains('not ok 1 test1') def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: From 9c988fae6f6ae3224a568ab985881b66bb50c9ec Mon Sep 17 00:00:00 2001 From: Rae Moar Date: Wed, 7 Dec 2022 01:40:23 +0000 Subject: [PATCH 27/29] kunit: add macro to allow conditionally exposing static symbols to tests Create two macros: VISIBLE_IF_KUNIT - A macro that sets symbols to be static if CONFIG_KUNIT is not enabled. Otherwise if CONFIG_KUNIT is enabled there is no change to the symbol definition. EXPORT_SYMBOL_IF_KUNIT(symbol) - Exports symbol into EXPORTED_FOR_KUNIT_TESTING namespace only if CONFIG_KUNIT is enabled. Must use MODULE_IMPORT_NS(EXPORTED_FOR_KUNIT_TESTING) in test file in order to use symbols. Signed-off-by: Rae Moar Reviewed-by: John Johansen Reviewed-by: David Gow Signed-off-by: Shuah Khan --- include/kunit/visibility.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 include/kunit/visibility.h diff --git a/include/kunit/visibility.h b/include/kunit/visibility.h new file mode 100644 index 000000000000..0dfe35feeec6 --- /dev/null +++ b/include/kunit/visibility.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * KUnit API to allow symbols to be conditionally visible during KUnit + * testing + * + * Copyright (C) 2022, Google LLC. + * Author: Rae Moar + */ + +#ifndef _KUNIT_VISIBILITY_H +#define _KUNIT_VISIBILITY_H + +#if IS_ENABLED(CONFIG_KUNIT) + /** + * VISIBLE_IF_KUNIT - A macro that sets symbols to be static if + * CONFIG_KUNIT is not enabled. Otherwise if CONFIG_KUNIT is enabled + * there is no change to the symbol definition. + */ + #define VISIBLE_IF_KUNIT + /** + * EXPORT_SYMBOL_IF_KUNIT(symbol) - Exports symbol into + * EXPORTED_FOR_KUNIT_TESTING namespace only if CONFIG_KUNIT is + * enabled. Must use MODULE_IMPORT_NS(EXPORTED_FOR_KUNIT_TESTING) + * in test file in order to use symbols. + */ + #define EXPORT_SYMBOL_IF_KUNIT(symbol) EXPORT_SYMBOL_NS(symbol, \ + EXPORTED_FOR_KUNIT_TESTING) +#else + #define VISIBLE_IF_KUNIT static + #define EXPORT_SYMBOL_IF_KUNIT(symbol) +#endif + +#endif /* _KUNIT_VISIBILITY_H */ From b11e51dd70947107fa4076c6286dce301671afc1 Mon Sep 17 00:00:00 2001 From: Rae Moar Date: Wed, 7 Dec 2022 01:40:24 +0000 Subject: [PATCH 28/29] apparmor: test: make static symbols visible during kunit testing Use macros, VISIBLE_IF_KUNIT and EXPORT_SYMBOL_IF_KUNIT, to allow static symbols to be conditionally set to be visible during apparmor_policy_unpack_test, which removes the need to include the testing file in the implementation file. Change the namespace of the symbols that are now conditionally visible (by adding the prefix aa_) to avoid confusion with symbols of the same name. Allow the test to be built as a module and namespace the module name from policy_unpack_test to apparmor_policy_unpack_test to improve clarity of the module name. Provide an example of how static symbols can be dealt with in testing. Signed-off-by: Rae Moar Reviewed-by: David Gow Acked-by: John Johansen Signed-off-by: Shuah Khan --- security/apparmor/Kconfig | 4 +- security/apparmor/Makefile | 3 + security/apparmor/include/policy_unpack.h | 50 +++++ security/apparmor/policy_unpack.c | 238 ++++++++++------------ security/apparmor/policy_unpack_test.c | 69 ++++--- 5 files changed, 196 insertions(+), 168 deletions(-) diff --git a/security/apparmor/Kconfig b/security/apparmor/Kconfig index cb3496e00d8a..f334e7cccf2d 100644 --- a/security/apparmor/Kconfig +++ b/security/apparmor/Kconfig @@ -106,8 +106,8 @@ config SECURITY_APPARMOR_PARANOID_LOAD Disabling the check will speed up policy loads. config SECURITY_APPARMOR_KUNIT_TEST - bool "Build KUnit tests for policy_unpack.c" if !KUNIT_ALL_TESTS - depends on KUNIT=y && SECURITY_APPARMOR + tristate "Build KUnit tests for policy_unpack.c" if !KUNIT_ALL_TESTS + depends on KUNIT && SECURITY_APPARMOR default KUNIT_ALL_TESTS help This builds the AppArmor KUnit tests. diff --git a/security/apparmor/Makefile b/security/apparmor/Makefile index ff23fcfefe19..065f4e346553 100644 --- a/security/apparmor/Makefile +++ b/security/apparmor/Makefile @@ -8,6 +8,9 @@ apparmor-y := apparmorfs.o audit.o capability.o task.o ipc.o lib.o match.o \ resource.o secid.o file.o policy_ns.o label.o mount.o net.o apparmor-$(CONFIG_SECURITY_APPARMOR_HASH) += crypto.o +obj-$(CONFIG_SECURITY_APPARMOR_KUNIT_TEST) += apparmor_policy_unpack_test.o +apparmor_policy_unpack_test-objs += policy_unpack_test.o + clean-files := capability_names.h rlim_names.h net_names.h # Build a lower case string table of address family names diff --git a/security/apparmor/include/policy_unpack.h b/security/apparmor/include/policy_unpack.h index eb5f7d7f132b..e89b701447bc 100644 --- a/security/apparmor/include/policy_unpack.h +++ b/security/apparmor/include/policy_unpack.h @@ -48,6 +48,43 @@ enum { AAFS_LOADDATA_NDENTS /* count of entries */ }; +/* + * The AppArmor interface treats data as a type byte followed by the + * actual data. The interface has the notion of a named entry + * which has a name (AA_NAME typecode followed by name string) followed by + * the entries typecode and data. Named types allow for optional + * elements and extensions to be added and tested for without breaking + * backwards compatibility. + */ + +enum aa_code { + AA_U8, + AA_U16, + AA_U32, + AA_U64, + AA_NAME, /* same as string except it is items name */ + AA_STRING, + AA_BLOB, + AA_STRUCT, + AA_STRUCTEND, + AA_LIST, + AA_LISTEND, + AA_ARRAY, + AA_ARRAYEND, +}; + +/* + * aa_ext is the read of the buffer containing the serialized profile. The + * data is copied into a kernel buffer in apparmorfs and then handed off to + * the unpack routines. + */ +struct aa_ext { + void *start; + void *end; + void *pos; /* pointer to current position in the buffer */ + u32 version; +}; + /* * struct aa_loaddata - buffer of policy raw_data set * @@ -126,4 +163,17 @@ static inline void aa_put_loaddata(struct aa_loaddata *data) kref_put(&data->count, aa_loaddata_kref); } +#if IS_ENABLED(CONFIG_KUNIT) +bool aa_inbounds(struct aa_ext *e, size_t size); +size_t aa_unpack_u16_chunk(struct aa_ext *e, char **chunk); +bool aa_unpack_X(struct aa_ext *e, enum aa_code code); +bool aa_unpack_nameX(struct aa_ext *e, enum aa_code code, const char *name); +bool aa_unpack_u32(struct aa_ext *e, u32 *data, const char *name); +bool aa_unpack_u64(struct aa_ext *e, u64 *data, const char *name); +size_t aa_unpack_array(struct aa_ext *e, const char *name); +size_t aa_unpack_blob(struct aa_ext *e, char **blob, const char *name); +int aa_unpack_str(struct aa_ext *e, const char **string, const char *name); +int aa_unpack_strdup(struct aa_ext *e, char **string, const char *name); +#endif + #endif /* __POLICY_INTERFACE_H */ diff --git a/security/apparmor/policy_unpack.c b/security/apparmor/policy_unpack.c index 55d31bac4f35..12e535fdfa8b 100644 --- a/security/apparmor/policy_unpack.c +++ b/security/apparmor/policy_unpack.c @@ -14,6 +14,7 @@ */ #include +#include #include #include #include @@ -37,43 +38,6 @@ #define v7 7 #define v8 8 /* full network masking */ -/* - * The AppArmor interface treats data as a type byte followed by the - * actual data. The interface has the notion of a named entry - * which has a name (AA_NAME typecode followed by name string) followed by - * the entries typecode and data. Named types allow for optional - * elements and extensions to be added and tested for without breaking - * backwards compatibility. - */ - -enum aa_code { - AA_U8, - AA_U16, - AA_U32, - AA_U64, - AA_NAME, /* same as string except it is items name */ - AA_STRING, - AA_BLOB, - AA_STRUCT, - AA_STRUCTEND, - AA_LIST, - AA_LISTEND, - AA_ARRAY, - AA_ARRAYEND, -}; - -/* - * aa_ext is the read of the buffer containing the serialized profile. The - * data is copied into a kernel buffer in apparmorfs and then handed off to - * the unpack routines. - */ -struct aa_ext { - void *start; - void *end; - void *pos; /* pointer to current position in the buffer */ - u32 version; -}; - /* audit callback for unpack fields */ static void audit_cb(struct audit_buffer *ab, void *va) { @@ -199,10 +163,11 @@ struct aa_loaddata *aa_loaddata_alloc(size_t size) } /* test if read will be in packed data bounds */ -static bool inbounds(struct aa_ext *e, size_t size) +VISIBLE_IF_KUNIT bool aa_inbounds(struct aa_ext *e, size_t size) { return (size <= e->end - e->pos); } +EXPORT_SYMBOL_IF_KUNIT(aa_inbounds); static void *kvmemdup(const void *src, size_t len) { @@ -214,22 +179,22 @@ static void *kvmemdup(const void *src, size_t len) } /** - * unpack_u16_chunk - test and do bounds checking for a u16 size based chunk + * aa_unpack_u16_chunk - test and do bounds checking for a u16 size based chunk * @e: serialized data read head (NOT NULL) * @chunk: start address for chunk of data (NOT NULL) * * Returns: the size of chunk found with the read head at the end of the chunk. */ -static size_t unpack_u16_chunk(struct aa_ext *e, char **chunk) +VISIBLE_IF_KUNIT size_t aa_unpack_u16_chunk(struct aa_ext *e, char **chunk) { size_t size = 0; void *pos = e->pos; - if (!inbounds(e, sizeof(u16))) + if (!aa_inbounds(e, sizeof(u16))) goto fail; size = le16_to_cpu(get_unaligned((__le16 *) e->pos)); e->pos += sizeof(__le16); - if (!inbounds(e, size)) + if (!aa_inbounds(e, size)) goto fail; *chunk = e->pos; e->pos += size; @@ -239,20 +204,22 @@ fail: e->pos = pos; return 0; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_u16_chunk); /* unpack control byte */ -static bool unpack_X(struct aa_ext *e, enum aa_code code) +VISIBLE_IF_KUNIT bool aa_unpack_X(struct aa_ext *e, enum aa_code code) { - if (!inbounds(e, 1)) + if (!aa_inbounds(e, 1)) return false; if (*(u8 *) e->pos != code) return false; e->pos++; return true; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_X); /** - * unpack_nameX - check is the next element is of type X with a name of @name + * aa_unpack_nameX - check is the next element is of type X with a name of @name * @e: serialized data extent information (NOT NULL) * @code: type code * @name: name to match to the serialized element. (MAYBE NULL) @@ -267,7 +234,7 @@ static bool unpack_X(struct aa_ext *e, enum aa_code code) * * Returns: false if either match fails, the read head does not move */ -static bool unpack_nameX(struct aa_ext *e, enum aa_code code, const char *name) +VISIBLE_IF_KUNIT bool aa_unpack_nameX(struct aa_ext *e, enum aa_code code, const char *name) { /* * May need to reset pos if name or type doesn't match @@ -277,9 +244,9 @@ static bool unpack_nameX(struct aa_ext *e, enum aa_code code, const char *name) * Check for presence of a tagname, and if present name size * AA_NAME tag value is a u16. */ - if (unpack_X(e, AA_NAME)) { + if (aa_unpack_X(e, AA_NAME)) { char *tag = NULL; - size_t size = unpack_u16_chunk(e, &tag); + size_t size = aa_unpack_u16_chunk(e, &tag); /* if a name is specified it must match. otherwise skip tag */ if (name && (!size || tag[size-1] != '\0' || strcmp(name, tag))) goto fail; @@ -289,20 +256,21 @@ static bool unpack_nameX(struct aa_ext *e, enum aa_code code, const char *name) } /* now check if type code matches */ - if (unpack_X(e, code)) + if (aa_unpack_X(e, code)) return true; fail: e->pos = pos; return false; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_nameX); static bool unpack_u8(struct aa_ext *e, u8 *data, const char *name) { void *pos = e->pos; - if (unpack_nameX(e, AA_U8, name)) { - if (!inbounds(e, sizeof(u8))) + if (aa_unpack_nameX(e, AA_U8, name)) { + if (!aa_inbounds(e, sizeof(u8))) goto fail; if (data) *data = *((u8 *)e->pos); @@ -315,12 +283,12 @@ fail: return false; } -static bool unpack_u32(struct aa_ext *e, u32 *data, const char *name) +VISIBLE_IF_KUNIT bool aa_unpack_u32(struct aa_ext *e, u32 *data, const char *name) { void *pos = e->pos; - if (unpack_nameX(e, AA_U32, name)) { - if (!inbounds(e, sizeof(u32))) + if (aa_unpack_nameX(e, AA_U32, name)) { + if (!aa_inbounds(e, sizeof(u32))) goto fail; if (data) *data = le32_to_cpu(get_unaligned((__le32 *) e->pos)); @@ -332,13 +300,14 @@ fail: e->pos = pos; return false; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_u32); -static bool unpack_u64(struct aa_ext *e, u64 *data, const char *name) +VISIBLE_IF_KUNIT bool aa_unpack_u64(struct aa_ext *e, u64 *data, const char *name) { void *pos = e->pos; - if (unpack_nameX(e, AA_U64, name)) { - if (!inbounds(e, sizeof(u64))) + if (aa_unpack_nameX(e, AA_U64, name)) { + if (!aa_inbounds(e, sizeof(u64))) goto fail; if (data) *data = le64_to_cpu(get_unaligned((__le64 *) e->pos)); @@ -350,14 +319,15 @@ fail: e->pos = pos; return false; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_u64); -static size_t unpack_array(struct aa_ext *e, const char *name) +VISIBLE_IF_KUNIT size_t aa_unpack_array(struct aa_ext *e, const char *name) { void *pos = e->pos; - if (unpack_nameX(e, AA_ARRAY, name)) { + if (aa_unpack_nameX(e, AA_ARRAY, name)) { int size; - if (!inbounds(e, sizeof(u16))) + if (!aa_inbounds(e, sizeof(u16))) goto fail; size = (int)le16_to_cpu(get_unaligned((__le16 *) e->pos)); e->pos += sizeof(u16); @@ -368,18 +338,19 @@ fail: e->pos = pos; return 0; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_array); -static size_t unpack_blob(struct aa_ext *e, char **blob, const char *name) +VISIBLE_IF_KUNIT size_t aa_unpack_blob(struct aa_ext *e, char **blob, const char *name) { void *pos = e->pos; - if (unpack_nameX(e, AA_BLOB, name)) { + if (aa_unpack_nameX(e, AA_BLOB, name)) { u32 size; - if (!inbounds(e, sizeof(u32))) + if (!aa_inbounds(e, sizeof(u32))) goto fail; size = le32_to_cpu(get_unaligned((__le32 *) e->pos)); e->pos += sizeof(u32); - if (inbounds(e, (size_t) size)) { + if (aa_inbounds(e, (size_t) size)) { *blob = e->pos; e->pos += size; return size; @@ -390,15 +361,16 @@ fail: e->pos = pos; return 0; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_blob); -static int unpack_str(struct aa_ext *e, const char **string, const char *name) +VISIBLE_IF_KUNIT int aa_unpack_str(struct aa_ext *e, const char **string, const char *name) { char *src_str; size_t size = 0; void *pos = e->pos; *string = NULL; - if (unpack_nameX(e, AA_STRING, name)) { - size = unpack_u16_chunk(e, &src_str); + if (aa_unpack_nameX(e, AA_STRING, name)) { + size = aa_unpack_u16_chunk(e, &src_str); if (size) { /* strings are null terminated, length is size - 1 */ if (src_str[size - 1] != 0) @@ -413,12 +385,13 @@ fail: e->pos = pos; return 0; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_str); -static int unpack_strdup(struct aa_ext *e, char **string, const char *name) +VISIBLE_IF_KUNIT int aa_unpack_strdup(struct aa_ext *e, char **string, const char *name) { const char *tmp; void *pos = e->pos; - int res = unpack_str(e, &tmp, name); + int res = aa_unpack_str(e, &tmp, name); *string = NULL; if (!res) @@ -432,6 +405,7 @@ static int unpack_strdup(struct aa_ext *e, char **string, const char *name) return res; } +EXPORT_SYMBOL_IF_KUNIT(aa_unpack_strdup); /** @@ -446,7 +420,7 @@ static struct aa_dfa *unpack_dfa(struct aa_ext *e) size_t size; struct aa_dfa *dfa = NULL; - size = unpack_blob(e, &blob, "aadfa"); + size = aa_unpack_blob(e, &blob, "aadfa"); if (size) { /* * The dfa is aligned with in the blob to 8 bytes @@ -482,10 +456,10 @@ static bool unpack_trans_table(struct aa_ext *e, struct aa_profile *profile) void *saved_pos = e->pos; /* exec table is optional */ - if (unpack_nameX(e, AA_STRUCT, "xtable")) { + if (aa_unpack_nameX(e, AA_STRUCT, "xtable")) { int i, size; - size = unpack_array(e, NULL); + size = aa_unpack_array(e, NULL); /* currently 4 exec bits and entries 0-3 are reserved iupcx */ if (size > 16 - 4) goto fail; @@ -497,8 +471,8 @@ static bool unpack_trans_table(struct aa_ext *e, struct aa_profile *profile) profile->file.trans.size = size; for (i = 0; i < size; i++) { char *str; - int c, j, pos, size2 = unpack_strdup(e, &str, NULL); - /* unpack_strdup verifies that the last character is + int c, j, pos, size2 = aa_unpack_strdup(e, &str, NULL); + /* aa_unpack_strdup verifies that the last character is * null termination byte. */ if (!size2) @@ -521,7 +495,7 @@ static bool unpack_trans_table(struct aa_ext *e, struct aa_profile *profile) goto fail; /* beginning with : requires an embedded \0, * verify that exactly 1 internal \0 exists - * trailing \0 already verified by unpack_strdup + * trailing \0 already verified by aa_unpack_strdup * * convert \0 back to : for label_parse */ @@ -533,9 +507,9 @@ static bool unpack_trans_table(struct aa_ext *e, struct aa_profile *profile) /* fail - all other cases with embedded \0 */ goto fail; } - if (!unpack_nameX(e, AA_ARRAYEND, NULL)) + if (!aa_unpack_nameX(e, AA_ARRAYEND, NULL)) goto fail; - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) goto fail; } return true; @@ -550,21 +524,21 @@ static bool unpack_xattrs(struct aa_ext *e, struct aa_profile *profile) { void *pos = e->pos; - if (unpack_nameX(e, AA_STRUCT, "xattrs")) { + if (aa_unpack_nameX(e, AA_STRUCT, "xattrs")) { int i, size; - size = unpack_array(e, NULL); + size = aa_unpack_array(e, NULL); profile->xattr_count = size; profile->xattrs = kcalloc(size, sizeof(char *), GFP_KERNEL); if (!profile->xattrs) goto fail; for (i = 0; i < size; i++) { - if (!unpack_strdup(e, &profile->xattrs[i], NULL)) + if (!aa_unpack_strdup(e, &profile->xattrs[i], NULL)) goto fail; } - if (!unpack_nameX(e, AA_ARRAYEND, NULL)) + if (!aa_unpack_nameX(e, AA_ARRAYEND, NULL)) goto fail; - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) goto fail; } @@ -580,8 +554,8 @@ static bool unpack_secmark(struct aa_ext *e, struct aa_profile *profile) void *pos = e->pos; int i, size; - if (unpack_nameX(e, AA_STRUCT, "secmark")) { - size = unpack_array(e, NULL); + if (aa_unpack_nameX(e, AA_STRUCT, "secmark")) { + size = aa_unpack_array(e, NULL); profile->secmark = kcalloc(size, sizeof(struct aa_secmark), GFP_KERNEL); @@ -595,12 +569,12 @@ static bool unpack_secmark(struct aa_ext *e, struct aa_profile *profile) goto fail; if (!unpack_u8(e, &profile->secmark[i].deny, NULL)) goto fail; - if (!unpack_strdup(e, &profile->secmark[i].label, NULL)) + if (!aa_unpack_strdup(e, &profile->secmark[i].label, NULL)) goto fail; } - if (!unpack_nameX(e, AA_ARRAYEND, NULL)) + if (!aa_unpack_nameX(e, AA_ARRAYEND, NULL)) goto fail; - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) goto fail; } @@ -624,26 +598,26 @@ static bool unpack_rlimits(struct aa_ext *e, struct aa_profile *profile) void *pos = e->pos; /* rlimits are optional */ - if (unpack_nameX(e, AA_STRUCT, "rlimits")) { + if (aa_unpack_nameX(e, AA_STRUCT, "rlimits")) { int i, size; u32 tmp = 0; - if (!unpack_u32(e, &tmp, NULL)) + if (!aa_unpack_u32(e, &tmp, NULL)) goto fail; profile->rlimits.mask = tmp; - size = unpack_array(e, NULL); + size = aa_unpack_array(e, NULL); if (size > RLIM_NLIMITS) goto fail; for (i = 0; i < size; i++) { u64 tmp2 = 0; int a = aa_map_resource(i); - if (!unpack_u64(e, &tmp2, NULL)) + if (!aa_unpack_u64(e, &tmp2, NULL)) goto fail; profile->rlimits.limits[a].rlim_max = tmp2; } - if (!unpack_nameX(e, AA_ARRAYEND, NULL)) + if (!aa_unpack_nameX(e, AA_ARRAYEND, NULL)) goto fail; - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) goto fail; } return true; @@ -691,9 +665,9 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) *ns_name = NULL; /* check that we have the right struct being passed */ - if (!unpack_nameX(e, AA_STRUCT, "profile")) + if (!aa_unpack_nameX(e, AA_STRUCT, "profile")) goto fail; - if (!unpack_str(e, &name, NULL)) + if (!aa_unpack_str(e, &name, NULL)) goto fail; if (*name == '\0') goto fail; @@ -713,10 +687,10 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) return ERR_PTR(-ENOMEM); /* profile renaming is optional */ - (void) unpack_str(e, &profile->rename, "rename"); + (void) aa_unpack_str(e, &profile->rename, "rename"); /* attachment string is optional */ - (void) unpack_str(e, &profile->attach, "attach"); + (void) aa_unpack_str(e, &profile->attach, "attach"); /* xmatch is optional and may be NULL */ profile->xmatch = unpack_dfa(e); @@ -728,7 +702,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) } /* xmatch_len is not optional if xmatch is set */ if (profile->xmatch) { - if (!unpack_u32(e, &tmp, NULL)) { + if (!aa_unpack_u32(e, &tmp, NULL)) { info = "missing xmatch len"; goto fail; } @@ -736,15 +710,15 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) } /* disconnected attachment string is optional */ - (void) unpack_str(e, &profile->disconnected, "disconnected"); + (void) aa_unpack_str(e, &profile->disconnected, "disconnected"); /* per profile debug flags (complain, audit) */ - if (!unpack_nameX(e, AA_STRUCT, "flags")) { + if (!aa_unpack_nameX(e, AA_STRUCT, "flags")) { info = "profile missing flags"; goto fail; } info = "failed to unpack profile flags"; - if (!unpack_u32(e, &tmp, NULL)) + if (!aa_unpack_u32(e, &tmp, NULL)) goto fail; if (tmp & PACKED_FLAG_HAT) profile->label.flags |= FLAG_HAT; @@ -752,7 +726,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) profile->label.flags |= FLAG_DEBUG1; if (tmp & PACKED_FLAG_DEBUG2) profile->label.flags |= FLAG_DEBUG2; - if (!unpack_u32(e, &tmp, NULL)) + if (!aa_unpack_u32(e, &tmp, NULL)) goto fail; if (tmp == PACKED_MODE_COMPLAIN || (e->version & FORCE_COMPLAIN_FLAG)) { profile->mode = APPARMOR_COMPLAIN; @@ -766,16 +740,16 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) } else { goto fail; } - if (!unpack_u32(e, &tmp, NULL)) + if (!aa_unpack_u32(e, &tmp, NULL)) goto fail; if (tmp) profile->audit = AUDIT_ALL; - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) goto fail; /* path_flags is optional */ - if (unpack_u32(e, &profile->path_flags, "path_flags")) + if (aa_unpack_u32(e, &profile->path_flags, "path_flags")) profile->path_flags |= profile->label.flags & PATH_MEDIATE_DELETED; else @@ -783,38 +757,38 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) profile->path_flags = PATH_MEDIATE_DELETED; info = "failed to unpack profile capabilities"; - if (!unpack_u32(e, &(profile->caps.allow.cap[0]), NULL)) + if (!aa_unpack_u32(e, &(profile->caps.allow.cap[0]), NULL)) goto fail; - if (!unpack_u32(e, &(profile->caps.audit.cap[0]), NULL)) + if (!aa_unpack_u32(e, &(profile->caps.audit.cap[0]), NULL)) goto fail; - if (!unpack_u32(e, &(profile->caps.quiet.cap[0]), NULL)) + if (!aa_unpack_u32(e, &(profile->caps.quiet.cap[0]), NULL)) goto fail; - if (!unpack_u32(e, &tmpcap.cap[0], NULL)) + if (!aa_unpack_u32(e, &tmpcap.cap[0], NULL)) goto fail; info = "failed to unpack upper profile capabilities"; - if (unpack_nameX(e, AA_STRUCT, "caps64")) { + if (aa_unpack_nameX(e, AA_STRUCT, "caps64")) { /* optional upper half of 64 bit caps */ - if (!unpack_u32(e, &(profile->caps.allow.cap[1]), NULL)) + if (!aa_unpack_u32(e, &(profile->caps.allow.cap[1]), NULL)) goto fail; - if (!unpack_u32(e, &(profile->caps.audit.cap[1]), NULL)) + if (!aa_unpack_u32(e, &(profile->caps.audit.cap[1]), NULL)) goto fail; - if (!unpack_u32(e, &(profile->caps.quiet.cap[1]), NULL)) + if (!aa_unpack_u32(e, &(profile->caps.quiet.cap[1]), NULL)) goto fail; - if (!unpack_u32(e, &(tmpcap.cap[1]), NULL)) + if (!aa_unpack_u32(e, &(tmpcap.cap[1]), NULL)) goto fail; - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) goto fail; } info = "failed to unpack extended profile capabilities"; - if (unpack_nameX(e, AA_STRUCT, "capsx")) { + if (aa_unpack_nameX(e, AA_STRUCT, "capsx")) { /* optional extended caps mediation mask */ - if (!unpack_u32(e, &(profile->caps.extended.cap[0]), NULL)) + if (!aa_unpack_u32(e, &(profile->caps.extended.cap[0]), NULL)) goto fail; - if (!unpack_u32(e, &(profile->caps.extended.cap[1]), NULL)) + if (!aa_unpack_u32(e, &(profile->caps.extended.cap[1]), NULL)) goto fail; - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) goto fail; } @@ -833,7 +807,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) goto fail; } - if (unpack_nameX(e, AA_STRUCT, "policydb")) { + if (aa_unpack_nameX(e, AA_STRUCT, "policydb")) { /* generic policy dfa - optional and may be NULL */ info = "failed to unpack policydb"; profile->policy.dfa = unpack_dfa(e); @@ -845,7 +819,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) error = -EPROTO; goto fail; } - if (!unpack_u32(e, &profile->policy.start[0], "start")) + if (!aa_unpack_u32(e, &profile->policy.start[0], "start")) /* default start state */ profile->policy.start[0] = DFA_START; /* setup class index */ @@ -855,7 +829,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) profile->policy.start[0], i); } - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) goto fail; } else profile->policy.dfa = aa_get_dfa(nulldfa); @@ -868,7 +842,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) info = "failed to unpack profile file rules"; goto fail; } else if (profile->file.dfa) { - if (!unpack_u32(e, &profile->file.start, "dfa_start")) + if (!aa_unpack_u32(e, &profile->file.start, "dfa_start")) /* default start state */ profile->file.start = DFA_START; } else if (profile->policy.dfa && @@ -883,7 +857,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) goto fail; } - if (unpack_nameX(e, AA_STRUCT, "data")) { + if (aa_unpack_nameX(e, AA_STRUCT, "data")) { info = "out of memory"; profile->data = kzalloc(sizeof(*profile->data), GFP_KERNEL); if (!profile->data) @@ -901,7 +875,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) goto fail; } - while (unpack_strdup(e, &key, NULL)) { + while (aa_unpack_strdup(e, &key, NULL)) { data = kzalloc(sizeof(*data), GFP_KERNEL); if (!data) { kfree_sensitive(key); @@ -909,7 +883,7 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) } data->key = key; - data->size = unpack_blob(e, &data->data, NULL); + data->size = aa_unpack_blob(e, &data->data, NULL); data->data = kvmemdup(data->data, data->size); if (data->size && !data->data) { kfree_sensitive(data->key); @@ -921,13 +895,13 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name) profile->data->p); } - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) { + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) { info = "failed to unpack end of key, value data table"; goto fail; } } - if (!unpack_nameX(e, AA_STRUCTEND, NULL)) { + if (!aa_unpack_nameX(e, AA_STRUCTEND, NULL)) { info = "failed to unpack end of profile"; goto fail; } @@ -960,7 +934,7 @@ static int verify_header(struct aa_ext *e, int required, const char **ns) *ns = NULL; /* get the interface version */ - if (!unpack_u32(e, &e->version, "version")) { + if (!aa_unpack_u32(e, &e->version, "version")) { if (required) { audit_iface(NULL, NULL, NULL, "invalid profile format", e, error); @@ -979,7 +953,7 @@ static int verify_header(struct aa_ext *e, int required, const char **ns) } /* read the namespace if present */ - if (unpack_str(e, &name, "namespace")) { + if (aa_unpack_str(e, &name, "namespace")) { if (*name == '\0') { audit_iface(NULL, NULL, NULL, "invalid namespace name", e, error); @@ -1251,7 +1225,3 @@ fail: return error; } - -#ifdef CONFIG_SECURITY_APPARMOR_KUNIT_TEST -#include "policy_unpack_test.c" -#endif /* CONFIG_SECURITY_APPARMOR_KUNIT_TEST */ diff --git a/security/apparmor/policy_unpack_test.c b/security/apparmor/policy_unpack_test.c index 0a969b2e03db..f25cf2a023d5 100644 --- a/security/apparmor/policy_unpack_test.c +++ b/security/apparmor/policy_unpack_test.c @@ -4,6 +4,7 @@ */ #include +#include #include "include/policy.h" #include "include/policy_unpack.h" @@ -43,6 +44,8 @@ #define TEST_ARRAY_BUF_OFFSET \ (TEST_NAMED_ARRAY_BUF_OFFSET + 3 + strlen(TEST_ARRAY_NAME) + 1) +MODULE_IMPORT_NS(EXPORTED_FOR_KUNIT_TESTING); + struct policy_unpack_fixture { struct aa_ext *e; size_t e_size; @@ -125,16 +128,16 @@ static void policy_unpack_test_inbounds_when_inbounds(struct kunit *test) { struct policy_unpack_fixture *puf = test->priv; - KUNIT_EXPECT_TRUE(test, inbounds(puf->e, 0)); - KUNIT_EXPECT_TRUE(test, inbounds(puf->e, puf->e_size / 2)); - KUNIT_EXPECT_TRUE(test, inbounds(puf->e, puf->e_size)); + KUNIT_EXPECT_TRUE(test, aa_inbounds(puf->e, 0)); + KUNIT_EXPECT_TRUE(test, aa_inbounds(puf->e, puf->e_size / 2)); + KUNIT_EXPECT_TRUE(test, aa_inbounds(puf->e, puf->e_size)); } static void policy_unpack_test_inbounds_when_out_of_bounds(struct kunit *test) { struct policy_unpack_fixture *puf = test->priv; - KUNIT_EXPECT_FALSE(test, inbounds(puf->e, puf->e_size + 1)); + KUNIT_EXPECT_FALSE(test, aa_inbounds(puf->e, puf->e_size + 1)); } static void policy_unpack_test_unpack_array_with_null_name(struct kunit *test) @@ -144,7 +147,7 @@ static void policy_unpack_test_unpack_array_with_null_name(struct kunit *test) puf->e->pos += TEST_ARRAY_BUF_OFFSET; - array_size = unpack_array(puf->e, NULL); + array_size = aa_unpack_array(puf->e, NULL); KUNIT_EXPECT_EQ(test, array_size, (u16)TEST_ARRAY_SIZE); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -159,7 +162,7 @@ static void policy_unpack_test_unpack_array_with_name(struct kunit *test) puf->e->pos += TEST_NAMED_ARRAY_BUF_OFFSET; - array_size = unpack_array(puf->e, name); + array_size = aa_unpack_array(puf->e, name); KUNIT_EXPECT_EQ(test, array_size, (u16)TEST_ARRAY_SIZE); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -175,7 +178,7 @@ static void policy_unpack_test_unpack_array_out_of_bounds(struct kunit *test) puf->e->pos += TEST_NAMED_ARRAY_BUF_OFFSET; puf->e->end = puf->e->start + TEST_ARRAY_BUF_OFFSET + sizeof(u16); - array_size = unpack_array(puf->e, name); + array_size = aa_unpack_array(puf->e, name); KUNIT_EXPECT_EQ(test, array_size, 0); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -189,7 +192,7 @@ static void policy_unpack_test_unpack_blob_with_null_name(struct kunit *test) size_t size; puf->e->pos += TEST_BLOB_BUF_OFFSET; - size = unpack_blob(puf->e, &blob, NULL); + size = aa_unpack_blob(puf->e, &blob, NULL); KUNIT_ASSERT_EQ(test, size, TEST_BLOB_DATA_SIZE); KUNIT_EXPECT_TRUE(test, @@ -203,7 +206,7 @@ static void policy_unpack_test_unpack_blob_with_name(struct kunit *test) size_t size; puf->e->pos += TEST_NAMED_BLOB_BUF_OFFSET; - size = unpack_blob(puf->e, &blob, TEST_BLOB_NAME); + size = aa_unpack_blob(puf->e, &blob, TEST_BLOB_NAME); KUNIT_ASSERT_EQ(test, size, TEST_BLOB_DATA_SIZE); KUNIT_EXPECT_TRUE(test, @@ -222,7 +225,7 @@ static void policy_unpack_test_unpack_blob_out_of_bounds(struct kunit *test) puf->e->end = puf->e->start + TEST_BLOB_BUF_OFFSET + TEST_BLOB_DATA_SIZE - 1; - size = unpack_blob(puf->e, &blob, TEST_BLOB_NAME); + size = aa_unpack_blob(puf->e, &blob, TEST_BLOB_NAME); KUNIT_EXPECT_EQ(test, size, 0); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, start); @@ -235,7 +238,7 @@ static void policy_unpack_test_unpack_str_with_null_name(struct kunit *test) size_t size; puf->e->pos += TEST_STRING_BUF_OFFSET; - size = unpack_str(puf->e, &string, NULL); + size = aa_unpack_str(puf->e, &string, NULL); KUNIT_EXPECT_EQ(test, size, strlen(TEST_STRING_DATA) + 1); KUNIT_EXPECT_STREQ(test, string, TEST_STRING_DATA); @@ -247,7 +250,7 @@ static void policy_unpack_test_unpack_str_with_name(struct kunit *test) const char *string = NULL; size_t size; - size = unpack_str(puf->e, &string, TEST_STRING_NAME); + size = aa_unpack_str(puf->e, &string, TEST_STRING_NAME); KUNIT_EXPECT_EQ(test, size, strlen(TEST_STRING_DATA) + 1); KUNIT_EXPECT_STREQ(test, string, TEST_STRING_DATA); @@ -263,7 +266,7 @@ static void policy_unpack_test_unpack_str_out_of_bounds(struct kunit *test) puf->e->end = puf->e->pos + TEST_STRING_BUF_OFFSET + strlen(TEST_STRING_DATA) - 1; - size = unpack_str(puf->e, &string, TEST_STRING_NAME); + size = aa_unpack_str(puf->e, &string, TEST_STRING_NAME); KUNIT_EXPECT_EQ(test, size, 0); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, start); @@ -276,7 +279,7 @@ static void policy_unpack_test_unpack_strdup_with_null_name(struct kunit *test) size_t size; puf->e->pos += TEST_STRING_BUF_OFFSET; - size = unpack_strdup(puf->e, &string, NULL); + size = aa_unpack_strdup(puf->e, &string, NULL); KUNIT_EXPECT_EQ(test, size, strlen(TEST_STRING_DATA) + 1); KUNIT_EXPECT_FALSE(test, @@ -291,7 +294,7 @@ static void policy_unpack_test_unpack_strdup_with_name(struct kunit *test) char *string = NULL; size_t size; - size = unpack_strdup(puf->e, &string, TEST_STRING_NAME); + size = aa_unpack_strdup(puf->e, &string, TEST_STRING_NAME); KUNIT_EXPECT_EQ(test, size, strlen(TEST_STRING_DATA) + 1); KUNIT_EXPECT_FALSE(test, @@ -310,7 +313,7 @@ static void policy_unpack_test_unpack_strdup_out_of_bounds(struct kunit *test) puf->e->end = puf->e->pos + TEST_STRING_BUF_OFFSET + strlen(TEST_STRING_DATA) - 1; - size = unpack_strdup(puf->e, &string, TEST_STRING_NAME); + size = aa_unpack_strdup(puf->e, &string, TEST_STRING_NAME); KUNIT_EXPECT_EQ(test, size, 0); KUNIT_EXPECT_NULL(test, string); @@ -324,7 +327,7 @@ static void policy_unpack_test_unpack_nameX_with_null_name(struct kunit *test) puf->e->pos += TEST_U32_BUF_OFFSET; - success = unpack_nameX(puf->e, AA_U32, NULL); + success = aa_unpack_nameX(puf->e, AA_U32, NULL); KUNIT_EXPECT_TRUE(test, success); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -338,7 +341,7 @@ static void policy_unpack_test_unpack_nameX_with_wrong_code(struct kunit *test) puf->e->pos += TEST_U32_BUF_OFFSET; - success = unpack_nameX(puf->e, AA_BLOB, NULL); + success = aa_unpack_nameX(puf->e, AA_BLOB, NULL); KUNIT_EXPECT_FALSE(test, success); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -353,7 +356,7 @@ static void policy_unpack_test_unpack_nameX_with_name(struct kunit *test) puf->e->pos += TEST_NAMED_U32_BUF_OFFSET; - success = unpack_nameX(puf->e, AA_U32, name); + success = aa_unpack_nameX(puf->e, AA_U32, name); KUNIT_EXPECT_TRUE(test, success); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -368,7 +371,7 @@ static void policy_unpack_test_unpack_nameX_with_wrong_name(struct kunit *test) puf->e->pos += TEST_NAMED_U32_BUF_OFFSET; - success = unpack_nameX(puf->e, AA_U32, name); + success = aa_unpack_nameX(puf->e, AA_U32, name); KUNIT_EXPECT_FALSE(test, success); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -389,7 +392,7 @@ static void policy_unpack_test_unpack_u16_chunk_basic(struct kunit *test) */ puf->e->end += TEST_U16_DATA; - size = unpack_u16_chunk(puf->e, &chunk); + size = aa_unpack_u16_chunk(puf->e, &chunk); KUNIT_EXPECT_PTR_EQ(test, chunk, puf->e->start + TEST_U16_OFFSET + 2); @@ -406,7 +409,7 @@ static void policy_unpack_test_unpack_u16_chunk_out_of_bounds_1( puf->e->pos = puf->e->end - 1; - size = unpack_u16_chunk(puf->e, &chunk); + size = aa_unpack_u16_chunk(puf->e, &chunk); KUNIT_EXPECT_EQ(test, size, 0); KUNIT_EXPECT_NULL(test, chunk); @@ -428,7 +431,7 @@ static void policy_unpack_test_unpack_u16_chunk_out_of_bounds_2( */ puf->e->end = puf->e->pos + TEST_U16_DATA - 1; - size = unpack_u16_chunk(puf->e, &chunk); + size = aa_unpack_u16_chunk(puf->e, &chunk); KUNIT_EXPECT_EQ(test, size, 0); KUNIT_EXPECT_NULL(test, chunk); @@ -443,7 +446,7 @@ static void policy_unpack_test_unpack_u32_with_null_name(struct kunit *test) puf->e->pos += TEST_U32_BUF_OFFSET; - success = unpack_u32(puf->e, &data, NULL); + success = aa_unpack_u32(puf->e, &data, NULL); KUNIT_EXPECT_TRUE(test, success); KUNIT_EXPECT_EQ(test, data, TEST_U32_DATA); @@ -460,7 +463,7 @@ static void policy_unpack_test_unpack_u32_with_name(struct kunit *test) puf->e->pos += TEST_NAMED_U32_BUF_OFFSET; - success = unpack_u32(puf->e, &data, name); + success = aa_unpack_u32(puf->e, &data, name); KUNIT_EXPECT_TRUE(test, success); KUNIT_EXPECT_EQ(test, data, TEST_U32_DATA); @@ -478,7 +481,7 @@ static void policy_unpack_test_unpack_u32_out_of_bounds(struct kunit *test) puf->e->pos += TEST_NAMED_U32_BUF_OFFSET; puf->e->end = puf->e->start + TEST_U32_BUF_OFFSET + sizeof(u32); - success = unpack_u32(puf->e, &data, name); + success = aa_unpack_u32(puf->e, &data, name); KUNIT_EXPECT_FALSE(test, success); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -493,7 +496,7 @@ static void policy_unpack_test_unpack_u64_with_null_name(struct kunit *test) puf->e->pos += TEST_U64_BUF_OFFSET; - success = unpack_u64(puf->e, &data, NULL); + success = aa_unpack_u64(puf->e, &data, NULL); KUNIT_EXPECT_TRUE(test, success); KUNIT_EXPECT_EQ(test, data, TEST_U64_DATA); @@ -510,7 +513,7 @@ static void policy_unpack_test_unpack_u64_with_name(struct kunit *test) puf->e->pos += TEST_NAMED_U64_BUF_OFFSET; - success = unpack_u64(puf->e, &data, name); + success = aa_unpack_u64(puf->e, &data, name); KUNIT_EXPECT_TRUE(test, success); KUNIT_EXPECT_EQ(test, data, TEST_U64_DATA); @@ -528,7 +531,7 @@ static void policy_unpack_test_unpack_u64_out_of_bounds(struct kunit *test) puf->e->pos += TEST_NAMED_U64_BUF_OFFSET; puf->e->end = puf->e->start + TEST_U64_BUF_OFFSET + sizeof(u64); - success = unpack_u64(puf->e, &data, name); + success = aa_unpack_u64(puf->e, &data, name); KUNIT_EXPECT_FALSE(test, success); KUNIT_EXPECT_PTR_EQ(test, puf->e->pos, @@ -538,7 +541,7 @@ static void policy_unpack_test_unpack_u64_out_of_bounds(struct kunit *test) static void policy_unpack_test_unpack_X_code_match(struct kunit *test) { struct policy_unpack_fixture *puf = test->priv; - bool success = unpack_X(puf->e, AA_NAME); + bool success = aa_unpack_X(puf->e, AA_NAME); KUNIT_EXPECT_TRUE(test, success); KUNIT_EXPECT_TRUE(test, puf->e->pos == puf->e->start + 1); @@ -547,7 +550,7 @@ static void policy_unpack_test_unpack_X_code_match(struct kunit *test) static void policy_unpack_test_unpack_X_code_mismatch(struct kunit *test) { struct policy_unpack_fixture *puf = test->priv; - bool success = unpack_X(puf->e, AA_STRING); + bool success = aa_unpack_X(puf->e, AA_STRING); KUNIT_EXPECT_FALSE(test, success); KUNIT_EXPECT_TRUE(test, puf->e->pos == puf->e->start); @@ -559,7 +562,7 @@ static void policy_unpack_test_unpack_X_out_of_bounds(struct kunit *test) bool success; puf->e->pos = puf->e->end; - success = unpack_X(puf->e, AA_NAME); + success = aa_unpack_X(puf->e, AA_NAME); KUNIT_EXPECT_FALSE(test, success); } @@ -605,3 +608,5 @@ static struct kunit_suite apparmor_policy_unpack_test_module = { }; kunit_test_suite(apparmor_policy_unpack_test_module); + +MODULE_LICENSE("GPL"); From 054be257f28ca8eeb8e3620766501b81ceb4b293 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Wed, 7 Dec 2022 13:03:02 +0000 Subject: [PATCH 29/29] Documentation: dev-tools: Clarify requirements for result description Currently the KTAP specification says that a test result line is [][ # [] []] and the description of a test can be "any sequence of words (can't include #)" which specifies that there may be more than one word but does not specify anything other than those words which might be used to separate the words which probably isn't what we want. Given that practically we have tests using a range of separators for words including combinations of spaces and combinations of other symbols like underscores or punctuation let's just clarify that the description can contain any character other than # (marking the start of the directive/diagnostic) or newline (marking the end of this test result). Signed-off-by: Mark Brown Reviewed-by: Kees Cook Reviewed-by: David Gow Signed-off-by: Shuah Khan --- Documentation/dev-tools/ktap.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst index d0a9565b0f44..414c105b10a9 100644 --- a/Documentation/dev-tools/ktap.rst +++ b/Documentation/dev-tools/ktap.rst @@ -80,8 +80,8 @@ have the number 1 and the number then must increase by 1 for each additional subtest within the same test at the same nesting level. The description is a description of the test, generally the name of -the test, and can be any string of words (can't include #). The -description is optional, but recommended. +the test, and can be any string of characters other than # or a +newline. The description is optional, but recommended. The directive and any diagnostic data is optional. If either are present, they must follow a hash sign, "#".