mirrors
/
linux-stable
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1
0
Fork 0
Code Issues Releases Wiki Activity
linux-stable/lib/test_xarray.c

2177 lines
56 KiB
C
Raw Permalink Normal View History

xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
// SPDX-License-Identifier: GPL-2.0+
/*
* test_xarray.c: Test the XArray API
* Copyright (c) 2017-2018 Microsoft Corporation
XArray: Fix xas_find returning too many entries If you call xas_find() with the initial index > max, it should have returned NULL but was returning the entry at index. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:13:21 +00:00
* Copyright (c) 2019-2020 Oracle
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
* Author: Matthew Wilcox <willy@infradead.org>
*/
#include <linux/xarray.h>
#include <linux/module.h>
static unsigned int tests_run;
static unsigned int tests_passed;
XArray: Fix xa_find_next for large multi-index entries Coverity pointed out that xas_sibling() was shifting xa_offset without promoting it to an unsigned long first, so the shift could cause an overflow and we'd get the wrong answer. The fix is obvious, and the new test-case provokes UBSAN to report an error: runtime error: shift exponent 60 is too large for 32-bit type 'int' Fixes: 19c30f4dd092 ("XArray: Fix xa_find_after with multi-index entries") Reported-by: Bjorn Helgaas <bhelgaas@google.com> Reported-by: Kees Cook <keescook@chromium.org> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-31 10:07:55 +00:00
static const unsigned int order_limit =
IS_ENABLED(CONFIG_XARRAY_MULTI) ? BITS_PER_LONG : 1;
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
#ifndef XA_DEBUG
# ifdef __KERNEL__
void xa_dump(const struct xarray *xa) { }
# endif
#undef XA_BUG_ON
#define XA_BUG_ON(xa, x) do { \
tests_run++; \
if (x) { \
printk("BUG at %s:%d\n", __func__, __LINE__); \
xa_dump(xa); \
dump_stack(); \
} else { \
tests_passed++; \
} \
} while (0)
#endif
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
static void *xa_mk_index(unsigned long index)
{
return xa_mk_value(index & LONG_MAX);
}
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
static void *xa_store_index(struct xarray *xa, unsigned long index, gfp_t gfp)
{
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
return xa_store(xa, index, xa_mk_index(index), gfp);
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
}
XArray tests: Add check_insert A simple test which just checks that inserting an entry into an empty array succeeds. Try various different interesting indices. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-03-10 03:25:27 +00:00
static void xa_insert_index(struct xarray *xa, unsigned long index)
{
XA_BUG_ON(xa, xa_insert(xa, index, xa_mk_index(index),
GFP_KERNEL) != 0);
}
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
static void xa_alloc_index(struct xarray *xa, unsigned long index, gfp_t gfp)
{
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
u32 id;
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
XA_BUG_ON(xa, xa_alloc(xa, &id, xa_mk_index(index), xa_limit_32b,
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
gfp) != 0);
XA_BUG_ON(xa, id != index);
}
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
static void xa_erase_index(struct xarray *xa, unsigned long index)
{
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, xa_erase(xa, index) != xa_mk_index(index));
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
XA_BUG_ON(xa, xa_load(xa, index) != NULL);
}
/*
* If anyone needs this, please move it to xarray.c. We have no current
* users outside the test suite because all current multislot users want
* to use the advanced API.
*/
static void *xa_store_order(struct xarray *xa, unsigned long index,
unsigned order, void *entry, gfp_t gfp)
{
XA_STATE_ORDER(xas, xa, index, order);
void *curr;
do {
xas_lock(&xas);
curr = xas_store(&xas, entry);
xas_unlock(&xas);
} while (xas_nomem(&xas, gfp));
return curr;
}
static noinline void check_xa_err(struct xarray *xa)
{
XA_BUG_ON(xa, xa_err(xa_store_index(xa, 0, GFP_NOWAIT)) != 0);
XA_BUG_ON(xa, xa_err(xa_erase(xa, 0)) != 0);
#ifndef __KERNEL__
/* The kernel does not fail GFP_NOWAIT allocations */
XA_BUG_ON(xa, xa_err(xa_store_index(xa, 1, GFP_NOWAIT)) != -ENOMEM);
XA_BUG_ON(xa, xa_err(xa_store_index(xa, 1, GFP_NOWAIT)) != -ENOMEM);
#endif
XA_BUG_ON(xa, xa_err(xa_store_index(xa, 1, GFP_KERNEL)) != 0);
XA_BUG_ON(xa, xa_err(xa_store(xa, 1, xa_mk_value(0), GFP_KERNEL)) != 0);
XA_BUG_ON(xa, xa_err(xa_erase(xa, 1)) != 0);
// kills the test-suite :-(
// XA_BUG_ON(xa, xa_err(xa_store(xa, 0, xa_mk_internal(0), 0)) != -EINVAL);
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
}
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
static noinline void check_xas_retry(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
void *entry;
xa_store_index(xa, 0, GFP_KERNEL);
xa_store_index(xa, 1, GFP_KERNEL);
rcu_read_lock();
XA_BUG_ON(xa, xas_find(&xas, ULONG_MAX) != xa_mk_value(0));
xa_erase_index(xa, 1);
XA_BUG_ON(xa, !xa_is_retry(xas_reload(&xas)));
XA_BUG_ON(xa, xas_retry(&xas, NULL));
XA_BUG_ON(xa, xas_retry(&xas, xa_mk_value(0)));
xas_reset(&xas);
XA_BUG_ON(xa, xas.xa_node != XAS_RESTART);
XA_BUG_ON(xa, xas_next_entry(&xas, ULONG_MAX) != xa_mk_value(0));
XA_BUG_ON(xa, xas.xa_node != NULL);
XArray tests: RCU lock prohibits GFP_KERNEL Drop and reacquire the RCU read lock while using GFP_KERNEL. Reported-by: Li RongQing <lirongqing@baidu.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-05 04:12:08 +00:00
rcu_read_unlock();
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
XA_BUG_ON(xa, xa_store_index(xa, 1, GFP_KERNEL) != NULL);
XArray tests: RCU lock prohibits GFP_KERNEL Drop and reacquire the RCU read lock while using GFP_KERNEL. Reported-by: Li RongQing <lirongqing@baidu.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-05 04:12:08 +00:00
rcu_read_lock();
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
XA_BUG_ON(xa, !xa_is_internal(xas_reload(&xas)));
xas.xa_node = XAS_RESTART;
XA_BUG_ON(xa, xas_next_entry(&xas, ULONG_MAX) != xa_mk_value(0));
rcu_read_unlock();
/* Make sure we can iterate through retry entries */
xas_lock(&xas);
xas_set(&xas, 0);
xas_store(&xas, XA_RETRY_ENTRY);
xas_set(&xas, 1);
xas_store(&xas, XA_RETRY_ENTRY);
xas_set(&xas, 0);
xas_for_each(&xas, entry, ULONG_MAX) {
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xas_store(&xas, xa_mk_index(xas.xa_index));
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
}
xas_unlock(&xas);
xa_erase_index(xa, 0);
xa_erase_index(xa, 1);
}
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
static noinline void check_xa_load(struct xarray *xa)
{
unsigned long i, j;
for (i = 0; i < 1024; i++) {
for (j = 0; j < 1024; j++) {
void *entry = xa_load(xa, j);
if (j < i)
XA_BUG_ON(xa, xa_to_value(entry) != j);
else
XA_BUG_ON(xa, entry);
}
XA_BUG_ON(xa, xa_store_index(xa, i, GFP_KERNEL) != NULL);
}
for (i = 0; i < 1024; i++) {
for (j = 0; j < 1024; j++) {
void *entry = xa_load(xa, j);
if (j >= i)
XA_BUG_ON(xa, xa_to_value(entry) != j);
else
XA_BUG_ON(xa, entry);
}
xa_erase_index(xa, i);
}
XA_BUG_ON(xa, !xa_empty(xa));
}
xarray: Add XArray marks XArray marks are like the radix tree tags, only slightly more strongly typed. They are renamed in order to distinguish them from tagged pointers. This commit adds the basic get/set/clear operations. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 14:34:31 +00:00
static noinline void check_xa_mark_1(struct xarray *xa, unsigned long index)
{
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
unsigned int order;
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 8 : 1;
xarray: Add XArray marks XArray marks are like the radix tree tags, only slightly more strongly typed. They are renamed in order to distinguish them from tagged pointers. This commit adds the basic get/set/clear operations. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 14:34:31 +00:00
/* NULL elements have no marks set */
XA_BUG_ON(xa, xa_get_mark(xa, index, XA_MARK_0));
xa_set_mark(xa, index, XA_MARK_0);
XA_BUG_ON(xa, xa_get_mark(xa, index, XA_MARK_0));
/* Storing a pointer will not make a mark appear */
XA_BUG_ON(xa, xa_store_index(xa, index, GFP_KERNEL) != NULL);
XA_BUG_ON(xa, xa_get_mark(xa, index, XA_MARK_0));
xa_set_mark(xa, index, XA_MARK_0);
XA_BUG_ON(xa, !xa_get_mark(xa, index, XA_MARK_0));
/* Setting one mark will not set another mark */
XA_BUG_ON(xa, xa_get_mark(xa, index + 1, XA_MARK_0));
XA_BUG_ON(xa, xa_get_mark(xa, index, XA_MARK_1));
/* Storing NULL clears marks, and they can't be set again */
xa_erase_index(xa, index);
XA_BUG_ON(xa, !xa_empty(xa));
XA_BUG_ON(xa, xa_get_mark(xa, index, XA_MARK_0));
xa_set_mark(xa, index, XA_MARK_0);
XA_BUG_ON(xa, xa_get_mark(xa, index, XA_MARK_0));
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
/*
* Storing a multi-index entry over entries with marks gives the
* entire entry the union of the marks
*/
BUG_ON((index % 4) != 0);
for (order = 2; order < max_order; order++) {
unsigned long base = round_down(index, 1UL << order);
unsigned long next = base + (1UL << order);
unsigned long i;
XA_BUG_ON(xa, xa_store_index(xa, index + 1, GFP_KERNEL));
xa_set_mark(xa, index + 1, XA_MARK_0);
XA_BUG_ON(xa, xa_store_index(xa, index + 2, GFP_KERNEL));
XArray tests: Check mark 2 gets squashed We do not currently check that the loop in xas_squash_marks() doesn't have an off-by-one error in it. It didn't, but a patch which introduced an off-by-one error wasn't caught by any existing test. Switch the roles of XA_MARK_1 and XA_MARK_2 to catch that bug. Reported-by: Cyrill Gorcunov <gorcunov@gmail.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-01-14 18:57:31 +00:00
xa_set_mark(xa, index + 2, XA_MARK_2);
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
XA_BUG_ON(xa, xa_store_index(xa, next, GFP_KERNEL));
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_store_order(xa, index, order, xa_mk_index(index),
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
GFP_KERNEL);
for (i = base; i < next; i++) {
xarray: Move multiorder_shrink to kernel tests Test this functionality inside the kernel as well as in userspace. Also remove insert_bug() as there's no comparable thing to test in the XArray code. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-08 16:09:52 +00:00
XA_STATE(xas, xa, i);
unsigned int seen = 0;
void *entry;
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
XA_BUG_ON(xa, !xa_get_mark(xa, i, XA_MARK_0));
XArray tests: Check mark 2 gets squashed We do not currently check that the loop in xas_squash_marks() doesn't have an off-by-one error in it. It didn't, but a patch which introduced an off-by-one error wasn't caught by any existing test. Switch the roles of XA_MARK_1 and XA_MARK_2 to catch that bug. Reported-by: Cyrill Gorcunov <gorcunov@gmail.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-01-14 18:57:31 +00:00
XA_BUG_ON(xa, xa_get_mark(xa, i, XA_MARK_1));
XA_BUG_ON(xa, !xa_get_mark(xa, i, XA_MARK_2));
xarray: Move multiorder_shrink to kernel tests Test this functionality inside the kernel as well as in userspace. Also remove insert_bug() as there's no comparable thing to test in the XArray code. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-08 16:09:52 +00:00
/* We should see two elements in the array */
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
rcu_read_lock();
xarray: Move multiorder_shrink to kernel tests Test this functionality inside the kernel as well as in userspace. Also remove insert_bug() as there's no comparable thing to test in the XArray code. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-08 16:09:52 +00:00
xas_for_each(&xas, entry, ULONG_MAX)
seen++;
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
rcu_read_unlock();
xarray: Move multiorder_shrink to kernel tests Test this functionality inside the kernel as well as in userspace. Also remove insert_bug() as there's no comparable thing to test in the XArray code. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-08 16:09:52 +00:00
XA_BUG_ON(xa, seen != 2);
/* One of which is marked */
xas_set(&xas, 0);
seen = 0;
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
rcu_read_lock();
xarray: Move multiorder_shrink to kernel tests Test this functionality inside the kernel as well as in userspace. Also remove insert_bug() as there's no comparable thing to test in the XArray code. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-08 16:09:52 +00:00
xas_for_each_marked(&xas, entry, ULONG_MAX, XA_MARK_0)
seen++;
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
rcu_read_unlock();
xarray: Move multiorder_shrink to kernel tests Test this functionality inside the kernel as well as in userspace. Also remove insert_bug() as there's no comparable thing to test in the XArray code. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-08 16:09:52 +00:00
XA_BUG_ON(xa, seen != 1);
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
}
XA_BUG_ON(xa, xa_get_mark(xa, next, XA_MARK_0));
XA_BUG_ON(xa, xa_get_mark(xa, next, XA_MARK_1));
XA_BUG_ON(xa, xa_get_mark(xa, next, XA_MARK_2));
xa_erase_index(xa, index);
xa_erase_index(xa, next);
XA_BUG_ON(xa, !xa_empty(xa));
}
XA_BUG_ON(xa, !xa_empty(xa));
xarray: Add XArray marks XArray marks are like the radix tree tags, only slightly more strongly typed. They are renamed in order to distinguish them from tagged pointers. This commit adds the basic get/set/clear operations. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 14:34:31 +00:00
}
radix tree: Remove radix_tree_clear_tags The page cache was the only user of this interface and it has now been converted to the XArray. Transform the test into a test of xas_init_marks(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-04-09 20:52:21 +00:00
static noinline void check_xa_mark_2(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
unsigned long index;
unsigned int count = 0;
void *entry;
xa_store_index(xa, 0, GFP_KERNEL);
xa_set_mark(xa, 0, XA_MARK_0);
xas_lock(&xas);
xas_load(&xas);
xas_init_marks(&xas);
xas_unlock(&xas);
XA_BUG_ON(xa, !xa_get_mark(xa, 0, XA_MARK_0) == 0);
for (index = 3500; index < 4500; index++) {
xa_store_index(xa, index, GFP_KERNEL);
xa_set_mark(xa, index, XA_MARK_0);
}
xas_reset(&xas);
rcu_read_lock();
xas_for_each_marked(&xas, entry, ULONG_MAX, XA_MARK_0)
count++;
rcu_read_unlock();
XA_BUG_ON(xa, count != 1000);
xas_lock(&xas);
xas_for_each(&xas, entry, ULONG_MAX) {
xas_init_marks(&xas);
XA_BUG_ON(xa, !xa_get_mark(xa, xas.xa_index, XA_MARK_0));
XA_BUG_ON(xa, !xas_get_mark(&xas, XA_MARK_0));
}
xas_unlock(&xas);
xa_destroy(xa);
}
XArray: Test marked multiorder iterations Demonstrate that starting a marked iteration partway through a marked multi-order entry works. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-06-15 01:52:04 +00:00
static noinline void check_xa_mark_3(struct xarray *xa)
{
#ifdef CONFIG_XARRAY_MULTI
XA_STATE(xas, xa, 0x41);
void *entry;
int count = 0;
xa_store_order(xa, 0x40, 2, xa_mk_index(0x40), GFP_KERNEL);
xa_set_mark(xa, 0x41, XA_MARK_0);
rcu_read_lock();
xas_for_each_marked(&xas, entry, ULONG_MAX, XA_MARK_0) {
count++;
XA_BUG_ON(xa, entry != xa_mk_index(0x40));
}
XA_BUG_ON(xa, count != 1);
rcu_read_unlock();
xa_destroy(xa);
#endif
}
xarray: Add XArray marks XArray marks are like the radix tree tags, only slightly more strongly typed. They are renamed in order to distinguish them from tagged pointers. This commit adds the basic get/set/clear operations. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 14:34:31 +00:00
static noinline void check_xa_mark(struct xarray *xa)
{
unsigned long index;
for (index = 0; index < 16384; index += 4)
check_xa_mark_1(xa, index);
radix tree: Remove radix_tree_clear_tags The page cache was the only user of this interface and it has now been converted to the XArray. Transform the test into a test of xas_init_marks(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-04-09 20:52:21 +00:00
check_xa_mark_2(xa);
XArray: Test marked multiorder iterations Demonstrate that starting a marked iteration partway through a marked multi-order entry works. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-06-15 01:52:04 +00:00
check_xa_mark_3(xa);
xarray: Add XArray marks XArray marks are like the radix tree tags, only slightly more strongly typed. They are renamed in order to distinguish them from tagged pointers. This commit adds the basic get/set/clear operations. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 14:34:31 +00:00
}
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
static noinline void check_xa_shrink(struct xarray *xa)
{
XA_STATE(xas, xa, 1);
struct xa_node *node;
xarray: Move multiorder_shrink to kernel tests Test this functionality inside the kernel as well as in userspace. Also remove insert_bug() as there's no comparable thing to test in the XArray code. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-08 16:09:52 +00:00
unsigned int order;
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 15 : 1;
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
XA_BUG_ON(xa, !xa_empty(xa));
XA_BUG_ON(xa, xa_store_index(xa, 0, GFP_KERNEL) != NULL);
XA_BUG_ON(xa, xa_store_index(xa, 1, GFP_KERNEL) != NULL);
/*
* Check that erasing the entry at 1 shrinks the tree and properly
* marks the node as being deleted.
*/
xas_lock(&xas);
XA_BUG_ON(xa, xas_load(&xas) != xa_mk_value(1));
node = xas.xa_node;
XA_BUG_ON(xa, xa_entry_locked(xa, node, 0) != xa_mk_value(0));
XA_BUG_ON(xa, xas_store(&xas, NULL) != xa_mk_value(1));
XA_BUG_ON(xa, xa_load(xa, 1) != NULL);
XA_BUG_ON(xa, xas.xa_node != XAS_BOUNDS);
XA_BUG_ON(xa, xa_entry_locked(xa, node, 0) != XA_RETRY_ENTRY);
XA_BUG_ON(xa, xas_load(&xas) != NULL);
xas_unlock(&xas);
XA_BUG_ON(xa, xa_load(xa, 0) != xa_mk_value(0));
xa_erase_index(xa, 0);
XA_BUG_ON(xa, !xa_empty(xa));
xarray: Move multiorder_shrink to kernel tests Test this functionality inside the kernel as well as in userspace. Also remove insert_bug() as there's no comparable thing to test in the XArray code. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-08 16:09:52 +00:00
for (order = 0; order < max_order; order++) {
unsigned long max = (1UL << order) - 1;
xa_store_order(xa, 0, order, xa_mk_value(0), GFP_KERNEL);
XA_BUG_ON(xa, xa_load(xa, max) != xa_mk_value(0));
XA_BUG_ON(xa, xa_load(xa, max + 1) != NULL);
rcu_read_lock();
node = xa_head(xa);
rcu_read_unlock();
XA_BUG_ON(xa, xa_store_index(xa, ULONG_MAX, GFP_KERNEL) !=
NULL);
rcu_read_lock();
XA_BUG_ON(xa, xa_head(xa) == node);
rcu_read_unlock();
XA_BUG_ON(xa, xa_load(xa, max + 1) != NULL);
xa_erase_index(xa, ULONG_MAX);
XA_BUG_ON(xa, xa->xa_head != node);
xa_erase_index(xa, 0);
}
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
}
XArray tests: Add check_insert A simple test which just checks that inserting an entry into an empty array succeeds. Try various different interesting indices. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-03-10 03:25:27 +00:00
static noinline void check_insert(struct xarray *xa)
{
unsigned long i;
for (i = 0; i < 1024; i++) {
xa_insert_index(xa, i);
XA_BUG_ON(xa, xa_load(xa, i - 1) != NULL);
XA_BUG_ON(xa, xa_load(xa, i + 1) != NULL);
xa_erase_index(xa, i);
}
for (i = 10; i < BITS_PER_LONG; i++) {
xa_insert_index(xa, 1UL << i);
XA_BUG_ON(xa, xa_load(xa, (1UL << i) - 1) != NULL);
XA_BUG_ON(xa, xa_load(xa, (1UL << i) + 1) != NULL);
xa_erase_index(xa, 1UL << i);
xa_insert_index(xa, (1UL << i) - 1);
XA_BUG_ON(xa, xa_load(xa, (1UL << i) - 2) != NULL);
XA_BUG_ON(xa, xa_load(xa, 1UL << i) != NULL);
xa_erase_index(xa, (1UL << i) - 1);
}
xa_insert_index(xa, ~0UL);
XA_BUG_ON(xa, xa_load(xa, 0UL) != NULL);
XA_BUG_ON(xa, xa_load(xa, ~1UL) != NULL);
xa_erase_index(xa, ~0UL);
XA_BUG_ON(xa, !xa_empty(xa));
}
xarray: Add XArray conditional store operations Like cmpxchg(), xa_cmpxchg will only store to the index if the current entry matches the old entry. It returns the current entry, which is usually more useful than the errno returned by radix_tree_insert(). For the users who really only want the errno, the xa_insert() wrapper provides a more convenient calling convention. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:34:55 +00:00
static noinline void check_cmpxchg(struct xarray *xa)
{
void *FIVE = xa_mk_value(5);
void *SIX = xa_mk_value(6);
void *LOTS = xa_mk_value(12345678);
XA_BUG_ON(xa, !xa_empty(xa));
XA_BUG_ON(xa, xa_store_index(xa, 12345678, GFP_KERNEL) != NULL);
XArray: Change xa_insert to return -EBUSY Userspace translates EEXIST to "File exists" which isn't a very good error message for the problem. "Device or resource busy" is a better indication of what went wrong. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-06 18:07:11 +00:00
XA_BUG_ON(xa, xa_insert(xa, 12345678, xa, GFP_KERNEL) != -EBUSY);
xarray: Add XArray conditional store operations Like cmpxchg(), xa_cmpxchg will only store to the index if the current entry matches the old entry. It returns the current entry, which is usually more useful than the errno returned by radix_tree_insert(). For the users who really only want the errno, the xa_insert() wrapper provides a more convenient calling convention. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:34:55 +00:00
XA_BUG_ON(xa, xa_cmpxchg(xa, 12345678, SIX, FIVE, GFP_KERNEL) != LOTS);
XA_BUG_ON(xa, xa_cmpxchg(xa, 12345678, LOTS, FIVE, GFP_KERNEL) != LOTS);
XA_BUG_ON(xa, xa_cmpxchg(xa, 12345678, FIVE, LOTS, GFP_KERNEL) != FIVE);
XA_BUG_ON(xa, xa_cmpxchg(xa, 5, FIVE, NULL, GFP_KERNEL) != NULL);
XA_BUG_ON(xa, xa_cmpxchg(xa, 5, NULL, FIVE, GFP_KERNEL) != NULL);
XArray: Test two more things about xa_cmpxchg 1. If we xa_cmpxchg() an entry in, it marks the index as not free. 2. If we xa_cmpxchg() NULL in, it marks the index as free. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-03-31 18:23:59 +00:00
XA_BUG_ON(xa, xa_insert(xa, 5, FIVE, GFP_KERNEL) != -EBUSY);
XA_BUG_ON(xa, xa_cmpxchg(xa, 5, FIVE, NULL, GFP_KERNEL) != FIVE);
XA_BUG_ON(xa, xa_insert(xa, 5, FIVE, GFP_KERNEL) == -EBUSY);
xarray: Add XArray conditional store operations Like cmpxchg(), xa_cmpxchg will only store to the index if the current entry matches the old entry. It returns the current entry, which is usually more useful than the errno returned by radix_tree_insert(). For the users who really only want the errno, the xa_insert() wrapper provides a more convenient calling convention. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:34:55 +00:00
xa_erase_index(xa, 12345678);
xa_erase_index(xa, 5);
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray: add cmpxchg order test XArray multi-index entries do not keep track of the order stored once the entry is being marked as used with cmpxchg (conditionally replaced with NULL). Add a test to check the order is actually lost. The test also verifies the order and entries for all the tied indexes before and after the NULL replacement with xa_cmpxchg. Add another entry at 1 << order that keeps the node around and the order information for the NULL-entry after xa_cmpxchg. Link: https://lkml.kernel.org/r/20240131225125.1370598-3-mcgrof@kernel.org Signed-off-by: Daniel Gomez <da.gomez@samsung.com> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Cc: Darrick J. Wong <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Hannes Reinecke <hare@suse.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-01-31 22:51:25 +00:00
static noinline void check_cmpxchg_order(struct xarray *xa)
{
#ifdef CONFIG_XARRAY_MULTI
void *FIVE = xa_mk_value(5);
unsigned int i, order = 3;
XA_BUG_ON(xa, xa_store_order(xa, 0, order, FIVE, GFP_KERNEL));
/* Check entry FIVE has the order saved */
XA_BUG_ON(xa, xa_get_order(xa, xa_to_value(FIVE)) != order);
/* Check all the tied indexes have the same entry and order */
for (i = 0; i < (1 << order); i++) {
XA_BUG_ON(xa, xa_load(xa, i) != FIVE);
XA_BUG_ON(xa, xa_get_order(xa, i) != order);
}
/* Ensure that nothing is stored at index '1 << order' */
XA_BUG_ON(xa, xa_load(xa, 1 << order) != NULL);
/*
* Additionally, keep the node information and the order at
* '1 << order'
*/
XA_BUG_ON(xa, xa_store_order(xa, 1 << order, order, FIVE, GFP_KERNEL));
for (i = (1 << order); i < (1 << order) + (1 << order) - 1; i++) {
XA_BUG_ON(xa, xa_load(xa, i) != FIVE);
XA_BUG_ON(xa, xa_get_order(xa, i) != order);
}
/* Conditionally replace FIVE entry at index '0' with NULL */
XA_BUG_ON(xa, xa_cmpxchg(xa, 0, FIVE, NULL, GFP_KERNEL) != FIVE);
/* Verify the order is lost at FIVE (and old) entries */
XA_BUG_ON(xa, xa_get_order(xa, xa_to_value(FIVE)) != 0);
/* Verify the order and entries are lost in all the tied indexes */
for (i = 0; i < (1 << order); i++) {
XA_BUG_ON(xa, xa_load(xa, i) != NULL);
XA_BUG_ON(xa, xa_get_order(xa, i) != 0);
}
/* Verify node and order are kept at '1 << order' */
for (i = (1 << order); i < (1 << order) + (1 << order) - 1; i++) {
XA_BUG_ON(xa, xa_load(xa, i) != FIVE);
XA_BUG_ON(xa, xa_get_order(xa, i) != order);
}
xa_store_order(xa, 0, BITS_PER_LONG - 1, NULL, GFP_KERNEL);
XA_BUG_ON(xa, !xa_empty(xa));
#endif
}
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
static noinline void check_reserve(struct xarray *xa)
{
void *entry;
XArray: Change xa_for_each iterator There were three problems with this API: 1. It took too many arguments; almost all users wanted to iterate over every element in the array rather than a subset. 2. It required that 'index' be initialised before use, and there's no realistic way to make GCC catch that. 3. 'index' and 'entry' were the opposite way round from every other member of the XArray APIs. So split it into three different APIs: xa_for_each(xa, index, entry) xa_for_each_start(xa, index, entry, start) xa_for_each_marked(xa, index, entry, filter) Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-17 19:45:36 +00:00
unsigned long index;
XArray: Fix xa_release in allocating arrays xa_cmpxchg() was a little too magic in turning ZERO entries into NULL, and would leave the entry set to the ZERO entry instead of releasing it for future use. After careful review of existing users of xa_cmpxchg(), change the semantics so that it does not translate either incoming argument from NULL into ZERO entries. Add several tests to the test-suite to make sure this problem doesn't come back. Reported-by: Jason Gunthorpe <jgg@ziepe.ca> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-20 16:30:49 +00:00
int count;
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
/* An array with a reserved entry is not empty */
XA_BUG_ON(xa, !xa_empty(xa));
XArray: Mark xa_insert and xa_reserve as must_check If the user doesn't care about the return value from xa_insert(), then they should be using xa_store() instead. The point of xa_reserve() is to get the return value early before taking another lock, so this should also be __must_check. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-08 19:02:45 +00:00
XA_BUG_ON(xa, xa_reserve(xa, 12345678, GFP_KERNEL) != 0);
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
XA_BUG_ON(xa, xa_empty(xa));
XA_BUG_ON(xa, xa_load(xa, 12345678));
xa_release(xa, 12345678);
XA_BUG_ON(xa, !xa_empty(xa));
/* Releasing a used entry does nothing */
XArray: Mark xa_insert and xa_reserve as must_check If the user doesn't care about the return value from xa_insert(), then they should be using xa_store() instead. The point of xa_reserve() is to get the return value early before taking another lock, so this should also be __must_check. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-08 19:02:45 +00:00
XA_BUG_ON(xa, xa_reserve(xa, 12345678, GFP_KERNEL) != 0);
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
XA_BUG_ON(xa, xa_store_index(xa, 12345678, GFP_NOWAIT) != NULL);
xa_release(xa, 12345678);
xa_erase_index(xa, 12345678);
XA_BUG_ON(xa, !xa_empty(xa));
XArray: Fix xa_release in allocating arrays xa_cmpxchg() was a little too magic in turning ZERO entries into NULL, and would leave the entry set to the ZERO entry instead of releasing it for future use. After careful review of existing users of xa_cmpxchg(), change the semantics so that it does not translate either incoming argument from NULL into ZERO entries. Add several tests to the test-suite to make sure this problem doesn't come back. Reported-by: Jason Gunthorpe <jgg@ziepe.ca> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-20 16:30:49 +00:00
/* cmpxchg sees a reserved entry as ZERO */
XArray: Mark xa_insert and xa_reserve as must_check If the user doesn't care about the return value from xa_insert(), then they should be using xa_store() instead. The point of xa_reserve() is to get the return value early before taking another lock, so this should also be __must_check. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-08 19:02:45 +00:00
XA_BUG_ON(xa, xa_reserve(xa, 12345678, GFP_KERNEL) != 0);
XArray: Fix xa_release in allocating arrays xa_cmpxchg() was a little too magic in turning ZERO entries into NULL, and would leave the entry set to the ZERO entry instead of releasing it for future use. After careful review of existing users of xa_cmpxchg(), change the semantics so that it does not translate either incoming argument from NULL into ZERO entries. Add several tests to the test-suite to make sure this problem doesn't come back. Reported-by: Jason Gunthorpe <jgg@ziepe.ca> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-20 16:30:49 +00:00
XA_BUG_ON(xa, xa_cmpxchg(xa, 12345678, XA_ZERO_ENTRY,
xa_mk_value(12345678), GFP_NOWAIT) != NULL);
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
xa_release(xa, 12345678);
xa_erase_index(xa, 12345678);
XA_BUG_ON(xa, !xa_empty(xa));
XArray: Fix xa_release in allocating arrays xa_cmpxchg() was a little too magic in turning ZERO entries into NULL, and would leave the entry set to the ZERO entry instead of releasing it for future use. After careful review of existing users of xa_cmpxchg(), change the semantics so that it does not translate either incoming argument from NULL into ZERO entries. Add several tests to the test-suite to make sure this problem doesn't come back. Reported-by: Jason Gunthorpe <jgg@ziepe.ca> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-20 16:30:49 +00:00
/* xa_insert treats it as busy */
XArray: Mark xa_insert and xa_reserve as must_check If the user doesn't care about the return value from xa_insert(), then they should be using xa_store() instead. The point of xa_reserve() is to get the return value early before taking another lock, so this should also be __must_check. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-08 19:02:45 +00:00
XA_BUG_ON(xa, xa_reserve(xa, 12345678, GFP_KERNEL) != 0);
XArray: Honour reserved entries in xa_insert xa_insert() should treat reserved entries as occupied, not as available. Also, it should treat requests to insert a NULL pointer as a request to reserve the slot. Add xa_insert_bh() and xa_insert_irq() for completeness. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-01-02 18:57:03 +00:00
XA_BUG_ON(xa, xa_insert(xa, 12345678, xa_mk_value(12345678), 0) !=
XArray: Change xa_insert to return -EBUSY Userspace translates EEXIST to "File exists" which isn't a very good error message for the problem. "Device or resource busy" is a better indication of what went wrong. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-06 18:07:11 +00:00
-EBUSY);
XArray: Honour reserved entries in xa_insert xa_insert() should treat reserved entries as occupied, not as available. Also, it should treat requests to insert a NULL pointer as a request to reserve the slot. Add xa_insert_bh() and xa_insert_irq() for completeness. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-01-02 18:57:03 +00:00
XA_BUG_ON(xa, xa_empty(xa));
XA_BUG_ON(xa, xa_erase(xa, 12345678) != NULL);
XArray: Regularise xa_reserve The xa_reserve() function was a little unusual in that it attempted to be callable for all kinds of locking scenarios. Make it look like the other APIs with __xa_reserve, xa_reserve_bh and xa_reserve_irq variants. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-30 13:45:55 +00:00
XA_BUG_ON(xa, !xa_empty(xa));
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
/* Can iterate through a reserved entry */
xa_store_index(xa, 5, GFP_KERNEL);
XArray: Mark xa_insert and xa_reserve as must_check If the user doesn't care about the return value from xa_insert(), then they should be using xa_store() instead. The point of xa_reserve() is to get the return value early before taking another lock, so this should also be __must_check. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-08 19:02:45 +00:00
XA_BUG_ON(xa, xa_reserve(xa, 6, GFP_KERNEL) != 0);
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
xa_store_index(xa, 7, GFP_KERNEL);
XArray: Fix xa_release in allocating arrays xa_cmpxchg() was a little too magic in turning ZERO entries into NULL, and would leave the entry set to the ZERO entry instead of releasing it for future use. After careful review of existing users of xa_cmpxchg(), change the semantics so that it does not translate either incoming argument from NULL into ZERO entries. Add several tests to the test-suite to make sure this problem doesn't come back. Reported-by: Jason Gunthorpe <jgg@ziepe.ca> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-20 16:30:49 +00:00
count = 0;
XArray: Change xa_for_each iterator There were three problems with this API: 1. It took too many arguments; almost all users wanted to iterate over every element in the array rather than a subset. 2. It required that 'index' be initialised before use, and there's no realistic way to make GCC catch that. 3. 'index' and 'entry' were the opposite way round from every other member of the XArray APIs. So split it into three different APIs: xa_for_each(xa, index, entry) xa_for_each_start(xa, index, entry, start) xa_for_each_marked(xa, index, entry, filter) Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-17 19:45:36 +00:00
xa_for_each(xa, index, entry) {
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
XA_BUG_ON(xa, index != 5 && index != 7);
XArray: Fix xa_release in allocating arrays xa_cmpxchg() was a little too magic in turning ZERO entries into NULL, and would leave the entry set to the ZERO entry instead of releasing it for future use. After careful review of existing users of xa_cmpxchg(), change the semantics so that it does not translate either incoming argument from NULL into ZERO entries. Add several tests to the test-suite to make sure this problem doesn't come back. Reported-by: Jason Gunthorpe <jgg@ziepe.ca> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-20 16:30:49 +00:00
count++;
}
XA_BUG_ON(xa, count != 2);
/* If we free a reserved entry, we should be able to allocate it */
if (xa->xa_flags & XA_FLAGS_ALLOC) {
u32 id;
XA_BUG_ON(xa, xa_alloc(xa, &id, xa_mk_value(8),
XA_LIMIT(5, 10), GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != 8);
xa_release(xa, 6);
XA_BUG_ON(xa, xa_alloc(xa, &id, xa_mk_value(6),
XA_LIMIT(5, 10), GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != 6);
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
}
XArray: Fix xa_release in allocating arrays xa_cmpxchg() was a little too magic in turning ZERO entries into NULL, and would leave the entry set to the ZERO entry instead of releasing it for future use. After careful review of existing users of xa_cmpxchg(), change the semantics so that it does not translate either incoming argument from NULL into ZERO entries. Add several tests to the test-suite to make sure this problem doesn't come back. Reported-by: Jason Gunthorpe <jgg@ziepe.ca> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-20 16:30:49 +00:00
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
xa_destroy(xa);
}
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
static noinline void check_xas_erase(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
void *entry;
unsigned long i, j;
for (i = 0; i < 200; i++) {
for (j = i; j < 2 * i + 17; j++) {
xas_set(&xas, j);
do {
xas_lock(&xas);
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xas_store(&xas, xa_mk_index(j));
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
xas_unlock(&xas);
} while (xas_nomem(&xas, GFP_KERNEL));
}
xas_set(&xas, ULONG_MAX);
do {
xas_lock(&xas);
xas_store(&xas, xa_mk_value(0));
xas_unlock(&xas);
} while (xas_nomem(&xas, GFP_KERNEL));
xas_lock(&xas);
xas_store(&xas, NULL);
xas_set(&xas, 0);
j = i;
xas_for_each(&xas, entry, ULONG_MAX) {
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry != xa_mk_index(j));
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
xas_store(&xas, NULL);
j++;
}
xas_unlock(&xas);
XA_BUG_ON(xa, !xa_empty(xa));
}
}
xarray: Move multiorder_check to in-kernel tests This version is a little less thorough in order to be a little quicker, but tests the important edge cases. Also test adding a multiorder entry at a non-canonical index, and erasing it. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-09 05:52:17 +00:00
#ifdef CONFIG_XARRAY_MULTI
static noinline void check_multi_store_1(struct xarray *xa, unsigned long index,
unsigned int order)
{
XA_STATE(xas, xa, index);
unsigned long min = index & ~((1UL << order) - 1);
unsigned long max = min + (1UL << order);
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_store_order(xa, index, order, xa_mk_index(index), GFP_KERNEL);
XA_BUG_ON(xa, xa_load(xa, min) != xa_mk_index(index));
XA_BUG_ON(xa, xa_load(xa, max - 1) != xa_mk_index(index));
xarray: Move multiorder_check to in-kernel tests This version is a little less thorough in order to be a little quicker, but tests the important edge cases. Also test adding a multiorder entry at a non-canonical index, and erasing it. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-09 05:52:17 +00:00
XA_BUG_ON(xa, xa_load(xa, max) != NULL);
XA_BUG_ON(xa, xa_load(xa, min - 1) != NULL);
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
xas_lock(&xas);
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, xas_store(&xas, xa_mk_index(min)) != xa_mk_index(index));
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
xas_unlock(&xas);
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, xa_load(xa, min) != xa_mk_index(min));
XA_BUG_ON(xa, xa_load(xa, max - 1) != xa_mk_index(min));
xarray: Move multiorder_check to in-kernel tests This version is a little less thorough in order to be a little quicker, but tests the important edge cases. Also test adding a multiorder entry at a non-canonical index, and erasing it. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-09 05:52:17 +00:00
XA_BUG_ON(xa, xa_load(xa, max) != NULL);
XA_BUG_ON(xa, xa_load(xa, min - 1) != NULL);
xa_erase_index(xa, min);
XA_BUG_ON(xa, !xa_empty(xa));
}
static noinline void check_multi_store_2(struct xarray *xa, unsigned long index,
unsigned int order)
{
XA_STATE(xas, xa, index);
xa_store_order(xa, index, order, xa_mk_value(0), GFP_KERNEL);
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
xas_lock(&xas);
xarray: Move multiorder_check to in-kernel tests This version is a little less thorough in order to be a little quicker, but tests the important edge cases. Also test adding a multiorder entry at a non-canonical index, and erasing it. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-09 05:52:17 +00:00
XA_BUG_ON(xa, xas_store(&xas, xa_mk_value(1)) != xa_mk_value(0));
XA_BUG_ON(xa, xas.xa_index != index);
XA_BUG_ON(xa, xas_store(&xas, NULL) != xa_mk_value(1));
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
xas_unlock(&xas);
xarray: Move multiorder_check to in-kernel tests This version is a little less thorough in order to be a little quicker, but tests the important edge cases. Also test adding a multiorder entry at a non-canonical index, and erasing it. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-09 05:52:17 +00:00
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray tests: Check iterating over multiorder entries There was no bug here, but there was no test coverage for this scenario. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-29 21:04:35 +00:00
static noinline void check_multi_store_3(struct xarray *xa, unsigned long index,
unsigned int order)
{
XA_STATE(xas, xa, 0);
void *entry;
int n = 0;
xa_store_order(xa, index, order, xa_mk_index(index), GFP_KERNEL);
xas_lock(&xas);
xas_for_each(&xas, entry, ULONG_MAX) {
XA_BUG_ON(xa, entry != xa_mk_index(index));
n++;
}
XA_BUG_ON(xa, n != 1);
xas_set(&xas, index + 1);
xas_for_each(&xas, entry, ULONG_MAX) {
XA_BUG_ON(xa, entry != xa_mk_index(index));
n++;
}
XA_BUG_ON(xa, n != 2);
xas_unlock(&xas);
xa_destroy(xa);
}
xarray: Move multiorder_check to in-kernel tests This version is a little less thorough in order to be a little quicker, but tests the important edge cases. Also test adding a multiorder entry at a non-canonical index, and erasing it. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-09 05:52:17 +00:00
#endif
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
static noinline void check_multi_store(struct xarray *xa)
{
#ifdef CONFIG_XARRAY_MULTI
unsigned long i, j, k;
unsigned int max_order = (sizeof(long) == 4) ? 30 : 60;
/* Loading from any position returns the same value */
xa_store_order(xa, 0, 1, xa_mk_value(0), GFP_KERNEL);
XA_BUG_ON(xa, xa_load(xa, 0) != xa_mk_value(0));
XA_BUG_ON(xa, xa_load(xa, 1) != xa_mk_value(0));
XA_BUG_ON(xa, xa_load(xa, 2) != NULL);
rcu_read_lock();
XA_BUG_ON(xa, xa_to_node(xa_head(xa))->count != 2);
XA_BUG_ON(xa, xa_to_node(xa_head(xa))->nr_values != 2);
rcu_read_unlock();
/* Storing adjacent to the value does not alter the value */
xa_store(xa, 3, xa, GFP_KERNEL);
XA_BUG_ON(xa, xa_load(xa, 0) != xa_mk_value(0));
XA_BUG_ON(xa, xa_load(xa, 1) != xa_mk_value(0));
XA_BUG_ON(xa, xa_load(xa, 2) != NULL);
rcu_read_lock();
XA_BUG_ON(xa, xa_to_node(xa_head(xa))->count != 3);
XA_BUG_ON(xa, xa_to_node(xa_head(xa))->nr_values != 2);
rcu_read_unlock();
/* Overwriting multiple indexes works */
xa_store_order(xa, 0, 2, xa_mk_value(1), GFP_KERNEL);
XA_BUG_ON(xa, xa_load(xa, 0) != xa_mk_value(1));
XA_BUG_ON(xa, xa_load(xa, 1) != xa_mk_value(1));
XA_BUG_ON(xa, xa_load(xa, 2) != xa_mk_value(1));
XA_BUG_ON(xa, xa_load(xa, 3) != xa_mk_value(1));
XA_BUG_ON(xa, xa_load(xa, 4) != NULL);
rcu_read_lock();
XA_BUG_ON(xa, xa_to_node(xa_head(xa))->count != 4);
XA_BUG_ON(xa, xa_to_node(xa_head(xa))->nr_values != 4);
rcu_read_unlock();
/* We can erase multiple values with a single store */
XArray tests: Correct some 64-bit assumptions The test-suite caught these two mistakes when compiled for 32-bit. I had only been running the test-suite in 64-bit mode. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 14:34:04 +00:00
xa_store_order(xa, 0, BITS_PER_LONG - 1, NULL, GFP_KERNEL);
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
XA_BUG_ON(xa, !xa_empty(xa));
/* Even when the first slot is empty but the others aren't */
xa_store_index(xa, 1, GFP_KERNEL);
xa_store_index(xa, 2, GFP_KERNEL);
xa_store_order(xa, 0, 2, NULL, GFP_KERNEL);
XA_BUG_ON(xa, !xa_empty(xa));
for (i = 0; i < max_order; i++) {
for (j = 0; j < max_order; j++) {
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_store_order(xa, 0, i, xa_mk_index(i), GFP_KERNEL);
xa_store_order(xa, 0, j, xa_mk_index(j), GFP_KERNEL);
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
for (k = 0; k < max_order; k++) {
void *entry = xa_load(xa, (1UL << k) - 1);
if ((i < k) && (j < k))
XA_BUG_ON(xa, entry != NULL);
else
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry != xa_mk_index(j));
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
}
xa_erase(xa, 0);
XA_BUG_ON(xa, !xa_empty(xa));
}
}
xarray: Move multiorder_check to in-kernel tests This version is a little less thorough in order to be a little quicker, but tests the important edge cases. Also test adding a multiorder entry at a non-canonical index, and erasing it. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-09-09 05:52:17 +00:00
for (i = 0; i < 20; i++) {
check_multi_store_1(xa, 200, i);
check_multi_store_1(xa, 0, i);
check_multi_store_1(xa, (1UL << i) + 1, i);
}
check_multi_store_2(xa, 4095, 9);
XArray tests: Check iterating over multiorder entries There was no bug here, but there was no test coverage for this scenario. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-29 21:04:35 +00:00
for (i = 1; i < 20; i++) {
check_multi_store_3(xa, 0, i);
check_multi_store_3(xa, 1UL << i, i);
}
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
#endif
}
test_xarray: add tests for advanced multi-index use Patch series "test_xarray: advanced API multi-index tests", v2. This is a respin of the test_xarray multi-index tests [0] which use and demonstrate the advanced API which is used by the page cache. This should let folks more easily follow how we use multi-index to support for example a min order later in the page cache. It also lets us grow the selftests to mimic more of what we do in the page cache. This patch (of 2): The multi index selftests are great but they don't replicate how we deal with the page cache exactly, which makes it a bit hard to follow as the page cache uses the advanced API. Add tests which use the advanced API, mimicking what we do in the page cache, while at it, extend the example to do what is needed for min order support. [mcgrof@kernel.org: fix soft lockup for advanced-api tests] Link: https://lkml.kernel.org/r/20240216194329.840555-1-mcgrof@kernel.org [akpm@linux-foundation.org: s/i/loops/, make non-static] [akpm@linux-foundation.org: restore static storage for loop counter] Link: https://lkml.kernel.org/r/20240131225125.1370598-1-mcgrof@kernel.org Link: https://lkml.kernel.org/r/20240131225125.1370598-2-mcgrof@kernel.org Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Tested-by: Daniel Gomez <da.gomez@samsung.com> Cc: Darrick J. Wong <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Hannes Reinecke <hare@suse.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-01-31 22:51:24 +00:00
#ifdef CONFIG_XARRAY_MULTI
/* mimics page cache __filemap_add_folio() */
static noinline void check_xa_multi_store_adv_add(struct xarray *xa,
unsigned long index,
unsigned int order,
void *p)
{
XA_STATE(xas, xa, index);
unsigned int nrpages = 1UL << order;
/* users are responsible for index alignemnt to the order when adding */
XA_BUG_ON(xa, index & (nrpages - 1));
xas_set_order(&xas, index, order);
do {
xas_lock_irq(&xas);
xas_store(&xas, p);
xas_unlock_irq(&xas);
lib/test_xarray.c: fix error assumptions on check_xa_multi_store_adv_add() While testing lib/test_xarray in userspace I've noticed we can fail with: make -C tools/testing/radix-tree ./tools/testing/radix-tree/xarray BUG at check_xa_multi_store_adv_add:749 xarray: 0x55905fb21a00x head 0x55905fa1d8e0x flags 0 marks 0 0 0 0: 0x55905fa1d8e0x xarray: ../../../lib/test_xarray.c:749: check_xa_multi_store_adv_add: Assertion `0' failed. Aborted We get a failure with a BUG_ON(), and that is because we actually can fail due to -ENOMEM, the check in xas_nomem() will fix this for us so it makes no sense to expect no failure inside the loop. So modify the check and since this is also useful for instructional purposes clarify the situation. The check for XA_BUG_ON(xa, xa_load(xa, index) != p) is already done at the end of the loop so just remove the bogus on inside the loop. With this we now pass the test in both kernel and userspace: In userspace: ./tools/testing/radix-tree/xarray XArray: 149092856 of 149092856 tests passed In kernel space: XArray: 148257077 of 148257077 tests passed Link: https://lkml.kernel.org/r/20240423192221.301095-3-mcgrof@kernel.org Fixes: a60cc288a1a2 ("test_xarray: add tests for advanced multi-index use") Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Cc: Daniel Gomez <da.gomez@samsung.com> Cc: Darrick J. Wong <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-04-23 19:22:21 +00:00
/*
* In our selftest case the only failure we can expect is for
* there not to be enough memory as we're not mimicking the
* entire page cache, so verify that's the only error we can run
* into here. The xas_nomem() which follows will ensure to fix
* that condition for us so to chug on on the loop.
*/
XA_BUG_ON(xa, xas_error(&xas) && xas_error(&xas) != -ENOMEM);
test_xarray: add tests for advanced multi-index use Patch series "test_xarray: advanced API multi-index tests", v2. This is a respin of the test_xarray multi-index tests [0] which use and demonstrate the advanced API which is used by the page cache. This should let folks more easily follow how we use multi-index to support for example a min order later in the page cache. It also lets us grow the selftests to mimic more of what we do in the page cache. This patch (of 2): The multi index selftests are great but they don't replicate how we deal with the page cache exactly, which makes it a bit hard to follow as the page cache uses the advanced API. Add tests which use the advanced API, mimicking what we do in the page cache, while at it, extend the example to do what is needed for min order support. [mcgrof@kernel.org: fix soft lockup for advanced-api tests] Link: https://lkml.kernel.org/r/20240216194329.840555-1-mcgrof@kernel.org [akpm@linux-foundation.org: s/i/loops/, make non-static] [akpm@linux-foundation.org: restore static storage for loop counter] Link: https://lkml.kernel.org/r/20240131225125.1370598-1-mcgrof@kernel.org Link: https://lkml.kernel.org/r/20240131225125.1370598-2-mcgrof@kernel.org Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Tested-by: Daniel Gomez <da.gomez@samsung.com> Cc: Darrick J. Wong <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Hannes Reinecke <hare@suse.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-01-31 22:51:24 +00:00
} while (xas_nomem(&xas, GFP_KERNEL));
XA_BUG_ON(xa, xas_error(&xas));
lib/test_xarray.c: fix error assumptions on check_xa_multi_store_adv_add() While testing lib/test_xarray in userspace I've noticed we can fail with: make -C tools/testing/radix-tree ./tools/testing/radix-tree/xarray BUG at check_xa_multi_store_adv_add:749 xarray: 0x55905fb21a00x head 0x55905fa1d8e0x flags 0 marks 0 0 0 0: 0x55905fa1d8e0x xarray: ../../../lib/test_xarray.c:749: check_xa_multi_store_adv_add: Assertion `0' failed. Aborted We get a failure with a BUG_ON(), and that is because we actually can fail due to -ENOMEM, the check in xas_nomem() will fix this for us so it makes no sense to expect no failure inside the loop. So modify the check and since this is also useful for instructional purposes clarify the situation. The check for XA_BUG_ON(xa, xa_load(xa, index) != p) is already done at the end of the loop so just remove the bogus on inside the loop. With this we now pass the test in both kernel and userspace: In userspace: ./tools/testing/radix-tree/xarray XArray: 149092856 of 149092856 tests passed In kernel space: XArray: 148257077 of 148257077 tests passed Link: https://lkml.kernel.org/r/20240423192221.301095-3-mcgrof@kernel.org Fixes: a60cc288a1a2 ("test_xarray: add tests for advanced multi-index use") Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Cc: Daniel Gomez <da.gomez@samsung.com> Cc: Darrick J. Wong <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-04-23 19:22:21 +00:00
XA_BUG_ON(xa, xa_load(xa, index) != p);
test_xarray: add tests for advanced multi-index use Patch series "test_xarray: advanced API multi-index tests", v2. This is a respin of the test_xarray multi-index tests [0] which use and demonstrate the advanced API which is used by the page cache. This should let folks more easily follow how we use multi-index to support for example a min order later in the page cache. It also lets us grow the selftests to mimic more of what we do in the page cache. This patch (of 2): The multi index selftests are great but they don't replicate how we deal with the page cache exactly, which makes it a bit hard to follow as the page cache uses the advanced API. Add tests which use the advanced API, mimicking what we do in the page cache, while at it, extend the example to do what is needed for min order support. [mcgrof@kernel.org: fix soft lockup for advanced-api tests] Link: https://lkml.kernel.org/r/20240216194329.840555-1-mcgrof@kernel.org [akpm@linux-foundation.org: s/i/loops/, make non-static] [akpm@linux-foundation.org: restore static storage for loop counter] Link: https://lkml.kernel.org/r/20240131225125.1370598-1-mcgrof@kernel.org Link: https://lkml.kernel.org/r/20240131225125.1370598-2-mcgrof@kernel.org Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Tested-by: Daniel Gomez <da.gomez@samsung.com> Cc: Darrick J. Wong <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Hannes Reinecke <hare@suse.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-01-31 22:51:24 +00:00
}
/* mimics page_cache_delete() */
static noinline void check_xa_multi_store_adv_del_entry(struct xarray *xa,
unsigned long index,
unsigned int order)
{
XA_STATE(xas, xa, index);
xas_set_order(&xas, index, order);
xas_store(&xas, NULL);
xas_init_marks(&xas);
}
static noinline void check_xa_multi_store_adv_delete(struct xarray *xa,
unsigned long index,
unsigned int order)
{
xa_lock_irq(xa);
check_xa_multi_store_adv_del_entry(xa, index, order);
xa_unlock_irq(xa);
}
/* mimics page cache filemap_get_entry() */
static noinline void *test_get_entry(struct xarray *xa, unsigned long index)
{
XA_STATE(xas, xa, index);
void *p;
static unsigned int loops = 0;
rcu_read_lock();
repeat:
xas_reset(&xas);
p = xas_load(&xas);
if (xas_retry(&xas, p))
goto repeat;
rcu_read_unlock();
/*
* This is not part of the page cache, this selftest is pretty
* aggressive and does not want to trust the xarray API but rather
* test it, and for order 20 (4 GiB block size) we can loop over
* over a million entries which can cause a soft lockup. Page cache
* APIs won't be stupid, proper page cache APIs loop over the proper
* order so when using a larger order we skip shared entries.
*/
if (++loops % XA_CHECK_SCHED == 0)
schedule();
return p;
}
static unsigned long some_val = 0xdeadbeef;
static unsigned long some_val_2 = 0xdeaddead;
/* mimics the page cache usage */
static noinline void check_xa_multi_store_adv(struct xarray *xa,
unsigned long pos,
unsigned int order)
{
unsigned int nrpages = 1UL << order;
unsigned long index, base, next_index, next_next_index;
unsigned int i;
index = pos >> PAGE_SHIFT;
base = round_down(index, nrpages);
next_index = round_down(base + nrpages, nrpages);
next_next_index = round_down(next_index + nrpages, nrpages);
check_xa_multi_store_adv_add(xa, base, order, &some_val);
for (i = 0; i < nrpages; i++)
XA_BUG_ON(xa, test_get_entry(xa, base + i) != &some_val);
XA_BUG_ON(xa, test_get_entry(xa, next_index) != NULL);
/* Use order 0 for the next item */
check_xa_multi_store_adv_add(xa, next_index, 0, &some_val_2);
XA_BUG_ON(xa, test_get_entry(xa, next_index) != &some_val_2);
/* Remove the next item */
check_xa_multi_store_adv_delete(xa, next_index, 0);
/* Now use order for a new pointer */
check_xa_multi_store_adv_add(xa, next_index, order, &some_val_2);
for (i = 0; i < nrpages; i++)
XA_BUG_ON(xa, test_get_entry(xa, next_index + i) != &some_val_2);
check_xa_multi_store_adv_delete(xa, next_index, order);
check_xa_multi_store_adv_delete(xa, base, order);
XA_BUG_ON(xa, !xa_empty(xa));
/* starting fresh again */
/* let's test some holes now */
/* hole at base and next_next */
check_xa_multi_store_adv_add(xa, next_index, order, &some_val_2);
for (i = 0; i < nrpages; i++)
XA_BUG_ON(xa, test_get_entry(xa, base + i) != NULL);
for (i = 0; i < nrpages; i++)
XA_BUG_ON(xa, test_get_entry(xa, next_index + i) != &some_val_2);
for (i = 0; i < nrpages; i++)
XA_BUG_ON(xa, test_get_entry(xa, next_next_index + i) != NULL);
check_xa_multi_store_adv_delete(xa, next_index, order);
XA_BUG_ON(xa, !xa_empty(xa));
/* hole at base and next */
check_xa_multi_store_adv_add(xa, next_next_index, order, &some_val_2);
for (i = 0; i < nrpages; i++)
XA_BUG_ON(xa, test_get_entry(xa, base + i) != NULL);
for (i = 0; i < nrpages; i++)
XA_BUG_ON(xa, test_get_entry(xa, next_index + i) != NULL);
for (i = 0; i < nrpages; i++)
XA_BUG_ON(xa, test_get_entry(xa, next_next_index + i) != &some_val_2);
check_xa_multi_store_adv_delete(xa, next_next_index, order);
XA_BUG_ON(xa, !xa_empty(xa));
}
#endif
static noinline void check_multi_store_advanced(struct xarray *xa)
{
#ifdef CONFIG_XARRAY_MULTI
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1;
unsigned long end = ULONG_MAX/2;
unsigned long pos, i;
/*
* About 117 million tests below.
*/
for (pos = 7; pos < end; pos = (pos * pos) + 564) {
for (i = 0; i < max_order; i++) {
check_xa_multi_store_adv(xa, pos, i);
check_xa_multi_store_adv(xa, pos + 157, i);
}
}
#endif
}
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
static noinline void check_xa_alloc_1(struct xarray *xa, unsigned int base)
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
{
int i;
u32 id;
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
XA_BUG_ON(xa, !xa_empty(xa));
/* An empty array should assign %base to the first alloc */
xa_alloc_index(xa, base, GFP_KERNEL);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
/* Erasing it should make the array empty again */
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
xa_erase_index(xa, base);
XA_BUG_ON(xa, !xa_empty(xa));
/* And it should assign %base again */
xa_alloc_index(xa, base, GFP_KERNEL);
/* Allocating and then erasing a lot should not lose base */
for (i = base + 1; i < 2 * XA_CHUNK_SIZE; i++)
xa_alloc_index(xa, i, GFP_KERNEL);
for (i = base; i < 2 * XA_CHUNK_SIZE; i++)
xa_erase_index(xa, i);
xa_alloc_index(xa, base, GFP_KERNEL);
/* Destroying the array should do the same as erasing */
xa_destroy(xa);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
/* And it should assign %base again */
xa_alloc_index(xa, base, GFP_KERNEL);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
/* The next assigned ID should be base+1 */
xa_alloc_index(xa, base + 1, GFP_KERNEL);
xa_erase_index(xa, base + 1);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
/* Storing a value should mark it used */
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
xa_store_index(xa, base + 1, GFP_KERNEL);
xa_alloc_index(xa, base + 2, GFP_KERNEL);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
/* If we then erase base, it should be free */
xa_erase_index(xa, base);
xa_alloc_index(xa, base, GFP_KERNEL);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
xa_erase_index(xa, base + 1);
xa_erase_index(xa, base + 2);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
for (i = 1; i < 5000; i++) {
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
xa_alloc_index(xa, base + i, GFP_KERNEL);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
}
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
xa_destroy(xa);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
/* Check that we fail properly at the limit of allocation */
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
XA_BUG_ON(xa, xa_alloc(xa, &id, xa_mk_index(UINT_MAX - 1),
XA_LIMIT(UINT_MAX - 1, UINT_MAX),
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
GFP_KERNEL) != 0);
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
XA_BUG_ON(xa, id != 0xfffffffeU);
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
XA_BUG_ON(xa, xa_alloc(xa, &id, xa_mk_index(UINT_MAX),
XA_LIMIT(UINT_MAX - 1, UINT_MAX),
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
GFP_KERNEL) != 0);
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
XA_BUG_ON(xa, id != 0xffffffffU);
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
id = 3;
XA_BUG_ON(xa, xa_alloc(xa, &id, xa_mk_index(0),
XA_LIMIT(UINT_MAX - 1, UINT_MAX),
GFP_KERNEL) != -EBUSY);
XA_BUG_ON(xa, id != 3);
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
xa_destroy(xa);
XArray: Fix xa_alloc when id exceeds max Specifying a starting ID greater than the maximum ID isn't something attempted very often, but it should fail. It was succeeding due to xas_find_marked() returning the wrong error state, so add tests for both xa_alloc() and xas_find_marked(). Fixes: b803b42823d0 ("xarray: Add XArray iterators") Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-13 18:57:42 +00:00
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
XA_BUG_ON(xa, xa_alloc(xa, &id, xa_mk_index(10), XA_LIMIT(10, 5),
GFP_KERNEL) != -EBUSY);
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
XA_BUG_ON(xa, xa_store_index(xa, 3, GFP_KERNEL) != 0);
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
XA_BUG_ON(xa, xa_alloc(xa, &id, xa_mk_index(10), XA_LIMIT(10, 5),
GFP_KERNEL) != -EBUSY);
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
xa_erase_index(xa, 3);
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
static noinline void check_xa_alloc_2(struct xarray *xa, unsigned int base)
{
unsigned int i, id;
unsigned long index;
void *entry;
/* Allocate and free a NULL and check xa_empty() behaves */
XA_BUG_ON(xa, !xa_empty(xa));
XA_BUG_ON(xa, xa_alloc(xa, &id, NULL, xa_limit_32b, GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != base);
XA_BUG_ON(xa, xa_empty(xa));
XA_BUG_ON(xa, xa_erase(xa, id) != NULL);
XA_BUG_ON(xa, !xa_empty(xa));
/* Ditto, but check destroy instead of erase */
XA_BUG_ON(xa, !xa_empty(xa));
XA_BUG_ON(xa, xa_alloc(xa, &id, NULL, xa_limit_32b, GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != base);
XA_BUG_ON(xa, xa_empty(xa));
xa_destroy(xa);
XA_BUG_ON(xa, !xa_empty(xa));
for (i = base; i < base + 10; i++) {
XA_BUG_ON(xa, xa_alloc(xa, &id, NULL, xa_limit_32b,
GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != i);
}
XA_BUG_ON(xa, xa_store(xa, 3, xa_mk_index(3), GFP_KERNEL) != NULL);
XA_BUG_ON(xa, xa_store(xa, 4, xa_mk_index(4), GFP_KERNEL) != NULL);
XA_BUG_ON(xa, xa_store(xa, 4, NULL, GFP_KERNEL) != xa_mk_index(4));
XA_BUG_ON(xa, xa_erase(xa, 5) != NULL);
XA_BUG_ON(xa, xa_alloc(xa, &id, NULL, xa_limit_32b, GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != 5);
xa_for_each(xa, index, entry) {
xa_erase_index(xa, index);
}
for (i = base; i < base + 9; i++) {
XA_BUG_ON(xa, xa_erase(xa, i) != NULL);
XA_BUG_ON(xa, xa_empty(xa));
}
XA_BUG_ON(xa, xa_erase(xa, 8) != NULL);
XA_BUG_ON(xa, xa_empty(xa));
XA_BUG_ON(xa, xa_erase(xa, base + 9) != NULL);
XA_BUG_ON(xa, !xa_empty(xa));
xa_destroy(xa);
}
XArray: Add cyclic allocation This differs slightly from the IDR equivalent in five ways. 1. It can allocate up to UINT_MAX instead of being limited to INT_MAX, like xa_alloc(). Also like xa_alloc(), it will write to the 'id' pointer before placing the entry in the XArray. 2. The 'next' cursor is allocated separately from the XArray instead of being part of the IDR. This saves memory for all the users which do not use the cyclic allocation API and suits some users better. 3. It returns -EBUSY instead of -ENOSPC. 4. It will attempt to wrap back to the minimum value on memory allocation failure as well as on an -EBUSY error, assuming that a user would rather allocate a small ID than suffer an ID allocation failure. 5. It reports whether it has wrapped, which is important to some users. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-06 19:13:35 +00:00
static noinline void check_xa_alloc_3(struct xarray *xa, unsigned int base)
{
struct xa_limit limit = XA_LIMIT(1, 0x3fff);
u32 next = 0;
unsigned int i, id;
unsigned long index;
void *entry;
XA_BUG_ON(xa, xa_alloc_cyclic(xa, &id, xa_mk_index(1), limit,
&next, GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != 1);
next = 0x3ffd;
XA_BUG_ON(xa, xa_alloc_cyclic(xa, &id, xa_mk_index(0x3ffd), limit,
&next, GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != 0x3ffd);
xa_erase_index(xa, 0x3ffd);
xa_erase_index(xa, 1);
XA_BUG_ON(xa, !xa_empty(xa));
for (i = 0x3ffe; i < 0x4003; i++) {
if (i < 0x4000)
entry = xa_mk_index(i);
else
entry = xa_mk_index(i - 0x3fff);
XA_BUG_ON(xa, xa_alloc_cyclic(xa, &id, entry, limit,
&next, GFP_KERNEL) != (id == 1));
XA_BUG_ON(xa, xa_mk_index(id) != entry);
}
/* Check wrap-around is handled correctly */
if (base != 0)
xa_erase_index(xa, base);
xa_erase_index(xa, base + 1);
next = UINT_MAX;
XA_BUG_ON(xa, xa_alloc_cyclic(xa, &id, xa_mk_index(UINT_MAX),
xa_limit_32b, &next, GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != UINT_MAX);
XA_BUG_ON(xa, xa_alloc_cyclic(xa, &id, xa_mk_index(base),
xa_limit_32b, &next, GFP_KERNEL) != 1);
XA_BUG_ON(xa, id != base);
XA_BUG_ON(xa, xa_alloc_cyclic(xa, &id, xa_mk_index(base + 1),
xa_limit_32b, &next, GFP_KERNEL) != 0);
XA_BUG_ON(xa, id != base + 1);
xa_for_each(xa, index, entry)
xa_erase_index(xa, index);
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray: Add support for 1s-based allocation A lot of places want to allocate IDs starting at 1 instead of 0. While the xa_alloc() API supports this, it's not very efficient if lots of IDs are allocated, due to having to walk down to the bottom of the tree to see if ID 1 is available, then all the way over to the next non-allocated ID. This method marks ID 0 as being occupied which wastes one slot in the XArray, but preserves xa_empty() as working. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-26 18:43:22 +00:00
static DEFINE_XARRAY_ALLOC(xa0);
static DEFINE_XARRAY_ALLOC1(xa1);
static noinline void check_xa_alloc(void)
{
check_xa_alloc_1(&xa0, 0);
check_xa_alloc_1(&xa1, 1);
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
check_xa_alloc_2(&xa0, 0);
check_xa_alloc_2(&xa1, 1);
XArray: Add cyclic allocation This differs slightly from the IDR equivalent in five ways. 1. It can allocate up to UINT_MAX instead of being limited to INT_MAX, like xa_alloc(). Also like xa_alloc(), it will write to the 'id' pointer before placing the entry in the XArray. 2. The 'next' cursor is allocated separately from the XArray instead of being part of the IDR. This saves memory for all the users which do not use the cyclic allocation API and suits some users better. 3. It returns -EBUSY instead of -ENOSPC. 4. It will attempt to wrap back to the minimum value on memory allocation failure as well as on an -EBUSY error, assuming that a user would rather allocate a small ID than suffer an ID allocation failure. 5. It reports whether it has wrapped, which is important to some users. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-06 19:13:35 +00:00
check_xa_alloc_3(&xa0, 0);
check_xa_alloc_3(&xa1, 1);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
}
xarray: Add xas_for_each_conflict This iterator iterates over each entry that is stored in the index or indices specified by the xa_state. This is intended for use for a conditional store of a multiindex entry, or to allow entries which are about to be removed from the xarray to be disposed of properly. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-06-02 02:46:02 +00:00
static noinline void __check_store_iter(struct xarray *xa, unsigned long start,
unsigned int order, unsigned int present)
{
XA_STATE_ORDER(xas, xa, start, order);
void *entry;
unsigned int count = 0;
retry:
xas_lock(&xas);
xas_for_each_conflict(&xas, entry) {
XA_BUG_ON(xa, !xa_is_value(entry));
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry < xa_mk_index(start));
XA_BUG_ON(xa, entry > xa_mk_index(start + (1UL << order) - 1));
xarray: Add xas_for_each_conflict This iterator iterates over each entry that is stored in the index or indices specified by the xa_state. This is intended for use for a conditional store of a multiindex entry, or to allow entries which are about to be removed from the xarray to be disposed of properly. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-06-02 02:46:02 +00:00
count++;
}
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xas_store(&xas, xa_mk_index(start));
xarray: Add xas_for_each_conflict This iterator iterates over each entry that is stored in the index or indices specified by the xa_state. This is intended for use for a conditional store of a multiindex entry, or to allow entries which are about to be removed from the xarray to be disposed of properly. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-06-02 02:46:02 +00:00
xas_unlock(&xas);
if (xas_nomem(&xas, GFP_KERNEL)) {
count = 0;
goto retry;
}
XA_BUG_ON(xa, xas_error(&xas));
XA_BUG_ON(xa, count != present);
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, xa_load(xa, start) != xa_mk_index(start));
xarray: Add xas_for_each_conflict This iterator iterates over each entry that is stored in the index or indices specified by the xa_state. This is intended for use for a conditional store of a multiindex entry, or to allow entries which are about to be removed from the xarray to be disposed of properly. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-06-02 02:46:02 +00:00
XA_BUG_ON(xa, xa_load(xa, start + (1UL << order) - 1) !=
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_mk_index(start));
xarray: Add xas_for_each_conflict This iterator iterates over each entry that is stored in the index or indices specified by the xa_state. This is intended for use for a conditional store of a multiindex entry, or to allow entries which are about to be removed from the xarray to be disposed of properly. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-06-02 02:46:02 +00:00
xa_erase_index(xa, start);
}
static noinline void check_store_iter(struct xarray *xa)
{
unsigned int i, j;
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1;
for (i = 0; i < max_order; i++) {
unsigned int min = 1 << i;
unsigned int max = (2 << i) - 1;
__check_store_iter(xa, 0, i, 0);
XA_BUG_ON(xa, !xa_empty(xa));
__check_store_iter(xa, min, i, 0);
XA_BUG_ON(xa, !xa_empty(xa));
xa_store_index(xa, min, GFP_KERNEL);
__check_store_iter(xa, min, i, 1);
XA_BUG_ON(xa, !xa_empty(xa));
xa_store_index(xa, max, GFP_KERNEL);
__check_store_iter(xa, min, i, 1);
XA_BUG_ON(xa, !xa_empty(xa));
for (j = 0; j < min; j++)
xa_store_index(xa, j, GFP_KERNEL);
__check_store_iter(xa, 0, i, min);
XA_BUG_ON(xa, !xa_empty(xa));
for (j = 0; j < min; j++)
xa_store_index(xa, min + j, GFP_KERNEL);
__check_store_iter(xa, min, i, min);
XA_BUG_ON(xa, !xa_empty(xa));
}
#ifdef CONFIG_XARRAY_MULTI
xa_store_index(xa, 63, GFP_KERNEL);
xa_store_index(xa, 65, GFP_KERNEL);
__check_store_iter(xa, 64, 2, 1);
xa_erase_index(xa, 63);
#endif
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
static noinline void check_multi_find_1(struct xarray *xa, unsigned order)
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
{
#ifdef CONFIG_XARRAY_MULTI
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
unsigned long multi = 3 << order;
unsigned long next = 4 << order;
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
unsigned long index;
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
xa_store_order(xa, multi, order, xa_mk_value(multi), GFP_KERNEL);
XA_BUG_ON(xa, xa_store_index(xa, next, GFP_KERNEL) != NULL);
XArray: Fix xas_find returning too many entries If you call xas_find() with the initial index > max, it should have returned NULL but was returning the entry at index. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:13:21 +00:00
XA_BUG_ON(xa, xa_store_index(xa, next + 1, GFP_KERNEL) != NULL);
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
index = 0;
XA_BUG_ON(xa, xa_find(xa, &index, ULONG_MAX, XA_PRESENT) !=
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
xa_mk_value(multi));
XA_BUG_ON(xa, index != multi);
index = multi + 1;
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
XA_BUG_ON(xa, xa_find(xa, &index, ULONG_MAX, XA_PRESENT) !=
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
xa_mk_value(multi));
XA_BUG_ON(xa, (index < multi) || (index >= next));
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
XA_BUG_ON(xa, xa_find_after(xa, &index, ULONG_MAX, XA_PRESENT) !=
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
xa_mk_value(next));
XA_BUG_ON(xa, index != next);
XArray: Fix xas_find returning too many entries If you call xas_find() with the initial index > max, it should have returned NULL but was returning the entry at index. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:13:21 +00:00
XA_BUG_ON(xa, xa_find_after(xa, &index, next, XA_PRESENT) != NULL);
XA_BUG_ON(xa, index != next);
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
xa_erase_index(xa, multi);
xa_erase_index(xa, next);
XArray: Fix xas_find returning too many entries If you call xas_find() with the initial index > max, it should have returned NULL but was returning the entry at index. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:13:21 +00:00
xa_erase_index(xa, next + 1);
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
XA_BUG_ON(xa, !xa_empty(xa));
#endif
}
static noinline void check_multi_find_2(struct xarray *xa)
{
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 10 : 1;
unsigned int i, j;
void *entry;
for (i = 0; i < max_order; i++) {
unsigned long index = 1UL << i;
for (j = 0; j < index; j++) {
XA_STATE(xas, xa, j + index);
xa_store_index(xa, index - 1, GFP_KERNEL);
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_store_order(xa, index, i, xa_mk_index(index),
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
GFP_KERNEL);
rcu_read_lock();
xas_for_each(&xas, entry, ULONG_MAX) {
xa_erase_index(xa, index);
}
rcu_read_unlock();
xa_erase_index(xa, index - 1);
XA_BUG_ON(xa, !xa_empty(xa));
}
}
}
XArray: Fix xa_find_next for large multi-index entries Coverity pointed out that xas_sibling() was shifting xa_offset without promoting it to an unsigned long first, so the shift could cause an overflow and we'd get the wrong answer. The fix is obvious, and the new test-case provokes UBSAN to report an error: runtime error: shift exponent 60 is too large for 32-bit type 'int' Fixes: 19c30f4dd092 ("XArray: Fix xa_find_after with multi-index entries") Reported-by: Bjorn Helgaas <bhelgaas@google.com> Reported-by: Kees Cook <keescook@chromium.org> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-31 10:07:55 +00:00
static noinline void check_multi_find_3(struct xarray *xa)
{
unsigned int order;
for (order = 5; order < order_limit; order++) {
unsigned long index = 1UL << (order - 5);
XA_BUG_ON(xa, !xa_empty(xa));
xa_store_order(xa, 0, order - 4, xa_mk_index(0), GFP_KERNEL);
XA_BUG_ON(xa, xa_find_after(xa, &index, ULONG_MAX, XA_PRESENT));
xa_erase_index(xa, 0);
}
}
XArray: Fix xa_for_each with a single element at 0 The following sequence of calls would result in an infinite loop in xa_find_after(): xa_store(xa, 0, x, GFP_KERNEL); index = 0; xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { } xa_find_after() was confusing the situation where we found no entry in the tree with finding a multiorder entry, so it would look for the successor entry forever. Just check for this case explicitly. Includes a few new checks in the test suite to be sure this doesn't reappear. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-01 20:55:19 +00:00
static noinline void check_find_1(struct xarray *xa)
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
{
unsigned long i, j, k;
XA_BUG_ON(xa, !xa_empty(xa));
/*
* Check xa_find with all pairs between 0 and 99 inclusive,
* starting at every index between 0 and 99
*/
for (i = 0; i < 100; i++) {
XA_BUG_ON(xa, xa_store_index(xa, i, GFP_KERNEL) != NULL);
xa_set_mark(xa, i, XA_MARK_0);
for (j = 0; j < i; j++) {
XA_BUG_ON(xa, xa_store_index(xa, j, GFP_KERNEL) !=
NULL);
xa_set_mark(xa, j, XA_MARK_0);
for (k = 0; k < 100; k++) {
unsigned long index = k;
void *entry = xa_find(xa, &index, ULONG_MAX,
XA_PRESENT);
if (k <= j)
XA_BUG_ON(xa, index != j);
else if (k <= i)
XA_BUG_ON(xa, index != i);
else
XA_BUG_ON(xa, entry != NULL);
index = k;
entry = xa_find(xa, &index, ULONG_MAX,
XA_MARK_0);
if (k <= j)
XA_BUG_ON(xa, index != j);
else if (k <= i)
XA_BUG_ON(xa, index != i);
else
XA_BUG_ON(xa, entry != NULL);
}
xa_erase_index(xa, j);
XA_BUG_ON(xa, xa_get_mark(xa, j, XA_MARK_0));
XA_BUG_ON(xa, !xa_get_mark(xa, i, XA_MARK_0));
}
xa_erase_index(xa, i);
XA_BUG_ON(xa, xa_get_mark(xa, i, XA_MARK_0));
}
XA_BUG_ON(xa, !xa_empty(xa));
XArray: Fix xa_for_each with a single element at 0 The following sequence of calls would result in an infinite loop in xa_find_after(): xa_store(xa, 0, x, GFP_KERNEL); index = 0; xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { } xa_find_after() was confusing the situation where we found no entry in the tree with finding a multiorder entry, so it would look for the successor entry forever. Just check for this case explicitly. Includes a few new checks in the test suite to be sure this doesn't reappear. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-01 20:55:19 +00:00
}
static noinline void check_find_2(struct xarray *xa)
{
void *entry;
XArray: Change xa_for_each iterator There were three problems with this API: 1. It took too many arguments; almost all users wanted to iterate over every element in the array rather than a subset. 2. It required that 'index' be initialised before use, and there's no realistic way to make GCC catch that. 3. 'index' and 'entry' were the opposite way round from every other member of the XArray APIs. So split it into three different APIs: xa_for_each(xa, index, entry) xa_for_each_start(xa, index, entry, start) xa_for_each_marked(xa, index, entry, filter) Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-17 19:45:36 +00:00
unsigned long i, j, index;
XArray: Fix xa_for_each with a single element at 0 The following sequence of calls would result in an infinite loop in xa_find_after(): xa_store(xa, 0, x, GFP_KERNEL); index = 0; xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { } xa_find_after() was confusing the situation where we found no entry in the tree with finding a multiorder entry, so it would look for the successor entry forever. Just check for this case explicitly. Includes a few new checks in the test suite to be sure this doesn't reappear. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-01 20:55:19 +00:00
XArray: Change xa_for_each iterator There were three problems with this API: 1. It took too many arguments; almost all users wanted to iterate over every element in the array rather than a subset. 2. It required that 'index' be initialised before use, and there's no realistic way to make GCC catch that. 3. 'index' and 'entry' were the opposite way round from every other member of the XArray APIs. So split it into three different APIs: xa_for_each(xa, index, entry) xa_for_each_start(xa, index, entry, start) xa_for_each_marked(xa, index, entry, filter) Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-17 19:45:36 +00:00
xa_for_each(xa, index, entry) {
XArray: Fix xa_for_each with a single element at 0 The following sequence of calls would result in an infinite loop in xa_find_after(): xa_store(xa, 0, x, GFP_KERNEL); index = 0; xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { } xa_find_after() was confusing the situation where we found no entry in the tree with finding a multiorder entry, so it would look for the successor entry forever. Just check for this case explicitly. Includes a few new checks in the test suite to be sure this doesn't reappear. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-01 20:55:19 +00:00
XA_BUG_ON(xa, true);
}
for (i = 0; i < 1024; i++) {
xa_store_index(xa, index, GFP_KERNEL);
j = 0;
XArray: Change xa_for_each iterator There were three problems with this API: 1. It took too many arguments; almost all users wanted to iterate over every element in the array rather than a subset. 2. It required that 'index' be initialised before use, and there's no realistic way to make GCC catch that. 3. 'index' and 'entry' were the opposite way round from every other member of the XArray APIs. So split it into three different APIs: xa_for_each(xa, index, entry) xa_for_each_start(xa, index, entry, start) xa_for_each_marked(xa, index, entry, filter) Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-17 19:45:36 +00:00
xa_for_each(xa, index, entry) {
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, xa_mk_index(index) != entry);
XArray: Fix xa_for_each with a single element at 0 The following sequence of calls would result in an infinite loop in xa_find_after(): xa_store(xa, 0, x, GFP_KERNEL); index = 0; xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { } xa_find_after() was confusing the situation where we found no entry in the tree with finding a multiorder entry, so it would look for the successor entry forever. Just check for this case explicitly. Includes a few new checks in the test suite to be sure this doesn't reappear. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-01 20:55:19 +00:00
XA_BUG_ON(xa, index != j++);
}
}
xa_destroy(xa);
}
XArray: Fix xa_alloc when id exceeds max Specifying a starting ID greater than the maximum ID isn't something attempted very often, but it should fail. It was succeeding due to xas_find_marked() returning the wrong error state, so add tests for both xa_alloc() and xas_find_marked(). Fixes: b803b42823d0 ("xarray: Add XArray iterators") Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-13 18:57:42 +00:00
static noinline void check_find_3(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
unsigned long i, j, k;
void *entry;
for (i = 0; i < 100; i++) {
for (j = 0; j < 100; j++) {
XArray tests: Add RCU locking 0day picked up that I'd forgotten to add locking to this new test. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-17 22:37:25 +00:00
rcu_read_lock();
XArray: Fix xa_alloc when id exceeds max Specifying a starting ID greater than the maximum ID isn't something attempted very often, but it should fail. It was succeeding due to xas_find_marked() returning the wrong error state, so add tests for both xa_alloc() and xas_find_marked(). Fixes: b803b42823d0 ("xarray: Add XArray iterators") Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-13 18:57:42 +00:00
for (k = 0; k < 100; k++) {
xas_set(&xas, j);
xas_for_each_marked(&xas, entry, k, XA_MARK_0)
;
if (j > k)
XA_BUG_ON(xa,
xas.xa_node != XAS_RESTART);
}
XArray tests: Add RCU locking 0day picked up that I'd forgotten to add locking to this new test. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-17 22:37:25 +00:00
rcu_read_unlock();
XArray: Fix xa_alloc when id exceeds max Specifying a starting ID greater than the maximum ID isn't something attempted very often, but it should fail. It was succeeding due to xas_find_marked() returning the wrong error state, so add tests for both xa_alloc() and xas_find_marked(). Fixes: b803b42823d0 ("xarray: Add XArray iterators") Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-13 18:57:42 +00:00
}
xa_store_index(xa, i, GFP_KERNEL);
xa_set_mark(xa, i, XA_MARK_0);
}
xa_destroy(xa);
}
XArray: Fix infinite loop with entry at ULONG_MAX If there is an entry at ULONG_MAX, xa_for_each() will overflow the 'index + 1' in xa_find_after() and wrap around to 0. Catch this case and terminate the loop by returning NULL. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-17 22:45:12 +00:00
static noinline void check_find_4(struct xarray *xa)
{
unsigned long index = 0;
void *entry;
xa_store_index(xa, ULONG_MAX, GFP_KERNEL);
entry = xa_find_after(xa, &index, ULONG_MAX, XA_PRESENT);
XA_BUG_ON(xa, entry != xa_mk_index(ULONG_MAX));
entry = xa_find_after(xa, &index, ULONG_MAX, XA_PRESENT);
XA_BUG_ON(xa, entry);
xa_erase_index(xa, ULONG_MAX);
}
XArray: Fix xa_for_each with a single element at 0 The following sequence of calls would result in an infinite loop in xa_find_after(): xa_store(xa, 0, x, GFP_KERNEL); index = 0; xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { } xa_find_after() was confusing the situation where we found no entry in the tree with finding a multiorder entry, so it would look for the successor entry forever. Just check for this case explicitly. Includes a few new checks in the test suite to be sure this doesn't reappear. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-01 20:55:19 +00:00
static noinline void check_find(struct xarray *xa)
{
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
unsigned i;
XArray: Fix xa_for_each with a single element at 0 The following sequence of calls would result in an infinite loop in xa_find_after(): xa_store(xa, 0, x, GFP_KERNEL); index = 0; xa_for_each(xa, entry, index, ULONG_MAX, XA_PRESENT) { } xa_find_after() was confusing the situation where we found no entry in the tree with finding a multiorder entry, so it would look for the successor entry forever. Just check for this case explicitly. Includes a few new checks in the test suite to be sure this doesn't reappear. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-01 20:55:19 +00:00
check_find_1(xa);
check_find_2(xa);
XArray: Fix xa_alloc when id exceeds max Specifying a starting ID greater than the maximum ID isn't something attempted very often, but it should fail. It was succeeding due to xas_find_marked() returning the wrong error state, so add tests for both xa_alloc() and xas_find_marked(). Fixes: b803b42823d0 ("xarray: Add XArray iterators") Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-13 18:57:42 +00:00
check_find_3(xa);
XArray: Fix infinite loop with entry at ULONG_MAX If there is an entry at ULONG_MAX, xa_for_each() will overflow the 'index + 1' in xa_find_after() and wrap around to 0. Catch this case and terminate the loop by returning NULL. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-17 22:45:12 +00:00
check_find_4(xa);
XArray: Fix xa_find_after with multi-index entries If the entry is of an order which is a multiple of XA_CHUNK_SIZE, the current detection of sibling entries does not work. Factor out an xas_sibling() function to make xa_find_after() a little more understandable, and write a new implementation that doesn't suffer from the same bug. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-18 03:00:41 +00:00
for (i = 2; i < 10; i++)
check_multi_find_1(xa, i);
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
check_multi_find_2(xa);
XArray: Fix xa_find_next for large multi-index entries Coverity pointed out that xas_sibling() was shifting xa_offset without promoting it to an unsigned long first, so the shift could cause an overflow and we'd get the wrong answer. The fix is obvious, and the new test-case provokes UBSAN to report an error: runtime error: shift exponent 60 is too large for 32-bit type 'int' Fixes: 19c30f4dd092 ("XArray: Fix xa_find_after with multi-index entries") Reported-by: Bjorn Helgaas <bhelgaas@google.com> Reported-by: Kees Cook <keescook@chromium.org> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-31 10:07:55 +00:00
check_multi_find_3(xa);
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
}
shmem: Convert find_swap_entry to XArray This is a 1:1 conversion. The major part of this patch is converting the test framework from userspace to kernel space and mirroring the algorithm now used in find_swap_entry(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-22 13:36:00 +00:00
/* See find_swap_entry() in mm/shmem.c */
static noinline unsigned long xa_find_entry(struct xarray *xa, void *item)
{
XA_STATE(xas, xa, 0);
unsigned int checked = 0;
void *entry;
rcu_read_lock();
xas_for_each(&xas, entry, ULONG_MAX) {
if (xas_retry(&xas, entry))
continue;
if (entry == item)
break;
checked++;
if ((checked % 4) != 0)
continue;
xas_pause(&xas);
}
rcu_read_unlock();
return entry ? xas.xa_index : -1;
}
static noinline void check_find_entry(struct xarray *xa)
{
#ifdef CONFIG_XARRAY_MULTI
unsigned int order;
unsigned long offset, index;
for (order = 0; order < 20; order++) {
for (offset = 0; offset < (1UL << (order + 3));
offset += (1UL << order)) {
for (index = 0; index < (1UL << (order + 5));
index += (1UL << order)) {
xa_store_order(xa, index, order,
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_mk_index(index), GFP_KERNEL);
shmem: Convert find_swap_entry to XArray This is a 1:1 conversion. The major part of this patch is converting the test framework from userspace to kernel space and mirroring the algorithm now used in find_swap_entry(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-22 13:36:00 +00:00
XA_BUG_ON(xa, xa_load(xa, index) !=
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_mk_index(index));
shmem: Convert find_swap_entry to XArray This is a 1:1 conversion. The major part of this patch is converting the test framework from userspace to kernel space and mirroring the algorithm now used in find_swap_entry(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-22 13:36:00 +00:00
XA_BUG_ON(xa, xa_find_entry(xa,
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_mk_index(index)) != index);
shmem: Convert find_swap_entry to XArray This is a 1:1 conversion. The major part of this patch is converting the test framework from userspace to kernel space and mirroring the algorithm now used in find_swap_entry(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-22 13:36:00 +00:00
}
XA_BUG_ON(xa, xa_find_entry(xa, xa) != -1);
xa_destroy(xa);
}
}
#endif
XA_BUG_ON(xa, xa_find_entry(xa, xa) != -1);
xa_store_index(xa, ULONG_MAX, GFP_KERNEL);
XA_BUG_ON(xa, xa_find_entry(xa, xa) != -1);
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, xa_find_entry(xa, xa_mk_index(ULONG_MAX)) != -1);
shmem: Convert find_swap_entry to XArray This is a 1:1 conversion. The major part of this patch is converting the test framework from userspace to kernel space and mirroring the algorithm now used in find_swap_entry(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-22 13:36:00 +00:00
xa_erase_index(xa, ULONG_MAX);
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray: Fix xas_pause for large multi-index entries Inspired by the recent Coverity report, I looked for other places where the offset wasn't being converted to an unsigned long before being shifted, and I found one in xas_pause() when the entry being paused is of order >32. Fixes: b803b42823d0 ("xarray: Add XArray iterators") Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-31 11:17:09 +00:00
static noinline void check_pause(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
void *entry;
unsigned int order;
unsigned long index = 1;
unsigned int count = 0;
for (order = 0; order < order_limit; order++) {
XA_BUG_ON(xa, xa_store_order(xa, index, order,
xa_mk_index(index), GFP_KERNEL));
index += 1UL << order;
}
rcu_read_lock();
xas_for_each(&xas, entry, ULONG_MAX) {
XA_BUG_ON(xa, entry != xa_mk_index(1UL << count));
count++;
}
rcu_read_unlock();
XA_BUG_ON(xa, count != order_limit);
count = 0;
xas_set(&xas, 0);
rcu_read_lock();
xas_for_each(&xas, entry, ULONG_MAX) {
XA_BUG_ON(xa, entry != xa_mk_index(1UL << count));
count++;
xas_pause(&xas);
}
rcu_read_unlock();
XA_BUG_ON(xa, count != order_limit);
xa_destroy(xa);
}
XArray: Fix xas_next() with a single entry at 0 If there is only a single entry at 0, the first time we call xas_next(), we return the entry. Unfortunately, all subsequent times we call xas_next(), we also return the entry at 0 instead of noticing that the xa_index is now greater than zero. This broke find_get_pages_contig(). Fixes: 64d3e9a9e0cc ("xarray: Step through an XArray") Reported-by: Kent Overstreet <kent.overstreet@gmail.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2019-07-01 21:03:29 +00:00
static noinline void check_move_tiny(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
XA_BUG_ON(xa, !xa_empty(xa));
rcu_read_lock();
XA_BUG_ON(xa, xas_next(&xas) != NULL);
XA_BUG_ON(xa, xas_next(&xas) != NULL);
rcu_read_unlock();
xa_store_index(xa, 0, GFP_KERNEL);
rcu_read_lock();
xas_set(&xas, 0);
XA_BUG_ON(xa, xas_next(&xas) != xa_mk_index(0));
XA_BUG_ON(xa, xas_next(&xas) != NULL);
xas_set(&xas, 0);
XA_BUG_ON(xa, xas_prev(&xas) != xa_mk_index(0));
XA_BUG_ON(xa, xas_prev(&xas) != NULL);
rcu_read_unlock();
xa_erase_index(xa, 0);
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray: Fix xas_pause at ULONG_MAX If we were unlucky enough to call xas_pause() when the index was at ULONG_MAX (or a multi-slot entry which ends at ULONG_MAX), we would wrap the index back around to 0 and restart the iteration from the beginning. Use the XAS_BOUNDS state to indicate that we should just stop the iteration. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2019-11-08 03:49:11 +00:00
static noinline void check_move_max(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
xa_store_index(xa, ULONG_MAX, GFP_KERNEL);
rcu_read_lock();
XA_BUG_ON(xa, xas_find(&xas, ULONG_MAX) != xa_mk_index(ULONG_MAX));
XA_BUG_ON(xa, xas_find(&xas, ULONG_MAX) != NULL);
rcu_read_unlock();
xas_set(&xas, 0);
rcu_read_lock();
XA_BUG_ON(xa, xas_find(&xas, ULONG_MAX) != xa_mk_index(ULONG_MAX));
xas_pause(&xas);
XA_BUG_ON(xa, xas_find(&xas, ULONG_MAX) != NULL);
rcu_read_unlock();
xa_erase_index(xa, ULONG_MAX);
XA_BUG_ON(xa, !xa_empty(xa));
}
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
static noinline void check_move_small(struct xarray *xa, unsigned long idx)
{
XA_STATE(xas, xa, 0);
unsigned long i;
xa_store_index(xa, 0, GFP_KERNEL);
xa_store_index(xa, idx, GFP_KERNEL);
rcu_read_lock();
for (i = 0; i < idx * 4; i++) {
void *entry = xas_next(&xas);
if (i <= idx)
XA_BUG_ON(xa, xas.xa_node == XAS_RESTART);
XA_BUG_ON(xa, xas.xa_index != i);
if (i == 0 || i == idx)
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry != xa_mk_index(i));
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
else
XA_BUG_ON(xa, entry != NULL);
}
xas_next(&xas);
XA_BUG_ON(xa, xas.xa_index != i);
do {
void *entry = xas_prev(&xas);
i--;
if (i <= idx)
XA_BUG_ON(xa, xas.xa_node == XAS_RESTART);
XA_BUG_ON(xa, xas.xa_index != i);
if (i == 0 || i == idx)
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry != xa_mk_index(i));
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
else
XA_BUG_ON(xa, entry != NULL);
} while (i > 0);
xas_set(&xas, ULONG_MAX);
XA_BUG_ON(xa, xas_next(&xas) != NULL);
XA_BUG_ON(xa, xas.xa_index != ULONG_MAX);
XA_BUG_ON(xa, xas_next(&xas) != xa_mk_value(0));
XA_BUG_ON(xa, xas.xa_index != 0);
XA_BUG_ON(xa, xas_prev(&xas) != NULL);
XA_BUG_ON(xa, xas.xa_index != ULONG_MAX);
rcu_read_unlock();
xa_erase_index(xa, 0);
xa_erase_index(xa, idx);
XA_BUG_ON(xa, !xa_empty(xa));
}
static noinline void check_move(struct xarray *xa)
{
XA_STATE(xas, xa, (1 << 16) - 1);
unsigned long i;
for (i = 0; i < (1 << 16); i++)
XA_BUG_ON(xa, xa_store_index(xa, i, GFP_KERNEL) != NULL);
rcu_read_lock();
do {
void *entry = xas_prev(&xas);
i--;
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry != xa_mk_index(i));
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
XA_BUG_ON(xa, i != xas.xa_index);
} while (i != 0);
XA_BUG_ON(xa, xas_prev(&xas) != NULL);
XA_BUG_ON(xa, xas.xa_index != ULONG_MAX);
do {
void *entry = xas_next(&xas);
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry != xa_mk_index(i));
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
XA_BUG_ON(xa, i != xas.xa_index);
i++;
} while (i < (1 << 16));
rcu_read_unlock();
for (i = (1 << 8); i < (1 << 15); i++)
xa_erase_index(xa, i);
i = xas.xa_index;
rcu_read_lock();
do {
void *entry = xas_prev(&xas);
i--;
if ((i < (1 << 8)) || (i >= (1 << 15)))
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry != xa_mk_index(i));
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
else
XA_BUG_ON(xa, entry != NULL);
XA_BUG_ON(xa, i != xas.xa_index);
} while (i != 0);
XA_BUG_ON(xa, xas_prev(&xas) != NULL);
XA_BUG_ON(xa, xas.xa_index != ULONG_MAX);
do {
void *entry = xas_next(&xas);
if ((i < (1 << 8)) || (i >= (1 << 15)))
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, entry != xa_mk_index(i));
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
else
XA_BUG_ON(xa, entry != NULL);
XA_BUG_ON(xa, i != xas.xa_index);
i++;
} while (i < (1 << 16));
rcu_read_unlock();
xa_destroy(xa);
XArray: Fix xas_next() with a single entry at 0 If there is only a single entry at 0, the first time we call xas_next(), we return the entry. Unfortunately, all subsequent times we call xas_next(), we also return the entry at 0 instead of noticing that the xa_index is now greater than zero. This broke find_get_pages_contig(). Fixes: 64d3e9a9e0cc ("xarray: Step through an XArray") Reported-by: Kent Overstreet <kent.overstreet@gmail.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2019-07-01 21:03:29 +00:00
check_move_tiny(xa);
XArray: Fix xas_pause at ULONG_MAX If we were unlucky enough to call xas_pause() when the index was at ULONG_MAX (or a multi-slot entry which ends at ULONG_MAX), we would wrap the index back around to 0 and restart the iteration from the beginning. Use the XAS_BOUNDS state to indicate that we should just stop the iteration. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2019-11-08 03:49:11 +00:00
check_move_max(xa);
XArray: Fix xas_next() with a single entry at 0 If there is only a single entry at 0, the first time we call xas_next(), we return the entry. Unfortunately, all subsequent times we call xas_next(), we also return the entry at 0 instead of noticing that the xa_index is now greater than zero. This broke find_get_pages_contig(). Fixes: 64d3e9a9e0cc ("xarray: Step through an XArray") Reported-by: Kent Overstreet <kent.overstreet@gmail.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2019-07-01 21:03:29 +00:00
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
for (i = 0; i < 16; i++)
check_move_small(xa, 1UL << i);
for (i = 2; i < 16; i++)
check_move_small(xa, (1UL << i) - 1);
}
xarray: Add xas_create_range This hopefully temporary function is useful for users who have not yet been converted to multi-index entries. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-04 05:11:48 +00:00
static noinline void xa_store_many_order(struct xarray *xa,
unsigned long index, unsigned order)
{
XA_STATE_ORDER(xas, xa, index, order);
unsigned int i = 0;
do {
xas_lock(&xas);
XA_BUG_ON(xa, xas_find_conflict(&xas));
xas_create_range(&xas);
if (xas_error(&xas))
goto unlock;
for (i = 0; i < (1U << order); i++) {
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, xas_store(&xas, xa_mk_index(index + i)));
xarray: Add xas_create_range This hopefully temporary function is useful for users who have not yet been converted to multi-index entries. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-04 05:11:48 +00:00
xas_next(&xas);
}
unlock:
xas_unlock(&xas);
} while (xas_nomem(&xas, GFP_KERNEL));
XA_BUG_ON(xa, xas_error(&xas));
}
static noinline void check_create_range_1(struct xarray *xa,
unsigned long index, unsigned order)
{
unsigned long i;
xa_store_many_order(xa, index, order);
for (i = index; i < index + (1UL << order); i++)
xa_erase_index(xa, i);
XA_BUG_ON(xa, !xa_empty(xa));
}
static noinline void check_create_range_2(struct xarray *xa, unsigned order)
{
unsigned long i;
unsigned long nr = 1UL << order;
for (i = 0; i < nr * nr; i += nr)
xa_store_many_order(xa, i, order);
for (i = 0; i < nr * nr; i++)
xa_erase_index(xa, i);
XA_BUG_ON(xa, !xa_empty(xa));
}
static noinline void check_create_range_3(void)
{
XA_STATE(xas, NULL, 0);
xas_set_err(&xas, -EEXIST);
xas_create_range(&xas);
XA_BUG_ON(NULL, xas_error(&xas) != -EEXIST);
}
static noinline void check_create_range_4(struct xarray *xa,
unsigned long index, unsigned order)
{
XA_STATE_ORDER(xas, xa, index, order);
unsigned long base = xas.xa_index;
unsigned long i = 0;
xa_store_index(xa, index, GFP_KERNEL);
do {
xas_lock(&xas);
xas_create_range(&xas);
if (xas_error(&xas))
goto unlock;
for (i = 0; i < (1UL << order); i++) {
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
void *old = xas_store(&xas, xa_mk_index(base + i));
xarray: Add xas_create_range This hopefully temporary function is useful for users who have not yet been converted to multi-index entries. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-04 05:11:48 +00:00
if (xas.xa_index == index)
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, old != xa_mk_index(base + i));
xarray: Add xas_create_range This hopefully temporary function is useful for users who have not yet been converted to multi-index entries. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-04 05:11:48 +00:00
else
XA_BUG_ON(xa, old != NULL);
xas_next(&xas);
}
unlock:
xas_unlock(&xas);
} while (xas_nomem(&xas, GFP_KERNEL));
XA_BUG_ON(xa, xas_error(&xas));
for (i = base; i < base + (1UL << order); i++)
xa_erase_index(xa, i);
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray: Fix xas_create_range() when multi-order entry present If there is already an entry present that is of order >= XA_CHUNK_SHIFT when we call xas_create_range(), xas_create_range() will misinterpret that entry as a node and dereference xa_node->parent, generally leading to a crash that looks something like this: general protection fault, probably for non-canonical address 0xdffffc0000000001: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x0000000000000008-0x000000000000000f] CPU: 0 PID: 32 Comm: khugepaged Not tainted 5.17.0-rc8-syzkaller-00003-g56e337f2cf13 #0 RIP: 0010:xa_parent_locked include/linux/xarray.h:1207 [inline] RIP: 0010:xas_create_range+0x2d9/0x6e0 lib/xarray.c:725 It's deterministically reproducable once you know what the problem is, but producing it in a live kernel requires khugepaged to hit a race. While the problem has been present since xas_create_range() was introduced, I'm not aware of a way to hit it before the page cache was converted to use multi-index entries. Fixes: 6b24ca4a1a8d ("mm: Use multi-index entries in the page cache") Reported-by: syzbot+0d2b0bf32ca5cfd09f2e@syzkaller.appspotmail.com Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2022-03-28 23:25:11 +00:00
static noinline void check_create_range_5(struct xarray *xa,
unsigned long index, unsigned int order)
{
XA_STATE_ORDER(xas, xa, index, order);
unsigned int i;
xa_store_order(xa, index, order, xa_mk_index(index), GFP_KERNEL);
for (i = 0; i < order + 10; i++) {
do {
xas_lock(&xas);
xas_create_range(&xas);
xas_unlock(&xas);
} while (xas_nomem(&xas, GFP_KERNEL));
}
xa_destroy(xa);
}
xarray: Add xas_create_range This hopefully temporary function is useful for users who have not yet been converted to multi-index entries. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-04 05:11:48 +00:00
static noinline void check_create_range(struct xarray *xa)
{
unsigned int order;
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 12 : 1;
for (order = 0; order < max_order; order++) {
check_create_range_1(xa, 0, order);
check_create_range_1(xa, 1U << order, order);
check_create_range_1(xa, 2U << order, order);
check_create_range_1(xa, 3U << order, order);
check_create_range_1(xa, 1U << 24, order);
if (order < 10)
check_create_range_2(xa, order);
check_create_range_4(xa, 0, order);
check_create_range_4(xa, 1U << order, order);
check_create_range_4(xa, 2U << order, order);
check_create_range_4(xa, 3U << order, order);
check_create_range_4(xa, 1U << 24, order);
check_create_range_4(xa, 1, order);
check_create_range_4(xa, (1U << order) + 1, order);
check_create_range_4(xa, (2U << order) + 1, order);
check_create_range_4(xa, (2U << order) - 1, order);
check_create_range_4(xa, (3U << order) + 1, order);
check_create_range_4(xa, (3U << order) - 1, order);
check_create_range_4(xa, (1U << 24) + 1, order);
XArray: Fix xas_create_range() when multi-order entry present If there is already an entry present that is of order >= XA_CHUNK_SHIFT when we call xas_create_range(), xas_create_range() will misinterpret that entry as a node and dereference xa_node->parent, generally leading to a crash that looks something like this: general protection fault, probably for non-canonical address 0xdffffc0000000001: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x0000000000000008-0x000000000000000f] CPU: 0 PID: 32 Comm: khugepaged Not tainted 5.17.0-rc8-syzkaller-00003-g56e337f2cf13 #0 RIP: 0010:xa_parent_locked include/linux/xarray.h:1207 [inline] RIP: 0010:xas_create_range+0x2d9/0x6e0 lib/xarray.c:725 It's deterministically reproducable once you know what the problem is, but producing it in a live kernel requires khugepaged to hit a race. While the problem has been present since xas_create_range() was introduced, I'm not aware of a way to hit it before the page cache was converted to use multi-index entries. Fixes: 6b24ca4a1a8d ("mm: Use multi-index entries in the page cache") Reported-by: syzbot+0d2b0bf32ca5cfd09f2e@syzkaller.appspotmail.com Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2022-03-28 23:25:11 +00:00
check_create_range_5(xa, 0, order);
check_create_range_5(xa, (1U << order), order);
xarray: Add xas_create_range This hopefully temporary function is useful for users who have not yet been converted to multi-index entries. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-04 05:11:48 +00:00
}
check_create_range_3();
}
xarray: Add range store functionality This version of xa_store_range() really only supports load and store. Our only user only needs basic load and store functionality, so there's no need to do the extra work to support marking and overlapping stores correctly yet. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-15 18:13:29 +00:00
static noinline void __check_store_range(struct xarray *xa, unsigned long first,
unsigned long last)
{
#ifdef CONFIG_XARRAY_MULTI
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_store_range(xa, first, last, xa_mk_index(first), GFP_KERNEL);
xarray: Add range store functionality This version of xa_store_range() really only supports load and store. Our only user only needs basic load and store functionality, so there's no need to do the extra work to support marking and overlapping stores correctly yet. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-15 18:13:29 +00:00
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
XA_BUG_ON(xa, xa_load(xa, first) != xa_mk_index(first));
XA_BUG_ON(xa, xa_load(xa, last) != xa_mk_index(first));
xarray: Add range store functionality This version of xa_store_range() really only supports load and store. Our only user only needs basic load and store functionality, so there's no need to do the extra work to support marking and overlapping stores correctly yet. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-15 18:13:29 +00:00
XA_BUG_ON(xa, xa_load(xa, first - 1) != NULL);
XA_BUG_ON(xa, xa_load(xa, last + 1) != NULL);
xa_store_range(xa, first, last, NULL, GFP_KERNEL);
#endif
XA_BUG_ON(xa, !xa_empty(xa));
}
static noinline void check_store_range(struct xarray *xa)
{
unsigned long i, j;
for (i = 0; i < 128; i++) {
for (j = i; j < 128; j++) {
__check_store_range(xa, i, j);
__check_store_range(xa, 128 + i, 128 + j);
__check_store_range(xa, 4095 + i, 4095 + j);
__check_store_range(xa, 4096 + i, 4096 + j);
__check_store_range(xa, 123456 + i, 123456 + j);
XArray tests: Correct some 64-bit assumptions The test-suite caught these two mistakes when compiled for 32-bit. I had only been running the test-suite in 64-bit mode. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 14:34:04 +00:00
__check_store_range(xa, (1 << 24) + i, (1 << 24) + j);
xarray: Add range store functionality This version of xa_store_range() really only supports load and store. Our only user only needs basic load and store functionality, so there's no need to do the extra work to support marking and overlapping stores correctly yet. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-15 18:13:29 +00:00
}
}
}
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
#ifdef CONFIG_XARRAY_MULTI
static void check_split_1(struct xarray *xa, unsigned long index,
XArray: Fix splitting to non-zero orders Splitting an order-4 entry into order-2 entries would leave the array containing pointers to 000040008000c000 instead of 000044448888cccc. This is a one-character fix, but enhance the test suite to check this case. Reported-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-11-19 13:32:31 +00:00
unsigned int order, unsigned int new_order)
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
{
XArray: Fix splitting to non-zero orders Splitting an order-4 entry into order-2 entries would leave the array containing pointers to 000040008000c000 instead of 000044448888cccc. This is a one-character fix, but enhance the test suite to check this case. Reported-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-11-19 13:32:31 +00:00
XA_STATE_ORDER(xas, xa, index, new_order);
XArray: set the marks correctly when splitting an entry If we created a new node to replace an entry which had search marks set, we were setting the search mark on every entry in that node. That works fine when we're splitting to order 0, but when splitting to a larger order, we must not set the search marks on the sibling entries. Link: https://lkml.kernel.org/r/20240501153120.4094530-1-willy@infradead.org Fixes: c010d47f107f ("mm: thp: split huge page to any lower order pages") Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reported-by: Luis Chamberlain <mcgrof@kernel.org> Link: https://lore.kernel.org/r/ZjFGCOYk3FK_zVy3@bombadil.infradead.org Tested-by: Luis Chamberlain <mcgrof@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-05-01 15:31:18 +00:00
unsigned int i, found;
void *entry;
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
xa_store_order(xa, index, order, xa, GFP_KERNEL);
XArray: set the marks correctly when splitting an entry If we created a new node to replace an entry which had search marks set, we were setting the search mark on every entry in that node. That works fine when we're splitting to order 0, but when splitting to a larger order, we must not set the search marks on the sibling entries. Link: https://lkml.kernel.org/r/20240501153120.4094530-1-willy@infradead.org Fixes: c010d47f107f ("mm: thp: split huge page to any lower order pages") Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reported-by: Luis Chamberlain <mcgrof@kernel.org> Link: https://lore.kernel.org/r/ZjFGCOYk3FK_zVy3@bombadil.infradead.org Tested-by: Luis Chamberlain <mcgrof@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-05-01 15:31:18 +00:00
xa_set_mark(xa, index, XA_MARK_1);
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
xas_split_alloc(&xas, xa, order, GFP_KERNEL);
xas_lock(&xas);
xas_split(&xas, xa, order);
XArray: Fix splitting to non-zero orders Splitting an order-4 entry into order-2 entries would leave the array containing pointers to 000040008000c000 instead of 000044448888cccc. This is a one-character fix, but enhance the test suite to check this case. Reported-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-11-19 13:32:31 +00:00
for (i = 0; i < (1 << order); i += (1 << new_order))
__xa_store(xa, index + i, xa_mk_index(index + i), 0);
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
xas_unlock(&xas);
XArray: Fix splitting to non-zero orders Splitting an order-4 entry into order-2 entries would leave the array containing pointers to 000040008000c000 instead of 000044448888cccc. This is a one-character fix, but enhance the test suite to check this case. Reported-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-11-19 13:32:31 +00:00
for (i = 0; i < (1 << order); i++) {
unsigned int val = index + (i & ~((1 << new_order) - 1));
XA_BUG_ON(xa, xa_load(xa, index + i) != xa_mk_index(val));
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
}
xa_set_mark(xa, index, XA_MARK_0);
XA_BUG_ON(xa, !xa_get_mark(xa, index, XA_MARK_0));
XArray: set the marks correctly when splitting an entry If we created a new node to replace an entry which had search marks set, we were setting the search mark on every entry in that node. That works fine when we're splitting to order 0, but when splitting to a larger order, we must not set the search marks on the sibling entries. Link: https://lkml.kernel.org/r/20240501153120.4094530-1-willy@infradead.org Fixes: c010d47f107f ("mm: thp: split huge page to any lower order pages") Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Reported-by: Luis Chamberlain <mcgrof@kernel.org> Link: https://lore.kernel.org/r/ZjFGCOYk3FK_zVy3@bombadil.infradead.org Tested-by: Luis Chamberlain <mcgrof@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-05-01 15:31:18 +00:00
xas_set_order(&xas, index, 0);
found = 0;
rcu_read_lock();
xas_for_each_marked(&xas, entry, ULONG_MAX, XA_MARK_1) {
found++;
XA_BUG_ON(xa, xa_is_internal(entry));
}
rcu_read_unlock();
XA_BUG_ON(xa, found != 1 << (order - new_order));
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
xa_destroy(xa);
}
static noinline void check_split(struct xarray *xa)
{
XArray: Fix splitting to non-zero orders Splitting an order-4 entry into order-2 entries would leave the array containing pointers to 000040008000c000 instead of 000044448888cccc. This is a one-character fix, but enhance the test suite to check this case. Reported-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-11-19 13:32:31 +00:00
unsigned int order, new_order;
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
XA_BUG_ON(xa, !xa_empty(xa));
for (order = 1; order < 2 * XA_CHUNK_SHIFT; order++) {
XArray: Fix splitting to non-zero orders Splitting an order-4 entry into order-2 entries would leave the array containing pointers to 000040008000c000 instead of 000044448888cccc. This is a one-character fix, but enhance the test suite to check this case. Reported-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-11-19 13:32:31 +00:00
for (new_order = 0; new_order < order; new_order++) {
check_split_1(xa, 0, order, new_order);
check_split_1(xa, 1UL << order, order, new_order);
check_split_1(xa, 3UL << order, order, new_order);
}
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
}
}
#else
static void check_split(struct xarray *xa) { }
#endif
XArray: Permit storing 2-byte-aligned pointers On m68k, statically allocated pointers may only be two-byte aligned. This clashes with the XArray's method for tagging internal pointers. Permit storing these pointers in single slots (ie not in multislots). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-29 04:20:44 +00:00
static void check_align_1(struct xarray *xa, char *name)
{
int i;
unsigned int id;
unsigned long index;
void *entry;
for (i = 0; i < 8; i++) {
XArray: Redesign xa_alloc API It was too easy to forget to initialise the start index. Add an xa_limit data structure which can be used to pass min & max, and define a couple of special values for common cases. Also add some more tests cribbed from the IDR test suite. Change the return value from -ENOSPC to -EBUSY to match xa_insert(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-31 15:41:01 +00:00
XA_BUG_ON(xa, xa_alloc(xa, &id, name + i, xa_limit_32b,
GFP_KERNEL) != 0);
XArray: Permit storing 2-byte-aligned pointers On m68k, statically allocated pointers may only be two-byte aligned. This clashes with the XArray's method for tagging internal pointers. Permit storing these pointers in single slots (ie not in multislots). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-29 04:20:44 +00:00
XA_BUG_ON(xa, id != i);
}
xa_for_each(xa, index, entry)
XA_BUG_ON(xa, xa_is_err(entry));
xa_destroy(xa);
}
XArray: Fix xa_reserve for 2-byte aligned entries If we reserve index 0, the next entry to be stored there might be 2-byte aligned. That means we have to create the root xa_node at the time of reserving the initial entry. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-21 22:54:44 +00:00
/*
* We should always be able to store without allocating memory after
* reserving a slot.
*/
XArray: Fix xa_erase of 2-byte aligned entries xas_store() was interpreting the entry it found in the array as a node entry if the bottom two bits had value 2. That's only true if either the entry is in the root node or in a non-leaf node. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-21 22:36:45 +00:00
static void check_align_2(struct xarray *xa, char *name)
{
int i;
XA_BUG_ON(xa, !xa_empty(xa));
for (i = 0; i < 8; i++) {
XA_BUG_ON(xa, xa_store(xa, 0, name + i, GFP_KERNEL) != NULL);
xa_erase(xa, 0);
}
XArray: Fix xa_reserve for 2-byte aligned entries If we reserve index 0, the next entry to be stored there might be 2-byte aligned. That means we have to create the root xa_node at the time of reserving the initial entry. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-21 22:54:44 +00:00
for (i = 0; i < 8; i++) {
XA_BUG_ON(xa, xa_reserve(xa, 0, GFP_KERNEL) != 0);
XA_BUG_ON(xa, xa_store(xa, 0, name + i, 0) != NULL);
xa_erase(xa, 0);
}
XArray: Fix xa_erase of 2-byte aligned entries xas_store() was interpreting the entry it found in the array as a node entry if the bottom two bits had value 2. That's only true if either the entry is in the root node or in a non-leaf node. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-21 22:36:45 +00:00
XA_BUG_ON(xa, !xa_empty(xa));
}
XArray: Permit storing 2-byte-aligned pointers On m68k, statically allocated pointers may only be two-byte aligned. This clashes with the XArray's method for tagging internal pointers. Permit storing these pointers in single slots (ie not in multislots). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-29 04:20:44 +00:00
static noinline void check_align(struct xarray *xa)
{
char name[] = "Motorola 68000";
check_align_1(xa, name);
check_align_1(xa, name + 1);
check_align_1(xa, name + 2);
check_align_1(xa, name + 3);
XArray: Fix xa_erase of 2-byte aligned entries xas_store() was interpreting the entry it found in the array as a node entry if the bottom two bits had value 2. That's only true if either the entry is in the root node or in a non-leaf node. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-21 22:36:45 +00:00
check_align_2(xa, name);
XArray: Permit storing 2-byte-aligned pointers On m68k, statically allocated pointers may only be two-byte aligned. This clashes with the XArray's method for tagging internal pointers. Permit storing these pointers in single slots (ie not in multislots). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-29 04:20:44 +00:00
}
mm: Convert workingset to XArray We construct an XA_STATE and use it to delete the node with xas_store() rather than adding a special function for this unique use case. Includes a test that simulates this usage for the test suite. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-24 19:24:59 +00:00
static LIST_HEAD(shadow_nodes);
static void test_update_node(struct xa_node *node)
{
if (node->count && node->count == node->nr_values) {
if (list_empty(&node->private_list))
list_add(&shadow_nodes, &node->private_list);
} else {
if (!list_empty(&node->private_list))
list_del_init(&node->private_list);
}
}
static noinline void shadow_remove(struct xarray *xa)
{
struct xa_node *node;
xa_lock(xa);
while ((node = list_first_entry_or_null(&shadow_nodes,
struct xa_node, private_list))) {
XA_BUG_ON(xa, node->array != xa);
list_del_init(&node->private_list);
XArray: Add private interface for workingset node deletion Move the tricky bits of dealing with the XArray from the workingset code to the XArray. Make it clear in the documentation that this is a private interface, and only export it for the benefit of the test suite. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
2020-08-18 13:05:56 +00:00
xa_delete_node(node, test_update_node);
mm: Convert workingset to XArray We construct an XA_STATE and use it to delete the node with xas_store() rather than adding a special function for this unique use case. Includes a test that simulates this usage for the test suite. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-24 19:24:59 +00:00
}
xa_unlock(xa);
}
static noinline void check_workingset(struct xarray *xa, unsigned long index)
{
XA_STATE(xas, xa, index);
xas_set_update(&xas, test_update_node);
do {
xas_lock(&xas);
xas_store(&xas, xa_mk_value(0));
xas_next(&xas);
xas_store(&xas, xa_mk_value(1));
xas_unlock(&xas);
} while (xas_nomem(&xas, GFP_KERNEL));
XA_BUG_ON(xa, list_empty(&shadow_nodes));
xas_lock(&xas);
xas_next(&xas);
xas_store(&xas, &xas);
XA_BUG_ON(xa, !list_empty(&shadow_nodes));
xas_store(&xas, xa_mk_value(2));
xas_unlock(&xas);
XA_BUG_ON(xa, list_empty(&shadow_nodes));
shadow_remove(xa);
XA_BUG_ON(xa, !list_empty(&shadow_nodes));
XA_BUG_ON(xa, !xa_empty(xa));
}
xarray: Move multiorder account test in-kernel Move this test to the in-kernel test suite, and enhance it to test several different orders. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-28 20:13:16 +00:00
/*
* Check that the pointer / value / sibling entries are accounted the
* way we expect them to be.
*/
static noinline void check_account(struct xarray *xa)
{
#ifdef CONFIG_XARRAY_MULTI
unsigned int order;
for (order = 1; order < 12; order++) {
XA_STATE(xas, xa, 1 << order);
xa_store_order(xa, 0, order, xa, GFP_KERNEL);
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
rcu_read_lock();
xarray: Move multiorder account test in-kernel Move this test to the in-kernel test suite, and enhance it to test several different orders. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-28 20:13:16 +00:00
xas_load(&xas);
XA_BUG_ON(xa, xas.xa_node->count == 0);
XA_BUG_ON(xa, xas.xa_node->count > (1 << order));
XA_BUG_ON(xa, xas.xa_node->nr_values != 0);
XArray tests: Add missing locking Lockdep caught me being sloppy in the test suite and failing to lock the XArray appropriately. Reported-by: kernel test robot <rong.a.chen@intel.com> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-19 14:36:29 +00:00
rcu_read_unlock();
xarray: Move multiorder account test in-kernel Move this test to the in-kernel test suite, and enhance it to test several different orders. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-28 20:13:16 +00:00
XArray tests: Handle larger indices more elegantly xa_mk_value() only handles values up to LONG_MAX. I successfully hid that inside xa_store_index() and xa_erase_index(), but it turned out I also needed it for testing xa_alloc() on 32-bit machines. So extract xa_mk_index() from the above two functions, and convert the non-constant users of xa_mk_value() to xa_mk_index(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-11-05 18:19:54 +00:00
xa_store_order(xa, 1 << order, order, xa_mk_index(1UL << order),
xarray: Move multiorder account test in-kernel Move this test to the in-kernel test suite, and enhance it to test several different orders. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-28 20:13:16 +00:00
GFP_KERNEL);
XA_BUG_ON(xa, xas.xa_node->count != xas.xa_node->nr_values * 2);
xa_erase(xa, 1 << order);
XA_BUG_ON(xa, xas.xa_node->nr_values != 0);
xa_erase(xa, 0);
XA_BUG_ON(xa, !xa_empty(xa));
}
#endif
}
XArray: add xa_get_order Patch series "Fix read-only THP for non-tmpfs filesystems". As described more verbosely in the [3/3] changelog, we can inadvertently put an order-0 page in the page cache which occupies 512 consecutive entries. Users are running into this if they enable the READ_ONLY_THP_FOR_FS config option; see https://bugzilla.kernel.org/show_bug.cgi?id=206569 and Qian Cai has also reported it here: https://lore.kernel.org/lkml/20200616013309.GB815@lca.pw/ This is a rather intrusive way of fixing the problem, but has the advantage that I've actually been testing it with the THP patches, which means that it sees far more use than it does upstream -- indeed, Song has been entirely unable to reproduce it. It also has the advantage that it removes a few patches from my gargantuan backlog of THP patches. This patch (of 3): This function returns the order of the entry at the index. We need this because there isn't space in the shadow entry to encode its order. [akpm@linux-foundation.org: export xa_get_order to modules] Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-1-willy@infradead.org Link: https://lkml.kernel.org/r/20200903183029.14930-2-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:13 +00:00
static noinline void check_get_order(struct xarray *xa)
{
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1;
unsigned int order;
unsigned long i, j;
for (i = 0; i < 3; i++)
XA_BUG_ON(xa, xa_get_order(xa, i) != 0);
for (order = 0; order < max_order; order++) {
for (i = 0; i < 10; i++) {
xa_store_order(xa, i << order, order,
xa_mk_index(i << order), GFP_KERNEL);
for (j = i << order; j < (i + 1) << order; j++)
XA_BUG_ON(xa, xa_get_order(xa, j) != order);
xa_erase(xa, i << order);
}
}
}
lib/xarray: introduce a new helper xas_get_order It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Added new test for xas_get_order, to make the test work, we have to export xas_get_order with EXPORT_SYMBOL_GPL. Also fix a sparse warning by checking the slot value with xa_entry instead of accessing it directly, as suggested by Matthew Wilcox. [kasong@tencent.com: simplify comment, sparse warning fix, per Matthew Wilcox] Link: https://lkml.kernel.org/r/20240416071722.45997-4-ryncsn@gmail.com Link: https://lkml.kernel.org/r/20240415171857.19244-4-ryncsn@gmail.com Signed-off-by: Kairui Song <kasong@tencent.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-04-15 17:18:55 +00:00
static noinline void check_xas_get_order(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1;
unsigned int order;
unsigned long i, j;
for (order = 0; order < max_order; order++) {
for (i = 0; i < 10; i++) {
xas_set_order(&xas, i << order, order);
do {
xas_lock(&xas);
xas_store(&xas, xa_mk_value(i));
xas_unlock(&xas);
} while (xas_nomem(&xas, GFP_KERNEL));
for (j = i << order; j < (i + 1) << order; j++) {
xas_set_order(&xas, j, 0);
rcu_read_lock();
xas_load(&xas);
XA_BUG_ON(xa, xas_get_order(&xas) != order);
rcu_read_unlock();
}
xas_lock(&xas);
xas_set_order(&xas, i << order, order);
xas_store(&xas, NULL);
xas_unlock(&xas);
}
}
}
mm/filemap: optimize filemap folio adding Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranged conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4K pages, in an 8G cgroup, with 16G brd as block device: echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap --rw=randread --time_based \ --ramp_time=30s --runtime=5m --group_reporting Before: bw ( MiB/s): min= 1027, max= 3520, per=100.00%, avg=2445.02, stdev=18.90, samples=8691 iops : min=263001, max=901288, avg=625924.36, stdev=4837.28, samples=8691 After (+7.3%): bw ( MiB/s): min= 493, max= 3947, per=100.00%, avg=2625.56, stdev=25.74, samples=8651 iops : min=126454, max=1010681, avg=672142.61, stdev=6590.48, samples=8651 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap -thp=1 --readonly \ --rw=randread --time_based --ramp_time=30s --runtime=10m \ --group_reporting fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap \ --rw=randread --time_based --runtime=5s --group_reporting Before: bw ( KiB/s): min= 4141, max=14202, per=100.00%, avg=7935.51, stdev=96.85, samples=18976 iops : min= 1029, max= 3548, avg=1979.52, stdev=24.23, samples=18976· READ: bw=4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=64.0KiB (65.5kB), run=14419-14419msec After (+12.5%): bw ( KiB/s): min= 4611, max=15370, per=100.00%, avg=8928.74, stdev=105.17, samples=19146 iops : min= 1151, max= 3842, avg=2231.27, stdev=26.29, samples=19146 READ: bw=4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=64.0KiB (65.5kB), run=14137-14137msec The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read. Link: https://lkml.kernel.org/r/20240415171857.19244-5-ryncsn@gmail.com Signed-off-by: Kairui Song <kasong@tencent.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-04-15 17:18:56 +00:00
static noinline void check_xas_conflict_get_order(struct xarray *xa)
{
XA_STATE(xas, xa, 0);
void *entry;
int only_once;
unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1;
unsigned int order;
unsigned long i, j, k;
for (order = 0; order < max_order; order++) {
for (i = 0; i < 10; i++) {
xas_set_order(&xas, i << order, order);
do {
xas_lock(&xas);
xas_store(&xas, xa_mk_value(i));
xas_unlock(&xas);
} while (xas_nomem(&xas, GFP_KERNEL));
/*
* Ensure xas_get_order works with xas_for_each_conflict.
*/
j = i << order;
for (k = 0; k < order; k++) {
only_once = 0;
xas_set_order(&xas, j + (1 << k), k);
xas_lock(&xas);
xas_for_each_conflict(&xas, entry) {
XA_BUG_ON(xa, entry != xa_mk_value(i));
XA_BUG_ON(xa, xas_get_order(&xas) != order);
only_once++;
}
XA_BUG_ON(xa, only_once != 1);
xas_unlock(&xas);
}
if (order < max_order - 1) {
only_once = 0;
xas_set_order(&xas, (i & ~1UL) << order, order + 1);
xas_lock(&xas);
xas_for_each_conflict(&xas, entry) {
XA_BUG_ON(xa, entry != xa_mk_value(i));
XA_BUG_ON(xa, xas_get_order(&xas) != order);
only_once++;
}
XA_BUG_ON(xa, only_once != 1);
xas_unlock(&xas);
}
xas_set_order(&xas, i << order, order);
xas_lock(&xas);
xas_store(&xas, NULL);
xas_unlock(&xas);
}
}
}
xarray: Destroy an XArray This function frees all the internal memory allocated to the xarray and reinitialises it to be empty. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-17 13:16:34 +00:00
static noinline void check_destroy(struct xarray *xa)
{
unsigned long index;
XA_BUG_ON(xa, !xa_empty(xa));
/* Destroying an empty array is a no-op */
xa_destroy(xa);
XA_BUG_ON(xa, !xa_empty(xa));
/* Destroying an array with a single entry */
for (index = 0; index < 1000; index++) {
xa_store_index(xa, index, GFP_KERNEL);
XA_BUG_ON(xa, xa_empty(xa));
xa_destroy(xa);
XA_BUG_ON(xa, !xa_empty(xa));
}
/* Destroying an array with a single entry at ULONG_MAX */
xa_store(xa, ULONG_MAX, xa, GFP_KERNEL);
XA_BUG_ON(xa, xa_empty(xa));
xa_destroy(xa);
XA_BUG_ON(xa, !xa_empty(xa));
#ifdef CONFIG_XARRAY_MULTI
/* Destroying an array with a multi-index entry */
xa_store_order(xa, 1 << 11, 11, xa, GFP_KERNEL);
XA_BUG_ON(xa, xa_empty(xa));
xa_destroy(xa);
XA_BUG_ON(xa, !xa_empty(xa));
#endif
}
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
static DEFINE_XARRAY(array);
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
static int xarray_checks(void)
{
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
check_xa_err(&array);
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
check_xas_retry(&array);
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
check_xa_load(&array);
xarray: Add XArray marks XArray marks are like the radix tree tags, only slightly more strongly typed. They are renamed in order to distinguish them from tagged pointers. This commit adds the basic get/set/clear operations. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 14:34:31 +00:00
check_xa_mark(&array);
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
check_xa_shrink(&array);
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
check_xas_erase(&array);
XArray tests: Add check_insert A simple test which just checks that inserting an entry into an empty array succeeds. Try various different interesting indices. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-03-10 03:25:27 +00:00
check_insert(&array);
xarray: Add XArray conditional store operations Like cmpxchg(), xa_cmpxchg will only store to the index if the current entry matches the old entry. It returns the current entry, which is usually more useful than the errno returned by radix_tree_insert(). For the users who really only want the errno, the xa_insert() wrapper provides a more convenient calling convention. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:34:55 +00:00
check_cmpxchg(&array);
XArray: add cmpxchg order test XArray multi-index entries do not keep track of the order stored once the entry is being marked as used with cmpxchg (conditionally replaced with NULL). Add a test to check the order is actually lost. The test also verifies the order and entries for all the tied indexes before and after the NULL replacement with xa_cmpxchg. Add another entry at 1 << order that keeps the node around and the order information for the NULL-entry after xa_cmpxchg. Link: https://lkml.kernel.org/r/20240131225125.1370598-3-mcgrof@kernel.org Signed-off-by: Daniel Gomez <da.gomez@samsung.com> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Cc: Darrick J. Wong <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Hannes Reinecke <hare@suse.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-01-31 22:51:25 +00:00
check_cmpxchg_order(&array);
xarray: Add xa_reserve and xa_release This function reserves a slot in the XArray for users which need to acquire multiple locks before storing their entry in the tree and so cannot use a plain xa_store(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-10-01 18:54:59 +00:00
check_reserve(&array);
XArray: Fix xa_release in allocating arrays xa_cmpxchg() was a little too magic in turning ZERO entries into NULL, and would leave the entry set to the ZERO entry instead of releasing it for future use. After careful review of existing users of xa_cmpxchg(), change the semantics so that it does not translate either incoming argument from NULL into ZERO entries. Add several tests to the test-suite to make sure this problem doesn't come back. Reported-by: Jason Gunthorpe <jgg@ziepe.ca> Signed-off-by: Matthew Wilcox <willy@infradead.org>
2019-02-20 16:30:49 +00:00
check_reserve(&xa0);
xarray: Add XArray unconditional store operations xa_store() differs from radix_tree_insert() in that it will overwrite an existing element in the array rather than returning an error. This is the behaviour which most users want, and those that want more complex behaviour generally want to use the xas family of routines anyway. For memory allocation, xa_store() will first attempt to request memory from the slab allocator; if memory is not immediately available, it will drop the xa_lock and allocate memory, keeping a pointer in the xa_state. It does not use the per-CPU cache, although those will continue to exist until all radix tree users are converted to the xarray. This patch also includes xa_erase() and __xa_erase() for a streamlined way to store NULL. Since there is no need to allocate memory in order to store a NULL in the XArray, we do not need to trouble the user with deciding what memory allocation flags to use. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-10 20:15:08 +00:00
check_multi_store(&array);
test_xarray: add tests for advanced multi-index use Patch series "test_xarray: advanced API multi-index tests", v2. This is a respin of the test_xarray multi-index tests [0] which use and demonstrate the advanced API which is used by the page cache. This should let folks more easily follow how we use multi-index to support for example a min order later in the page cache. It also lets us grow the selftests to mimic more of what we do in the page cache. This patch (of 2): The multi index selftests are great but they don't replicate how we deal with the page cache exactly, which makes it a bit hard to follow as the page cache uses the advanced API. Add tests which use the advanced API, mimicking what we do in the page cache, while at it, extend the example to do what is needed for min order support. [mcgrof@kernel.org: fix soft lockup for advanced-api tests] Link: https://lkml.kernel.org/r/20240216194329.840555-1-mcgrof@kernel.org [akpm@linux-foundation.org: s/i/loops/, make non-static] [akpm@linux-foundation.org: restore static storage for loop counter] Link: https://lkml.kernel.org/r/20240131225125.1370598-1-mcgrof@kernel.org Link: https://lkml.kernel.org/r/20240131225125.1370598-2-mcgrof@kernel.org Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Tested-by: Daniel Gomez <da.gomez@samsung.com> Cc: Darrick J. Wong <djwong@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Hannes Reinecke <hare@suse.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-01-31 22:51:24 +00:00
check_multi_store_advanced(&array);
XArray: add xa_get_order Patch series "Fix read-only THP for non-tmpfs filesystems". As described more verbosely in the [3/3] changelog, we can inadvertently put an order-0 page in the page cache which occupies 512 consecutive entries. Users are running into this if they enable the READ_ONLY_THP_FOR_FS config option; see https://bugzilla.kernel.org/show_bug.cgi?id=206569 and Qian Cai has also reported it here: https://lore.kernel.org/lkml/20200616013309.GB815@lca.pw/ This is a rather intrusive way of fixing the problem, but has the advantage that I've actually been testing it with the THP patches, which means that it sees far more use than it does upstream -- indeed, Song has been entirely unable to reproduce it. It also has the advantage that it removes a few patches from my gargantuan backlog of THP patches. This patch (of 3): This function returns the order of the entry at the index. We need this because there isn't space in the shadow entry to encode its order. [akpm@linux-foundation.org: export xa_get_order to modules] Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-1-willy@infradead.org Link: https://lkml.kernel.org/r/20200903183029.14930-2-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:13 +00:00
check_get_order(&array);
lib/xarray: introduce a new helper xas_get_order It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Added new test for xas_get_order, to make the test work, we have to export xas_get_order with EXPORT_SYMBOL_GPL. Also fix a sparse warning by checking the slot value with xa_entry instead of accessing it directly, as suggested by Matthew Wilcox. [kasong@tencent.com: simplify comment, sparse warning fix, per Matthew Wilcox] Link: https://lkml.kernel.org/r/20240416071722.45997-4-ryncsn@gmail.com Link: https://lkml.kernel.org/r/20240415171857.19244-4-ryncsn@gmail.com Signed-off-by: Kairui Song <kasong@tencent.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-04-15 17:18:55 +00:00
check_xas_get_order(&array);
mm/filemap: optimize filemap folio adding Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranged conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4K pages, in an 8G cgroup, with 16G brd as block device: echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap --rw=randread --time_based \ --ramp_time=30s --runtime=5m --group_reporting Before: bw ( MiB/s): min= 1027, max= 3520, per=100.00%, avg=2445.02, stdev=18.90, samples=8691 iops : min=263001, max=901288, avg=625924.36, stdev=4837.28, samples=8691 After (+7.3%): bw ( MiB/s): min= 493, max= 3947, per=100.00%, avg=2625.56, stdev=25.74, samples=8651 iops : min=126454, max=1010681, avg=672142.61, stdev=6590.48, samples=8651 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap -thp=1 --readonly \ --rw=randread --time_based --ramp_time=30s --runtime=10m \ --group_reporting fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap \ --rw=randread --time_based --runtime=5s --group_reporting Before: bw ( KiB/s): min= 4141, max=14202, per=100.00%, avg=7935.51, stdev=96.85, samples=18976 iops : min= 1029, max= 3548, avg=1979.52, stdev=24.23, samples=18976· READ: bw=4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=64.0KiB (65.5kB), run=14419-14419msec After (+12.5%): bw ( KiB/s): min= 4611, max=15370, per=100.00%, avg=8928.74, stdev=105.17, samples=19146 iops : min= 1151, max= 3842, avg=2231.27, stdev=26.29, samples=19146 READ: bw=4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=64.0KiB (65.5kB), run=14137-14137msec The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read. Link: https://lkml.kernel.org/r/20240415171857.19244-5-ryncsn@gmail.com Signed-off-by: Kairui Song <kasong@tencent.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2024-04-15 17:18:56 +00:00
check_xas_conflict_get_order(&array);
xarray: Track free entries in an XArray Add the optional ability to track which entries in an XArray are free and provide xa_alloc() to replace most of the functionality of the IDR. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-07-04 14:50:12 +00:00
check_xa_alloc();
xarray: Add XArray iterators The xa_for_each iterator allows the user to efficiently walk a range of the array, executing the loop body once for each entry in that range that matches the filter. This commit also includes xa_find() and xa_find_after() which are helper functions for xa_for_each() but may also be useful in their own right. In the xas family of functions, we have xas_for_each(), xas_find(), xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(), xas_next_tagged() and xas_pause(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-14 13:30:11 +00:00
check_find(&array);
shmem: Convert find_swap_entry to XArray This is a 1:1 conversion. The major part of this patch is converting the test framework from userspace to kernel space and mirroring the algorithm now used in find_swap_entry(). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-22 13:36:00 +00:00
check_find_entry(&array);
XArray: Fix xas_pause for large multi-index entries Inspired by the recent Coverity report, I looked for other places where the offset wasn't being converted to an unsigned long before being shifted, and I found one in xas_pause() when the entry being paused is of order >32. Fixes: b803b42823d0 ("xarray: Add XArray iterators") Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: stable@vger.kernel.org
2020-01-31 11:17:09 +00:00
check_pause(&array);
xarray: Move multiorder account test in-kernel Move this test to the in-kernel test suite, and enhance it to test several different orders. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-28 20:13:16 +00:00
check_account(&array);
xarray: Destroy an XArray This function frees all the internal memory allocated to the xarray and reinitialises it to be empty. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-17 13:16:34 +00:00
check_destroy(&array);
xarray: Step through an XArray The xas_next and xas_prev functions move the xas index by one position, and adjust the rest of the iterator state to match it. This is more efficient than calling xas_set() as it keeps the iterator at the leaves of the tree instead of walking the iterator from the root each time. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-01 05:06:52 +00:00
check_move(&array);
xarray: Add xas_create_range This hopefully temporary function is useful for users who have not yet been converted to multi-index entries. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-12-04 05:11:48 +00:00
check_create_range(&array);
xarray: Add range store functionality This version of xa_store_range() really only supports load and store. Our only user only needs basic load and store functionality, so there's no need to do the extra work to support marking and overlapping stores correctly yet. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-08-15 18:13:29 +00:00
check_store_range(&array);
xarray: Add xas_for_each_conflict This iterator iterates over each entry that is stored in the index or indices specified by the xa_state. This is intended for use for a conditional store of a multiindex entry, or to allow entries which are about to be removed from the xarray to be disposed of properly. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-06-02 02:46:02 +00:00
check_store_iter(&array);
XArray: Permit storing 2-byte-aligned pointers On m68k, statically allocated pointers may only be two-byte aligned. This clashes with the XArray's method for tagging internal pointers. Permit storing these pointers in single slots (ie not in multislots). Signed-off-by: Matthew Wilcox <willy@infradead.org>
2018-12-29 04:20:44 +00:00
check_align(&xa0);
XArray: add xas_split In order to use multi-index entries for huge pages in the page cache, we need to be able to split a multi-index entry (eg if a file is truncated in the middle of a huge page entry). This version does not support splitting more than one level of the tree at a time. This is an acceptable limitation for the page cache as we do not expect to support order-12 pages in the near future. [akpm@linux-foundation.org: export xas_split_alloc() to modules] [willy@infradead.org: fix xarray split] Link: https://lkml.kernel.org/r/20200910175450.GV6583@casper.infradead.org [willy@infradead.org: fix xarray] Link: https://lkml.kernel.org/r/20201001233943.GW20115@casper.infradead.org Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Cc: "Kirill A . Shutemov" <kirill@shutemov.name> Cc: Qian Cai <cai@lca.pw> Cc: Song Liu <songliubraving@fb.com> Link: https://lkml.kernel.org/r/20200903183029.14930-3-willy@infradead.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-16 03:05:16 +00:00
check_split(&array);
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
mm: Convert workingset to XArray We construct an XA_STATE and use it to delete the node with xas_store() rather than adding a special function for this unique use case. Includes a test that simulates this usage for the test suite. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-24 19:24:59 +00:00
check_workingset(&array, 0);
check_workingset(&array, 64);
check_workingset(&array, 4096);
xarray: Add XArray load operation The xa_load function brings with it a lot of infrastructure; xa_empty(), xa_is_err(), and large chunks of the XArray advanced API that are used to implement xa_load. As the test-suite demonstrates, it is possible to use the XArray functions on a radix tree. The radix tree functions depend on the GFP flags being stored in the root of the tree, so it's not possible to use the radix tree functions on an XArray. Signed-off-by: Matthew Wilcox <willy@infradead.org>
2017-11-07 19:57:46 +00:00
printk("XArray: %u of %u tests passed\n", tests_passed, tests_run);
return (tests_run == tests_passed) ? 0 : -EINVAL;
}
static void xarray_exit(void)
{
}
module_init(xarray_checks);
module_exit(xarray_exit);
MODULE_AUTHOR("Matthew Wilcox <willy@infradead.org>");
MODULE_LICENSE("GPL");