mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-09-04 09:59:32 +00:00
![Andrey Konovalov](/assets/img/avatar_default.png)
Patch series "kasan: add software tag-based mode for arm64", v13. This patchset adds a new software tag-based mode to KASAN [1]. (Initially this mode was called KHWASAN, but it got renamed, see the naming rationale at the end of this section). The plan is to implement HWASan [2] for the kernel with the incentive, that it's going to have comparable to KASAN performance, but in the same time consume much less memory, trading that off for somewhat imprecise bug detection and being supported only for arm64. The underlying ideas of the approach used by software tag-based KASAN are: 1. By using the Top Byte Ignore (TBI) arm64 CPU feature, we can store pointer tags in the top byte of each kernel pointer. 2. Using shadow memory, we can store memory tags for each chunk of kernel memory. 3. On each memory allocation, we can generate a random tag, embed it into the returned pointer and set the memory tags that correspond to this chunk of memory to the same value. 4. By using compiler instrumentation, before each memory access we can add a check that the pointer tag matches the tag of the memory that is being accessed. 5. On a tag mismatch we report an error. With this patchset the existing KASAN mode gets renamed to generic KASAN, with the word "generic" meaning that the implementation can be supported by any architecture as it is purely software. The new mode this patchset adds is called software tag-based KASAN. The word "tag-based" refers to the fact that this mode uses tags embedded into the top byte of kernel pointers and the TBI arm64 CPU feature that allows to dereference such pointers. The word "software" here means that shadow memory manipulation and tag checking on pointer dereference is done in software. As it is the only tag-based implementation right now, "software tag-based" KASAN is sometimes referred to as simply "tag-based" in this patchset. A potential expansion of this mode is a hardware tag-based mode, which would use hardware memory tagging support (announced by Arm [3]) instead of compiler instrumentation and manual shadow memory manipulation. Same as generic KASAN, software tag-based KASAN is strictly a debugging feature. [1] https://www.kernel.org/doc/html/latest/dev-tools/kasan.html [2] http://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html [3] https://community.arm.com/processors/b/blog/posts/arm-a-profile-architecture-2018-developments-armv85a ====== Rationale On mobile devices generic KASAN's memory usage is significant problem. One of the main reasons to have tag-based KASAN is to be able to perform a similar set of checks as the generic one does, but with lower memory requirements. Comment from Vishwath Mohan <vishwath@google.com>: I don't have data on-hand, but anecdotally both ASAN and KASAN have proven problematic to enable for environments that don't tolerate the increased memory pressure well. This includes (a) Low-memory form factors - Wear, TV, Things, lower-tier phones like Go, (c) Connected components like Pixel's visual core [1]. These are both places I'd love to have a low(er) memory footprint option at my disposal. Comment from Evgenii Stepanov <eugenis@google.com>: Looking at a live Android device under load, slab (according to /proc/meminfo) + kernel stack take 8-10% available RAM (~350MB). KASAN's overhead of 2x - 3x on top of it is not insignificant. Not having this overhead enables near-production use - ex. running KASAN/KHWASAN kernel on a personal, daily-use device to catch bugs that do not reproduce in test configuration. These are the ones that often cost the most engineering time to track down. CPU overhead is bad, but generally tolerable. RAM is critical, in our experience. Once it gets low enough, OOM-killer makes your life miserable. [1] https://www.blog.google/products/pixel/pixel-visual-core-image-processing-and-machine-learning-pixel-2/ ====== Technical details Software tag-based KASAN mode is implemented in a very similar way to the generic one. This patchset essentially does the following: 1. TCR_TBI1 is set to enable Top Byte Ignore. 2. Shadow memory is used (with a different scale, 1:16, so each shadow byte corresponds to 16 bytes of kernel memory) to store memory tags. 3. All slab objects are aligned to shadow scale, which is 16 bytes. 4. All pointers returned from the slab allocator are tagged with a random tag and the corresponding shadow memory is poisoned with the same value. 5. Compiler instrumentation is used to insert tag checks. Either by calling callbacks or by inlining them (CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE flags are reused). 6. When a tag mismatch is detected in callback instrumentation mode KASAN simply prints a bug report. In case of inline instrumentation, clang inserts a brk instruction, and KASAN has it's own brk handler, which reports the bug. 7. The memory in between slab objects is marked with a reserved tag, and acts as a redzone. 8. When a slab object is freed it's marked with a reserved tag. Bug detection is imprecise for two reasons: 1. We won't catch some small out-of-bounds accesses, that fall into the same shadow cell, as the last byte of a slab object. 2. We only have 1 byte to store tags, which means we have a 1/256 probability of a tag match for an incorrect access (actually even slightly less due to reserved tag values). Despite that there's a particular type of bugs that tag-based KASAN can detect compared to generic KASAN: use-after-free after the object has been allocated by someone else. ====== Testing Some kernel developers voiced a concern that changing the top byte of kernel pointers may lead to subtle bugs that are difficult to discover. To address this concern deliberate testing has been performed. It doesn't seem feasible to do some kind of static checking to find potential issues with pointer tagging, so a dynamic approach was taken. All pointer comparisons/subtractions have been instrumented in an LLVM compiler pass and a kernel module that would print a bug report whenever two pointers with different tags are being compared/subtracted (ignoring comparisons with NULL pointers and with pointers obtained by casting an error code to a pointer type) has been used. Then the kernel has been booted in QEMU and on an Odroid C2 board and syzkaller has been run. This yielded the following results. The two places that look interesting are: is_vmalloc_addr in include/linux/mm.h is_kernel_rodata in mm/util.c Here we compare a pointer with some fixed untagged values to make sure that the pointer lies in a particular part of the kernel address space. Since tag-based KASAN doesn't add tags to pointers that belong to rodata or vmalloc regions, this should work as is. To make sure debug checks to those two functions that check that the result doesn't change whether we operate on pointers with or without untagging has been added. A few other cases that don't look that interesting: Comparing pointers to achieve unique sorting order of pointee objects (e.g. sorting locks addresses before performing a double lock): tty_ldisc_lock_pair_timeout in drivers/tty/tty_ldisc.c pipe_double_lock in fs/pipe.c unix_state_double_lock in net/unix/af_unix.c lock_two_nondirectories in fs/inode.c mutex_lock_double in kernel/events/core.c ep_cmp_ffd in fs/eventpoll.c fsnotify_compare_groups fs/notify/mark.c Nothing needs to be done here, since the tags embedded into pointers don't change, so the sorting order would still be unique. Checks that a pointer belongs to some particular allocation: is_sibling_entry in lib/radix-tree.c object_is_on_stack in include/linux/sched/task_stack.h Nothing needs to be done here either, since two pointers can only belong to the same allocation if they have the same tag. Overall, since the kernel boots and works, there are no critical bugs. As for the rest, the traditional kernel testing way (use until fails) is the only one that looks feasible. Another point here is that tag-based KASAN is available under a separate config option that needs to be deliberately enabled. Even though it might be used in a "near-production" environment to find bugs that are not found during fuzzing or running tests, it is still a debug tool. ====== Benchmarks The following numbers were collected on Odroid C2 board. Both generic and tag-based KASAN were used in inline instrumentation mode. Boot time [1]: * ~1.7 sec for clean kernel * ~5.0 sec for generic KASAN * ~5.0 sec for tag-based KASAN Network performance [2]: * 8.33 Gbits/sec for clean kernel * 3.17 Gbits/sec for generic KASAN * 2.85 Gbits/sec for tag-based KASAN Slab memory usage after boot [3]: * ~40 kb for clean kernel * ~105 kb (~260% overhead) for generic KASAN * ~47 kb (~20% overhead) for tag-based KASAN KASAN memory overhead consists of three main parts: 1. Increased slab memory usage due to redzones. 2. Shadow memory (the whole reserved once during boot). 3. Quaratine (grows gradually until some preset limit; the more the limit, the more the chance to detect a use-after-free). Comparing tag-based vs generic KASAN for each of these points: 1. 20% vs 260% overhead. 2. 1/16th vs 1/8th of physical memory. 3. Tag-based KASAN doesn't require quarantine. [1] Time before the ext4 driver is initialized. [2] Measured as `iperf -s & iperf -c 127.0.0.1 -t 30`. [3] Measured as `cat /proc/meminfo | grep Slab`. ====== Some notes A few notes: 1. The patchset can be found here: https://github.com/xairy/kasan-prototype/tree/khwasan 2. Building requires a recent Clang version (7.0.0 or later). 3. Stack instrumentation is not supported yet and will be added later. This patch (of 25): Tag-based KASAN changes the value of the top byte of pointers returned from the kernel allocation functions (such as kmalloc). This patch updates KASAN hooks signatures and their usage in SLAB and SLUB code to reflect that. Link: http://lkml.kernel.org/r/aec2b5e3973781ff8a6bb6760f8543643202c451.1544099024.git.andreyknvl@google.com Signed-off-by: Andrey Konovalov <andreyknvl@google.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Cc: Christoph Lameter <cl@linux.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
531 lines
15 KiB
C
531 lines
15 KiB
C
/* SPDX-License-Identifier: GPL-2.0 */
|
|
#ifndef MM_SLAB_H
|
|
#define MM_SLAB_H
|
|
/*
|
|
* Internal slab definitions
|
|
*/
|
|
|
|
#ifdef CONFIG_SLOB
|
|
/*
|
|
* Common fields provided in kmem_cache by all slab allocators
|
|
* This struct is either used directly by the allocator (SLOB)
|
|
* or the allocator must include definitions for all fields
|
|
* provided in kmem_cache_common in their definition of kmem_cache.
|
|
*
|
|
* Once we can do anonymous structs (C11 standard) we could put a
|
|
* anonymous struct definition in these allocators so that the
|
|
* separate allocations in the kmem_cache structure of SLAB and
|
|
* SLUB is no longer needed.
|
|
*/
|
|
struct kmem_cache {
|
|
unsigned int object_size;/* The original size of the object */
|
|
unsigned int size; /* The aligned/padded/added on size */
|
|
unsigned int align; /* Alignment as calculated */
|
|
slab_flags_t flags; /* Active flags on the slab */
|
|
unsigned int useroffset;/* Usercopy region offset */
|
|
unsigned int usersize; /* Usercopy region size */
|
|
const char *name; /* Slab name for sysfs */
|
|
int refcount; /* Use counter */
|
|
void (*ctor)(void *); /* Called on object slot creation */
|
|
struct list_head list; /* List of all slab caches on the system */
|
|
};
|
|
|
|
#endif /* CONFIG_SLOB */
|
|
|
|
#ifdef CONFIG_SLAB
|
|
#include <linux/slab_def.h>
|
|
#endif
|
|
|
|
#ifdef CONFIG_SLUB
|
|
#include <linux/slub_def.h>
|
|
#endif
|
|
|
|
#include <linux/memcontrol.h>
|
|
#include <linux/fault-inject.h>
|
|
#include <linux/kasan.h>
|
|
#include <linux/kmemleak.h>
|
|
#include <linux/random.h>
|
|
#include <linux/sched/mm.h>
|
|
|
|
/*
|
|
* State of the slab allocator.
|
|
*
|
|
* This is used to describe the states of the allocator during bootup.
|
|
* Allocators use this to gradually bootstrap themselves. Most allocators
|
|
* have the problem that the structures used for managing slab caches are
|
|
* allocated from slab caches themselves.
|
|
*/
|
|
enum slab_state {
|
|
DOWN, /* No slab functionality yet */
|
|
PARTIAL, /* SLUB: kmem_cache_node available */
|
|
PARTIAL_NODE, /* SLAB: kmalloc size for node struct available */
|
|
UP, /* Slab caches usable but not all extras yet */
|
|
FULL /* Everything is working */
|
|
};
|
|
|
|
extern enum slab_state slab_state;
|
|
|
|
/* The slab cache mutex protects the management structures during changes */
|
|
extern struct mutex slab_mutex;
|
|
|
|
/* The list of all slab caches on the system */
|
|
extern struct list_head slab_caches;
|
|
|
|
/* The slab cache that manages slab cache information */
|
|
extern struct kmem_cache *kmem_cache;
|
|
|
|
/* A table of kmalloc cache names and sizes */
|
|
extern const struct kmalloc_info_struct {
|
|
const char *name;
|
|
unsigned int size;
|
|
} kmalloc_info[];
|
|
|
|
#ifndef CONFIG_SLOB
|
|
/* Kmalloc array related functions */
|
|
void setup_kmalloc_cache_index_table(void);
|
|
void create_kmalloc_caches(slab_flags_t);
|
|
|
|
/* Find the kmalloc slab corresponding for a certain size */
|
|
struct kmem_cache *kmalloc_slab(size_t, gfp_t);
|
|
#endif
|
|
|
|
|
|
/* Functions provided by the slab allocators */
|
|
int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
|
|
|
|
struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
|
|
slab_flags_t flags, unsigned int useroffset,
|
|
unsigned int usersize);
|
|
extern void create_boot_cache(struct kmem_cache *, const char *name,
|
|
unsigned int size, slab_flags_t flags,
|
|
unsigned int useroffset, unsigned int usersize);
|
|
|
|
int slab_unmergeable(struct kmem_cache *s);
|
|
struct kmem_cache *find_mergeable(unsigned size, unsigned align,
|
|
slab_flags_t flags, const char *name, void (*ctor)(void *));
|
|
#ifndef CONFIG_SLOB
|
|
struct kmem_cache *
|
|
__kmem_cache_alias(const char *name, unsigned int size, unsigned int align,
|
|
slab_flags_t flags, void (*ctor)(void *));
|
|
|
|
slab_flags_t kmem_cache_flags(unsigned int object_size,
|
|
slab_flags_t flags, const char *name,
|
|
void (*ctor)(void *));
|
|
#else
|
|
static inline struct kmem_cache *
|
|
__kmem_cache_alias(const char *name, unsigned int size, unsigned int align,
|
|
slab_flags_t flags, void (*ctor)(void *))
|
|
{ return NULL; }
|
|
|
|
static inline slab_flags_t kmem_cache_flags(unsigned int object_size,
|
|
slab_flags_t flags, const char *name,
|
|
void (*ctor)(void *))
|
|
{
|
|
return flags;
|
|
}
|
|
#endif
|
|
|
|
|
|
/* Legal flag mask for kmem_cache_create(), for various configurations */
|
|
#define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | SLAB_PANIC | \
|
|
SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS )
|
|
|
|
#if defined(CONFIG_DEBUG_SLAB)
|
|
#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)
|
|
#elif defined(CONFIG_SLUB_DEBUG)
|
|
#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
|
|
SLAB_TRACE | SLAB_CONSISTENCY_CHECKS)
|
|
#else
|
|
#define SLAB_DEBUG_FLAGS (0)
|
|
#endif
|
|
|
|
#if defined(CONFIG_SLAB)
|
|
#define SLAB_CACHE_FLAGS (SLAB_MEM_SPREAD | SLAB_NOLEAKTRACE | \
|
|
SLAB_RECLAIM_ACCOUNT | SLAB_TEMPORARY | \
|
|
SLAB_ACCOUNT)
|
|
#elif defined(CONFIG_SLUB)
|
|
#define SLAB_CACHE_FLAGS (SLAB_NOLEAKTRACE | SLAB_RECLAIM_ACCOUNT | \
|
|
SLAB_TEMPORARY | SLAB_ACCOUNT)
|
|
#else
|
|
#define SLAB_CACHE_FLAGS (0)
|
|
#endif
|
|
|
|
/* Common flags available with current configuration */
|
|
#define CACHE_CREATE_MASK (SLAB_CORE_FLAGS | SLAB_DEBUG_FLAGS | SLAB_CACHE_FLAGS)
|
|
|
|
/* Common flags permitted for kmem_cache_create */
|
|
#define SLAB_FLAGS_PERMITTED (SLAB_CORE_FLAGS | \
|
|
SLAB_RED_ZONE | \
|
|
SLAB_POISON | \
|
|
SLAB_STORE_USER | \
|
|
SLAB_TRACE | \
|
|
SLAB_CONSISTENCY_CHECKS | \
|
|
SLAB_MEM_SPREAD | \
|
|
SLAB_NOLEAKTRACE | \
|
|
SLAB_RECLAIM_ACCOUNT | \
|
|
SLAB_TEMPORARY | \
|
|
SLAB_ACCOUNT)
|
|
|
|
bool __kmem_cache_empty(struct kmem_cache *);
|
|
int __kmem_cache_shutdown(struct kmem_cache *);
|
|
void __kmem_cache_release(struct kmem_cache *);
|
|
int __kmem_cache_shrink(struct kmem_cache *);
|
|
void __kmemcg_cache_deactivate(struct kmem_cache *s);
|
|
void slab_kmem_cache_release(struct kmem_cache *);
|
|
|
|
struct seq_file;
|
|
struct file;
|
|
|
|
struct slabinfo {
|
|
unsigned long active_objs;
|
|
unsigned long num_objs;
|
|
unsigned long active_slabs;
|
|
unsigned long num_slabs;
|
|
unsigned long shared_avail;
|
|
unsigned int limit;
|
|
unsigned int batchcount;
|
|
unsigned int shared;
|
|
unsigned int objects_per_slab;
|
|
unsigned int cache_order;
|
|
};
|
|
|
|
void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo);
|
|
void slabinfo_show_stats(struct seq_file *m, struct kmem_cache *s);
|
|
ssize_t slabinfo_write(struct file *file, const char __user *buffer,
|
|
size_t count, loff_t *ppos);
|
|
|
|
/*
|
|
* Generic implementation of bulk operations
|
|
* These are useful for situations in which the allocator cannot
|
|
* perform optimizations. In that case segments of the object listed
|
|
* may be allocated or freed using these operations.
|
|
*/
|
|
void __kmem_cache_free_bulk(struct kmem_cache *, size_t, void **);
|
|
int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **);
|
|
|
|
#ifdef CONFIG_MEMCG_KMEM
|
|
|
|
/* List of all root caches. */
|
|
extern struct list_head slab_root_caches;
|
|
#define root_caches_node memcg_params.__root_caches_node
|
|
|
|
/*
|
|
* Iterate over all memcg caches of the given root cache. The caller must hold
|
|
* slab_mutex.
|
|
*/
|
|
#define for_each_memcg_cache(iter, root) \
|
|
list_for_each_entry(iter, &(root)->memcg_params.children, \
|
|
memcg_params.children_node)
|
|
|
|
static inline bool is_root_cache(struct kmem_cache *s)
|
|
{
|
|
return !s->memcg_params.root_cache;
|
|
}
|
|
|
|
static inline bool slab_equal_or_root(struct kmem_cache *s,
|
|
struct kmem_cache *p)
|
|
{
|
|
return p == s || p == s->memcg_params.root_cache;
|
|
}
|
|
|
|
/*
|
|
* We use suffixes to the name in memcg because we can't have caches
|
|
* created in the system with the same name. But when we print them
|
|
* locally, better refer to them with the base name
|
|
*/
|
|
static inline const char *cache_name(struct kmem_cache *s)
|
|
{
|
|
if (!is_root_cache(s))
|
|
s = s->memcg_params.root_cache;
|
|
return s->name;
|
|
}
|
|
|
|
/*
|
|
* Note, we protect with RCU only the memcg_caches array, not per-memcg caches.
|
|
* That said the caller must assure the memcg's cache won't go away by either
|
|
* taking a css reference to the owner cgroup, or holding the slab_mutex.
|
|
*/
|
|
static inline struct kmem_cache *
|
|
cache_from_memcg_idx(struct kmem_cache *s, int idx)
|
|
{
|
|
struct kmem_cache *cachep;
|
|
struct memcg_cache_array *arr;
|
|
|
|
rcu_read_lock();
|
|
arr = rcu_dereference(s->memcg_params.memcg_caches);
|
|
|
|
/*
|
|
* Make sure we will access the up-to-date value. The code updating
|
|
* memcg_caches issues a write barrier to match this (see
|
|
* memcg_create_kmem_cache()).
|
|
*/
|
|
cachep = READ_ONCE(arr->entries[idx]);
|
|
rcu_read_unlock();
|
|
|
|
return cachep;
|
|
}
|
|
|
|
static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s)
|
|
{
|
|
if (is_root_cache(s))
|
|
return s;
|
|
return s->memcg_params.root_cache;
|
|
}
|
|
|
|
static __always_inline int memcg_charge_slab(struct page *page,
|
|
gfp_t gfp, int order,
|
|
struct kmem_cache *s)
|
|
{
|
|
if (!memcg_kmem_enabled())
|
|
return 0;
|
|
if (is_root_cache(s))
|
|
return 0;
|
|
return memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg);
|
|
}
|
|
|
|
static __always_inline void memcg_uncharge_slab(struct page *page, int order,
|
|
struct kmem_cache *s)
|
|
{
|
|
if (!memcg_kmem_enabled())
|
|
return;
|
|
memcg_kmem_uncharge(page, order);
|
|
}
|
|
|
|
extern void slab_init_memcg_params(struct kmem_cache *);
|
|
extern void memcg_link_cache(struct kmem_cache *s);
|
|
extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,
|
|
void (*deact_fn)(struct kmem_cache *));
|
|
|
|
#else /* CONFIG_MEMCG_KMEM */
|
|
|
|
/* If !memcg, all caches are root. */
|
|
#define slab_root_caches slab_caches
|
|
#define root_caches_node list
|
|
|
|
#define for_each_memcg_cache(iter, root) \
|
|
for ((void)(iter), (void)(root); 0; )
|
|
|
|
static inline bool is_root_cache(struct kmem_cache *s)
|
|
{
|
|
return true;
|
|
}
|
|
|
|
static inline bool slab_equal_or_root(struct kmem_cache *s,
|
|
struct kmem_cache *p)
|
|
{
|
|
return true;
|
|
}
|
|
|
|
static inline const char *cache_name(struct kmem_cache *s)
|
|
{
|
|
return s->name;
|
|
}
|
|
|
|
static inline struct kmem_cache *
|
|
cache_from_memcg_idx(struct kmem_cache *s, int idx)
|
|
{
|
|
return NULL;
|
|
}
|
|
|
|
static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s)
|
|
{
|
|
return s;
|
|
}
|
|
|
|
static inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order,
|
|
struct kmem_cache *s)
|
|
{
|
|
return 0;
|
|
}
|
|
|
|
static inline void memcg_uncharge_slab(struct page *page, int order,
|
|
struct kmem_cache *s)
|
|
{
|
|
}
|
|
|
|
static inline void slab_init_memcg_params(struct kmem_cache *s)
|
|
{
|
|
}
|
|
|
|
static inline void memcg_link_cache(struct kmem_cache *s)
|
|
{
|
|
}
|
|
|
|
#endif /* CONFIG_MEMCG_KMEM */
|
|
|
|
static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)
|
|
{
|
|
struct kmem_cache *cachep;
|
|
struct page *page;
|
|
|
|
/*
|
|
* When kmemcg is not being used, both assignments should return the
|
|
* same value. but we don't want to pay the assignment price in that
|
|
* case. If it is not compiled in, the compiler should be smart enough
|
|
* to not do even the assignment. In that case, slab_equal_or_root
|
|
* will also be a constant.
|
|
*/
|
|
if (!memcg_kmem_enabled() &&
|
|
!unlikely(s->flags & SLAB_CONSISTENCY_CHECKS))
|
|
return s;
|
|
|
|
page = virt_to_head_page(x);
|
|
cachep = page->slab_cache;
|
|
if (slab_equal_or_root(cachep, s))
|
|
return cachep;
|
|
|
|
pr_err("%s: Wrong slab cache. %s but object is from %s\n",
|
|
__func__, s->name, cachep->name);
|
|
WARN_ON_ONCE(1);
|
|
return s;
|
|
}
|
|
|
|
static inline size_t slab_ksize(const struct kmem_cache *s)
|
|
{
|
|
#ifndef CONFIG_SLUB
|
|
return s->object_size;
|
|
|
|
#else /* CONFIG_SLUB */
|
|
# ifdef CONFIG_SLUB_DEBUG
|
|
/*
|
|
* Debugging requires use of the padding between object
|
|
* and whatever may come after it.
|
|
*/
|
|
if (s->flags & (SLAB_RED_ZONE | SLAB_POISON))
|
|
return s->object_size;
|
|
# endif
|
|
if (s->flags & SLAB_KASAN)
|
|
return s->object_size;
|
|
/*
|
|
* If we have the need to store the freelist pointer
|
|
* back there or track user information then we can
|
|
* only use the space before that information.
|
|
*/
|
|
if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER))
|
|
return s->inuse;
|
|
/*
|
|
* Else we can use all the padding etc for the allocation
|
|
*/
|
|
return s->size;
|
|
#endif
|
|
}
|
|
|
|
static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s,
|
|
gfp_t flags)
|
|
{
|
|
flags &= gfp_allowed_mask;
|
|
|
|
fs_reclaim_acquire(flags);
|
|
fs_reclaim_release(flags);
|
|
|
|
might_sleep_if(gfpflags_allow_blocking(flags));
|
|
|
|
if (should_failslab(s, flags))
|
|
return NULL;
|
|
|
|
if (memcg_kmem_enabled() &&
|
|
((flags & __GFP_ACCOUNT) || (s->flags & SLAB_ACCOUNT)))
|
|
return memcg_kmem_get_cache(s);
|
|
|
|
return s;
|
|
}
|
|
|
|
static inline void slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,
|
|
size_t size, void **p)
|
|
{
|
|
size_t i;
|
|
|
|
flags &= gfp_allowed_mask;
|
|
for (i = 0; i < size; i++) {
|
|
void *object = p[i];
|
|
|
|
kmemleak_alloc_recursive(object, s->object_size, 1,
|
|
s->flags, flags);
|
|
p[i] = kasan_slab_alloc(s, object, flags);
|
|
}
|
|
|
|
if (memcg_kmem_enabled())
|
|
memcg_kmem_put_cache(s);
|
|
}
|
|
|
|
#ifndef CONFIG_SLOB
|
|
/*
|
|
* The slab lists for all objects.
|
|
*/
|
|
struct kmem_cache_node {
|
|
spinlock_t list_lock;
|
|
|
|
#ifdef CONFIG_SLAB
|
|
struct list_head slabs_partial; /* partial list first, better asm code */
|
|
struct list_head slabs_full;
|
|
struct list_head slabs_free;
|
|
unsigned long total_slabs; /* length of all slab lists */
|
|
unsigned long free_slabs; /* length of free slab list only */
|
|
unsigned long free_objects;
|
|
unsigned int free_limit;
|
|
unsigned int colour_next; /* Per-node cache coloring */
|
|
struct array_cache *shared; /* shared per node */
|
|
struct alien_cache **alien; /* on other nodes */
|
|
unsigned long next_reap; /* updated without locking */
|
|
int free_touched; /* updated without locking */
|
|
#endif
|
|
|
|
#ifdef CONFIG_SLUB
|
|
unsigned long nr_partial;
|
|
struct list_head partial;
|
|
#ifdef CONFIG_SLUB_DEBUG
|
|
atomic_long_t nr_slabs;
|
|
atomic_long_t total_objects;
|
|
struct list_head full;
|
|
#endif
|
|
#endif
|
|
|
|
};
|
|
|
|
static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
|
|
{
|
|
return s->node[node];
|
|
}
|
|
|
|
/*
|
|
* Iterator over all nodes. The body will be executed for each node that has
|
|
* a kmem_cache_node structure allocated (which is true for all online nodes)
|
|
*/
|
|
#define for_each_kmem_cache_node(__s, __node, __n) \
|
|
for (__node = 0; __node < nr_node_ids; __node++) \
|
|
if ((__n = get_node(__s, __node)))
|
|
|
|
#endif
|
|
|
|
void *slab_start(struct seq_file *m, loff_t *pos);
|
|
void *slab_next(struct seq_file *m, void *p, loff_t *pos);
|
|
void slab_stop(struct seq_file *m, void *p);
|
|
void *memcg_slab_start(struct seq_file *m, loff_t *pos);
|
|
void *memcg_slab_next(struct seq_file *m, void *p, loff_t *pos);
|
|
void memcg_slab_stop(struct seq_file *m, void *p);
|
|
int memcg_slab_show(struct seq_file *m, void *p);
|
|
|
|
#if defined(CONFIG_SLAB) || defined(CONFIG_SLUB_DEBUG)
|
|
void dump_unreclaimable_slab(void);
|
|
#else
|
|
static inline void dump_unreclaimable_slab(void)
|
|
{
|
|
}
|
|
#endif
|
|
|
|
void ___cache_free(struct kmem_cache *cache, void *x, unsigned long addr);
|
|
|
|
#ifdef CONFIG_SLAB_FREELIST_RANDOM
|
|
int cache_random_seq_create(struct kmem_cache *cachep, unsigned int count,
|
|
gfp_t gfp);
|
|
void cache_random_seq_destroy(struct kmem_cache *cachep);
|
|
#else
|
|
static inline int cache_random_seq_create(struct kmem_cache *cachep,
|
|
unsigned int count, gfp_t gfp)
|
|
{
|
|
return 0;
|
|
}
|
|
static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { }
|
|
#endif /* CONFIG_SLAB_FREELIST_RANDOM */
|
|
|
|
#endif /* MM_SLAB_H */
|