mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-09-23 19:11:51 +00:00
54a611b605
Patch series "Introducing the Maple Tree" The maple tree is an RCU-safe range based B-tree designed to use modern processor cache efficiently. There are a number of places in the kernel that a non-overlapping range-based tree would be beneficial, especially one with a simple interface. If you use an rbtree with other data structures to improve performance or an interval tree to track non-overlapping ranges, then this is for you. The tree has a branching factor of 10 for non-leaf nodes and 16 for leaf nodes. With the increased branching factor, it is significantly shorter than the rbtree so it has fewer cache misses. The removal of the linked list between subsequent entries also reduces the cache misses and the need to pull in the previous and next VMA during many tree alterations. The first user that is covered in this patch set is the vm_area_struct, where three data structures are replaced by the maple tree: the augmented rbtree, the vma cache, and the linked list of VMAs in the mm_struct. The long term goal is to reduce or remove the mmap_lock contention. The plan is to get to the point where we use the maple tree in RCU mode. Readers will not block for writers. A single write operation will be allowed at a time. A reader re-walks if stale data is encountered. VMAs would be RCU enabled and this mode would be entered once multiple tasks are using the mm_struct. Davidlor said : Yes I like the maple tree, and at this stage I don't think we can ask for : more from this series wrt the MM - albeit there seems to still be some : folks reporting breakage. Fundamentally I see Liam's work to (re)move : complexity out of the MM (not to say that the actual maple tree is not : complex) by consolidating the three complimentary data structures very : much worth it considering performance does not take a hit. This was very : much a turn off with the range locking approach, which worst case scenario : incurred in prohibitive overhead. Also as Liam and Matthew have : mentioned, RCU opens up a lot of nice performance opportunities, and in : addition academia[1] has shown outstanding scalability of address spaces : with the foundation of replacing the locked rbtree with RCU aware trees. A similar work has been discovered in the academic press https://pdos.csail.mit.edu/papers/rcuvm:asplos12.pdf Sheer coincidence. We designed our tree with the intention of solving the hardest problem first. Upon settling on a b-tree variant and a rough outline, we researched ranged based b-trees and RCU b-trees and did find that article. So it was nice to find reassurances that we were on the right path, but our design choice of using ranges made that paper unusable for us. This patch (of 70): The maple tree is an RCU-safe range based B-tree designed to use modern processor cache efficiently. There are a number of places in the kernel that a non-overlapping range-based tree would be beneficial, especially one with a simple interface. If you use an rbtree with other data structures to improve performance or an interval tree to track non-overlapping ranges, then this is for you. The tree has a branching factor of 10 for non-leaf nodes and 16 for leaf nodes. With the increased branching factor, it is significantly shorter than the rbtree so it has fewer cache misses. The removal of the linked list between subsequent entries also reduces the cache misses and the need to pull in the previous and next VMA during many tree alterations. The first user that is covered in this patch set is the vm_area_struct, where three data structures are replaced by the maple tree: the augmented rbtree, the vma cache, and the linked list of VMAs in the mm_struct. The long term goal is to reduce or remove the mmap_lock contention. The plan is to get to the point where we use the maple tree in RCU mode. Readers will not block for writers. A single write operation will be allowed at a time. A reader re-walks if stale data is encountered. VMAs would be RCU enabled and this mode would be entered once multiple tasks are using the mm_struct. There is additional BUG_ON() calls added within the tree, most of which are in debug code. These will be replaced with a WARN_ON() call in the future. There is also additional BUG_ON() calls within the code which will also be reduced in number at a later date. These exist to catch things such as out-of-range accesses which would crash anyways. Link: https://lkml.kernel.org/r/20220906194824.2110408-1-Liam.Howlett@oracle.com Link: https://lkml.kernel.org/r/20220906194824.2110408-2-Liam.Howlett@oracle.com Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Tested-by: David Howells <dhowells@redhat.com> Tested-by: Sven Schnelle <svens@linux.ibm.com> Tested-by: Yu Zhao <yuzhao@google.com> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: David Hildenbrand <david@redhat.com> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: SeongJae Park <sj@kernel.org> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|---|---|---|
| .. | ||
| 842 | ||
| crypto | ||
| dim | ||
| fonts | ||
| kunit | ||
| livepatch | ||
| lz4 | ||
| lzo | ||
| math | ||
| mpi | ||
| pldmfw | ||
| raid6 | ||
| reed_solomon | ||
| test_fortify | ||
| vdso | ||
| xz | ||
| zlib_deflate | ||
| zlib_dfltcc | ||
| zlib_inflate | ||
| zstd | ||
| .gitignore | ||
| argv_split.c | ||
| ashldi3.c | ||
| ashrdi3.c | ||
| asn1_decoder.c | ||
| asn1_encoder.c | ||
| assoc_array.c | ||
| atomic64.c | ||
| atomic64_test.c | ||
| audit.c | ||
| base64.c | ||
| bcd.c | ||
| bch.c | ||
| bitfield_kunit.c | ||
| bitmap.c | ||
| bitrev.c | ||
| bootconfig-data.S | ||
| bootconfig.c | ||
| bsearch.c | ||
| btree.c | ||
| bucket_locks.c | ||
| bug.c | ||
| build_OID_registry | ||
| buildid.c | ||
| bust_spinlocks.c | ||
| check_signature.c | ||
| checksum.c | ||
| clz_ctz.c | ||
| clz_tab.c | ||
| cmdline.c | ||
| cmdline_kunit.c | ||
| cmpdi2.c | ||
| compat_audit.c | ||
| cpu_rmap.c | ||
| cpumask.c | ||
| cpumask_kunit.c | ||
| crc-ccitt.c | ||
| crc-itu-t.c | ||
| crc-t10dif.c | ||
| crc4.c | ||
| crc7.c | ||
| crc8.c | ||
| crc16.c | ||
| crc32.c | ||
| crc32defs.h | ||
| crc32test.c | ||
| crc64-rocksoft.c | ||
| crc64.c | ||
| ctype.c | ||
| debug_info.c | ||
| debug_locks.c | ||
| debugobjects.c | ||
| dec_and_lock.c | ||
| decompress.c | ||
| decompress_bunzip2.c | ||
| decompress_inflate.c | ||
| decompress_unlz4.c | ||
| decompress_unlzma.c | ||
| decompress_unlzo.c | ||
| decompress_unxz.c | ||
| decompress_unzstd.c | ||
| devmem_is_allowed.c | ||
| devres.c | ||
| digsig.c | ||
| dump_stack.c | ||
| dynamic_debug.c | ||
| dynamic_queue_limits.c | ||
| earlycpio.c | ||
| errname.c | ||
| error-inject.c | ||
| errseq.c | ||
| extable.c | ||
| fault-inject-usercopy.c | ||
| fault-inject.c | ||
| fdt.c | ||
| fdt_addresses.c | ||
| fdt_empty_tree.c | ||
| fdt_ro.c | ||
| fdt_rw.c | ||
| fdt_strerror.c | ||
| fdt_sw.c | ||
| fdt_wip.c | ||
| find_bit.c | ||
| find_bit_benchmark.c | ||
| flex_proportions.c | ||
| gen_crc32table.c | ||
| gen_crc64table.c | ||
| genalloc.c | ||
| generic-radix-tree.c | ||
| glob.c | ||
| globtest.c | ||
| hexdump.c | ||
| hweight.c | ||
| idr.c | ||
| inflate.c | ||
| interval_tree.c | ||
| interval_tree_test.c | ||
| iomap.c | ||
| iomap_copy.c | ||
| iommu-helper.c | ||
| iov_iter.c | ||
| irq_poll.c | ||
| irq_regs.c | ||
| is_single_threaded.c | ||
| kasprintf.c | ||
| Kconfig | ||
| Kconfig.debug | ||
| Kconfig.kasan | ||
| Kconfig.kcsan | ||
| Kconfig.kfence | ||
| Kconfig.kgdb | ||
| Kconfig.ubsan | ||
| kfifo.c | ||
| klist.c | ||
| kobject.c | ||
| kobject_uevent.c | ||
| kstrtox.c | ||
| kstrtox.h | ||
| libcrc32c.c | ||
| linear_ranges.c | ||
| list-test.c | ||
| list_debug.c | ||
| list_sort.c | ||
| llist.c | ||
| locking-selftest-hardirq.h | ||
| locking-selftest-mutex.h | ||
| locking-selftest-rlock-hardirq.h | ||
| locking-selftest-rlock-softirq.h | ||
| locking-selftest-rlock.h | ||
| locking-selftest-rsem.h | ||
| locking-selftest-rtmutex.h | ||
| locking-selftest-softirq.h | ||
| locking-selftest-spin-hardirq.h | ||
| locking-selftest-spin-softirq.h | ||
| locking-selftest-spin.h | ||
| locking-selftest-wlock-hardirq.h | ||
| locking-selftest-wlock-softirq.h | ||
| locking-selftest-wlock.h | ||
| locking-selftest-wsem.h | ||
| locking-selftest.c | ||
| lockref.c | ||
| logic_iomem.c | ||
| logic_pio.c | ||
| lru_cache.c | ||
| lshrdi3.c | ||
| Makefile | ||
| maple_tree.c | ||
| memcat_p.c | ||
| memcpy_kunit.c | ||
| memneq.c | ||
| memory-notifier-error-inject.c | ||
| memregion.c | ||
| memweight.c | ||
| muldi3.c | ||
| net_utils.c | ||
| netdev-notifier-error-inject.c | ||
| nlattr.c | ||
| nmi_backtrace.c | ||
| notifier-error-inject.c | ||
| notifier-error-inject.h | ||
| objagg.c | ||
| of-reconfig-notifier-error-inject.c | ||
| oid_registry.c | ||
| once.c | ||
| overflow_kunit.c | ||
| packing.c | ||
| parman.c | ||
| parser.c | ||
| pci_iomap.c | ||
| percpu-refcount.c | ||
| percpu_counter.c | ||
| percpu_test.c | ||
| plist.c | ||
| pm-notifier-error-inject.c | ||
| polynomial.c | ||
| radix-tree.c | ||
| random32.c | ||
| ratelimit.c | ||
| rbtree.c | ||
| rbtree_test.c | ||
| ref_tracker.c | ||
| refcount.c | ||
| rhashtable.c | ||
| sbitmap.c | ||
| scatterlist.c | ||
| seq_buf.c | ||
| sg_pool.c | ||
| sg_split.c | ||
| show_mem.c | ||
| siphash.c | ||
| slub_kunit.c | ||
| smp_processor_id.c | ||
| sort.c | ||
| stackdepot.c | ||
| stackinit_kunit.c | ||
| stmp_device.c | ||
| string.c | ||
| string_helpers.c | ||
| strncpy_from_user.c | ||
| strnlen_user.c | ||
| syscall.c | ||
| test-kstrtox.c | ||
| test-string_helpers.c | ||
| test_bitmap.c | ||
| test_bitops.c | ||
| test_bits.c | ||
| test_blackhole_dev.c | ||
| test_bpf.c | ||
| test_debug_virtual.c | ||
| test_firmware.c | ||
| test_fprobe.c | ||
| test_fpu.c | ||
| test_free_pages.c | ||
| test_hash.c | ||
| test_hexdump.c | ||
| test_hmm.c | ||
| test_hmm_uapi.h | ||
| test_ida.c | ||
| test_kasan.c | ||
| test_kasan_module.c | ||
| test_kmod.c | ||
| test_kprobes.c | ||
| test_linear_ranges.c | ||
| test_list_sort.c | ||
| test_lockup.c | ||
| test_memcat_p.c | ||
| test_meminit.c | ||
| test_min_heap.c | ||
| test_module.c | ||
| test_objagg.c | ||
| test_parman.c | ||
| test_printf.c | ||
| test_ref_tracker.c | ||
| test_rhashtable.c | ||
| test_scanf.c | ||
| test_siphash.c | ||
| test_sort.c | ||
| test_static_key_base.c | ||
| test_static_keys.c | ||
| test_string.c | ||
| test_strscpy.c | ||
| test_sysctl.c | ||
| test_ubsan.c | ||
| test_user_copy.c | ||
| test_uuid.c | ||
| test_vmalloc.c | ||
| test_xarray.c | ||
| textsearch.c | ||
| timerqueue.c | ||
| trace_readwrite.c | ||
| ts_bm.c | ||
| ts_fsm.c | ||
| ts_kmp.c | ||
| ubsan.c | ||
| ubsan.h | ||
| ucmpdi2.c | ||
| ucs2_string.c | ||
| usercopy.c | ||
| uuid.c | ||
| vsprintf.c | ||
| win_minmax.c | ||
| xarray.c | ||
| xxhash.c | ||