Lance Yang 1b68112c40 mm/madvise: introduce clear_young_dirty_ptes() batch helper ... Patch series "mm/madvise: enhance lazyfreeing with mTHP in madvise_free", v10. This patchset adds support for lazyfreeing multi-size THP (mTHP) without needing to first split the large folio via split_folio(). However, we still need to split a large folio that is not fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. Performance Testing =================== On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% This patch (of 4): This commit introduces clear_young_dirty_ptes() to replace mkold_ptes(). By doing so, we can use the same function for both use cases (madvise_pageout and madvise_free), and it also provides the flexibility to only clear the dirty flag in the future if needed. Link: https://lkml.kernel.org/r/20240418134435.6092-1-ioworker0@gmail.com Link: https://lkml.kernel.org/r/20240418134435.6092-2-ioworker0@gmail.com Signed-off-by: Lance Yang <ioworker0@gmail.com> Suggested-by: Ryan Roberts <ryan.roberts@arm.com> Acked-by: David Hildenbrand <david@redhat.com> Reviewed-by: Ryan Roberts <ryan.roberts@arm.com> Cc: Barry Song <21cnbao@gmail.com> Cc: Jeff Xie <xiehuan09@gmail.com> Cc: Kefeng Wang <wangkefeng.wang@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Minchan Kim <minchan@kernel.org> Cc: Muchun Song <songmuchun@bytedance.com> Cc: Peter Xu <peterx@redhat.com> Cc: Yang Shi <shy828301@gmail.com> Cc: Yin Fengwei <fengwei.yin@intel.com> Cc: Zach O'Keefe <zokeefe@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>		2024-05-05 17:53:42 -07:00
..
damon	mm: madvise: pageout: ignore references rather than clearing young	2024-03-04 17:01:18 -08:00
kasan	fix missing vmalloc.h includes	2024-04-25 20:55:49 -07:00
kfence	mm: introduce slabobj_ext to support slab object extensions	2024-04-25 20:55:51 -07:00
kmsan	mm: kmsan: remove runtime checks from kmsan_unpoison_memory()	2024-02-22 10:24:41 -08:00
Kconfig	mm/treewide: rename CONFIG_HAVE_FAST_GUP to CONFIG_HAVE_GUP_FAST	2024-04-25 20:56:41 -07:00
Kconfig.debug	mm/slub: unify all sl[au]b parameters with "slab_$param"	2024-01-22 10:31:08 +01:00
Makefile	mm/kmemleak: disable KASAN instrumentation in kmemleak	2024-04-25 20:56:05 -07:00
backing-dev.c	mm: backing-dev: use group allocation/free of per-cpu counters API	2024-04-25 20:56:12 -07:00
balloon_compaction.c	…
bootmem_info.c	bootmem: use kmemleak_free_part_phys in put_page_bootmem	2023-10-25 16:47:13 -07:00
cma.c	mm/cma: drop incorrect alignment check in cma_init_reserved_mem	2024-04-25 20:56:42 -07:00
cma.h	mm/cma: add sysfs file 'release_pages_success'	2024-02-22 10:24:57 -08:00
cma_debug.c	…
cma_sysfs.c	mm/cma: add sysfs file 'release_pages_success'	2024-02-22 10:24:57 -08:00
compaction.c	memory: remove the now superfluous sentinel element from ctl_table array	2024-04-25 20:56:32 -07:00
debug.c	mm/debug: print only page mapcount (excluding folio entire mapcount) in __dump_folio()	2024-05-05 17:53:31 -07:00
debug_page_alloc.c	mm: page_alloc: consolidate free page accounting	2024-04-25 20:56:04 -07:00
debug_page_ref.c	…
debug_vm_pgtable.c	fix missing vmalloc.h includes	2024-04-25 20:55:49 -07:00
dmapool.c	mm/mempool/dmapool: remove CONFIG_DEBUG_SLAB ifdefs	2023-12-05 11:17:58 +01:00
dmapool_test.c	…
early_ioremap.c	…
fadvise.c	…
fail_page_alloc.c	…
failslab.c	…
filemap.c	mm: filemap: batch mm counter updating in filemap_map_pages()	2024-05-05 17:53:36 -07:00
folio-compat.c	mm: remove __set_page_dirty_nobuffers()	2024-04-25 20:56:25 -07:00
gup.c	mm/treewide: rename CONFIG_HAVE_FAST_GUP to CONFIG_HAVE_GUP_FAST	2024-04-25 20:56:41 -07:00
gup_test.c	…
gup_test.h	…
highmem.c	x86/kexec: use pr_err() instead of kexec_dprintk() when an error occurs	2023-12-29 12:22:28 -08:00
hmm.c	mm/treewide: replace pXd_huge() with pXd_leaf()	2024-04-25 20:55:46 -07:00
huge_memory.c	mm/huge_memory: improve split_huge_page_to_list_to_order() return value documentation	2024-05-05 17:53:42 -07:00
hugetlb.c	mm/hugetlb: rename dissolve_free_huge_pages() to dissolve_free_hugetlb_folios()	2024-05-05 17:53:35 -07:00
hugetlb_cgroup.c	mm/hugetlb: assert hugetlb_lock in __hugetlb_cgroup_commit_charge	2024-05-05 17:53:41 -07:00
hugetlb_vmemmap.c	memory: remove the now superfluous sentinel element from ctl_table array	2024-04-25 20:56:32 -07:00
hugetlb_vmemmap.h	mm: hugetlb_vmemmap: fix reference to nonexistent file	2023-10-25 16:47:14 -07:00
hwpoison-inject.c	…
init-mm.c	mm: Deprecate pasid field	2023-12-12 10:11:32 +01:00
internal.h	mm: track mapcount of large folios in single value	2024-05-05 17:53:28 -07:00
interval_tree.c	…
io-mapping.c	…
ioremap.c	…
khugepaged.c	mm: track mapcount of large folios in single value	2024-05-05 17:53:28 -07:00
kmemleak.c	mm/kmemleak: compact kmemleak_object further	2024-04-25 20:56:05 -07:00
ksm.c	mm/ksm: remove page_mapcount() usage in stable_tree_search()	2024-05-05 17:53:38 -07:00
list_lru.c	mm/zswap: stop lru list shrinking when encounter warm region	2024-02-22 10:24:54 -08:00
maccess.c	…
madvise.c	mm/madvise: introduce clear_young_dirty_ptes() batch helper	2024-05-05 17:53:42 -07:00
mapping_dirty_helpers.c	mm: fix clean_record_shared_mapping_range kernel-doc	2023-08-24 16:20:30 -07:00
memblock.c	cxl fixes for 6.8-rc6	2024-02-24 15:53:40 -08:00
memcontrol.c	mm, slab: move slab_memcg hooks to mm/memcontrol.c	2024-04-25 20:56:16 -07:00
memfd.c	mm/memfd: refactor memfd_tag_pins() and memfd_wait_for_pins()	2024-03-04 17:01:21 -08:00
memory-failure.c	mm/hugetlb: convert dissolve_free_huge_pages() to folios	2024-05-05 17:53:35 -07:00
memory-tiers.c	memory tier: create CPUless memory tiers after obtaining HMAT info	2024-05-05 17:53:26 -07:00
memory.c	mm: move mm counter updating out of set_pte_range()	2024-05-05 17:53:36 -07:00
memory_hotplug.c	mm/hugetlb: rename dissolve_free_huge_pages() to dissolve_free_hugetlb_folios()	2024-05-05 17:53:35 -07:00
mempolicy.c	mm: add pmd_folio()	2024-04-25 20:56:19 -07:00
mempool.c	mempool: hook up to memory allocation profiling	2024-04-25 20:55:56 -07:00
memremap.c	mm: convert free_zone_device_page to free_zone_device_folio	2024-04-25 20:56:44 -07:00
memtest.c	memtest: use {READ,WRITE}_ONCE in memory scanning	2024-03-13 12:12:21 -07:00
migrate.c	mm/ksm: convert chain series funcs and replace get_ksm_page	2024-05-05 17:53:34 -07:00
migrate_device.c	mm/migrate_device: use folio_mapcount() in migrate_vma_check_page()	2024-05-05 17:53:31 -07:00
mincore.c	mm: enable page walking API to lock vmas during the walk	2023-08-21 13:07:20 -07:00
mlock.c	mm: add pmd_folio()	2024-04-25 20:56:19 -07:00
mm_init.c	mm: init_mlocked_on_free_v3	2024-04-25 20:56:29 -07:00
mm_slot.h	…
mmap.c	mm/mmap: make accountable_mapping return bool	2024-05-05 17:53:26 -07:00
mmap_lock.c	…
mmu_gather.c	mm/mmu_gather: improve cond_resched() handling with large folios and expensive page freeing	2024-02-22 15:27:17 -08:00
mmu_notifier.c	…
mmzone.c	zswap: shrink zswap pool based on memory pressure	2023-12-12 10:57:02 -08:00
mprotect.c	mm: support multi-size THP numa balancing	2024-04-25 20:56:30 -07:00
mremap.c	mm: remove "prot" parameter from move_pte()	2024-04-25 20:56:24 -07:00
msync.c	…
nommu.c	mm: remove follow_pfn	2024-04-25 20:56:12 -07:00
oom_kill.c	memory: remove the now superfluous sentinel element from ctl_table array	2024-04-25 20:56:32 -07:00
page-writeback.c	doc: improve the description of __folio_mark_dirty	2024-05-05 17:53:39 -07:00
page_alloc.c	mm: page_alloc: allowing mTHP compaction to capture the freed page directly	2024-05-05 17:53:37 -07:00
page_counter.c	…
page_ext.c	mm: make page_ext_get() take a const argument	2024-04-25 20:56:14 -07:00
page_idle.c	…
page_io.c	mm: add per-order mTHP anon_swpout and anon_swpout_fallback counters	2024-05-05 17:53:35 -07:00
page_isolation.c	mm: page_isolation: prepare for hygienic freelists	2024-04-25 20:56:04 -07:00
page_owner.c	mm: introduce slabobj_ext to support slab object extensions	2024-04-25 20:55:51 -07:00
page_poison.c	mm/page_poison: replace kmap_atomic() with kmap_local_page()	2023-12-10 16:51:50 -08:00
page_reporting.c	mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER	2024-01-08 15:27:15 -08:00
page_reporting.h	…
page_table_check.c	mm/page_table_check: support userfault wr-protect entries	2024-05-05 17:53:41 -07:00
page_vma_mapped.c	mm: rename vma_pgoff_address back to vma_address	2024-04-25 20:56:31 -07:00
pagewalk.c	mm: pagewalk: assert write mmap lock only for walking the user page tables	2023-12-10 16:51:53 -08:00
percpu-internal.h	mm: percpu: add codetag reference into pcpuobj_ext	2024-04-25 20:55:56 -07:00
percpu-km.c	…
percpu-stats.c	…
percpu-vm.c	percpu: clean up all mappings when pcpu_map_pages() fails	2024-04-25 20:55:49 -07:00
percpu.c	mm: percpu: enable per-cpu allocation tagging	2024-04-25 20:55:56 -07:00
pgalloc-track.h	…
pgtable-generic.c	…
process_vm_access.c	mm: fix process_vm_rw page counts	2023-12-10 16:51:39 -08:00
ptdump.c	mm: ptdump: add check_wx_pages debugfs attribute	2024-02-22 10:24:47 -08:00
readahead.c	mm/readahead: break read-ahead loop if filemap_add_folio return -ENOMEM	2024-04-25 20:56:07 -07:00
rmap.c	mm: track mapcount of large folios in single value	2024-05-05 17:53:28 -07:00
rodata_test.c	…
secretmem.c	mm/secretmem: use a folio in secretmem_fault()	2023-08-21 13:38:02 -07:00
shmem.c	mm: switch mm->get_unmapped_area() to a flag	2024-04-25 20:56:25 -07:00
shmem_quota.c	tmpfs: fix race on handling dquot rbtree	2024-03-26 11:07:23 -07:00
show_mem.c	lib: add memory allocations report in show_mem()	2024-04-25 20:55:57 -07:00
shrinker.c	mm: shrinker: use kvzalloc_node() from expand_one_shrinker_info()	2024-01-05 09:58:32 -08:00
shrinker_debug.c	mm: shrinker: convert shrinker_rwsem to mutex	2023-10-04 10:32:26 -07:00
shuffle.c	…
shuffle.h	mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER	2024-01-08 15:27:15 -08:00
slab.h	mm, slab: move slab_memcg hooks to mm/memcontrol.c	2024-04-25 20:56:16 -07:00
slab_common.c	mm/slab: enable slab allocation tagging for kmalloc and friends	2024-04-25 20:55:55 -07:00
slub.c	mm, slab: move slab_memcg hooks to mm/memcontrol.c	2024-04-25 20:56:16 -07:00
sparse-vmemmap.c	mm/vmemmap: allow architectures to override how vmemmap optimization works	2023-08-18 10:12:53 -07:00
sparse.c	mm/sparse: guard the size of mem_section is power of 2	2024-05-05 17:53:40 -07:00
swap.c	mm: convert free_zone_device_page to free_zone_device_folio	2024-04-25 20:56:44 -07:00
swap.h	mm/swap: fix race when skipping swapcache	2024-02-20 14:20:48 -08:00
swap_cgroup.c	…
swap_slots.c	mm: swap: update get_swap_pages() to take folio order	2024-04-25 20:56:37 -07:00
swap_state.c	mm: remove struct page from get_shadow_from_swap_cache	2024-04-25 20:56:40 -07:00
swapfile.c	mm: swapfile: check usable swap device in __folio_throttle_swaprate()	2024-05-05 17:53:42 -07:00
truncate.c	mm: convert pagecache_isize_extended to use a folio	2024-04-25 20:56:43 -07:00
usercopy.c	…
userfaultfd.c	userfaultfd: remove WRITE_ONCE when setting folio->index during UFFDIO_MOVE	2024-05-05 17:53:37 -07:00
util.c	mm: switch mm->get_unmapped_area() to a flag	2024-04-25 20:56:25 -07:00
vmalloc.c	mm/vmalloc.c: optimize to reduce arguments of alloc_vmap_area()	2024-04-25 20:56:08 -07:00
vmpressure.c	eventfd: simplify eventfd_signal()	2023-11-28 14:08:38 +01:00
vmscan.c	mm: add per-order mTHP anon_swpout and anon_swpout_fallback counters	2024-05-05 17:53:35 -07:00
vmstat.c	mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER	2024-01-08 15:27:15 -08:00
workingset.c	mm: move mapping_set_update out of <linux/swap.h>	2024-02-21 11:36:50 +05:30
z3fold.c	mm: zpool: return pool size in pages	2024-04-25 20:55:48 -07:00
zbud.c	mm: zpool: return pool size in pages	2024-04-25 20:55:48 -07:00
zpool.c	mm: zpool: return pool size in pages	2024-04-25 20:55:48 -07:00
zsmalloc.c	mm: zpool: return pool size in pages	2024-04-25 20:55:48 -07:00
zswap.c	mm: zswap: remove same_filled module params	2024-05-05 17:53:38 -07:00