Aneesh Kumar K.V 2613baa3ab mm/mremap: hold the rmap lock in write mode when moving page table entries. ... commit 97113eb39f upstream. To avoid a race between rmap walk and mremap, mremap does take_rmap_locks(). The lock was taken to ensure that rmap walk don't miss a page table entry due to PTE moves via move_pagetables(). The kernel does further optimization of this lock such that if we are going to find the newly added vma after the old vma, the rmap lock is not taken. This is because rmap walk would find the vmas in the same order and if we don't find the page table attached to older vma we would find it with the new vma which we would iterate later. As explained in commit eb66ae0308 ("mremap: properly flush TLB before releasing the page") mremap is special in that it doesn't take ownership of the page. The optimized version for PUD/PMD aligned mremap also doesn't hold the ptl lock. This can result in stale TLB entries as show below. This patch updates the rmap locking requirement in mremap to handle the race condition explained below with optimized mremap:: Optmized PMD move CPU 1 CPU 2 CPU 3 mremap(old_addr, new_addr) page_shrinker/try_to_unmap_one mmap_write_lock_killable() addr = old_addr lock(pte_ptl) lock(pmd_ptl) pmd = *old_pmd pmd_clear(old_pmd) flush_tlb_range(old_addr) *new_pmd = pmd *new_addr = 10; and fills TLB with new addr and old pfn unlock(pmd_ptl) ptep_clear_flush() old pfn is free. Stale TLB entry Optimized PUD move also suffers from a similar race. Both the above race condition can be fixed if we force mremap path to take rmap lock. Link: https://lkml.kernel.org/r/20210616045239.370802-7-aneesh.kumar@linux.ibm.com Fixes: 2c91bd4a4e ("mm: speed up mremap by 20x on large regions") Fixes: c49dd34018 ("mm: speedup mremap on 1GB or larger regions") Link: https://lore.kernel.org/linux-mm/CAHk-=wgXVR04eBNtxQfevontWnP6FDm+oj5vauQXP3S-huwbPw@mail.gmail.com Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Acked-by: Hugh Dickins <hughd@google.com> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Cc: Christophe Leroy <christophe.leroy@csgroup.eu> Cc: Joel Fernandes <joel@joelfernandes.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Kirill A. Shutemov <kirill@shutemov.name> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Stephen Rothwell <sfr@canb.auug.org.au> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> [patch rewritten for backport since the code was refactored since] Signed-off-by: Jann Horn <jannh@google.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>		2022-08-21 15:15:21 +02:00
..
kasan	kasan: prevent cpu_quarantine corruption when CPU offline and cache shrink occur at same time	2022-05-09 09:05:07 +02:00
backing-dev.c	mm: bdi: initialize bdi_min_ratio when bdi is unregistered	2021-12-14 11:32:37 +01:00
balloon_compaction.c
cleancache.c
cma.c	cma: don't quit at first error when activating reserved areas	2020-08-12 10:57:57 -07:00
cma.h	mm: cma: use CMA_MAX_NAME to define the length of cma name array	2020-09-01 09:19:43 +02:00
cma_debug.c	debugfs: make sure we can remove u32_array files cleanly	2020-07-10 13:54:00 -07:00
compaction.c	mm, compaction: fast_find_migrateblock() should return pfn in the target zone	2022-06-09 10:21:23 +02:00
debug.c	mm, dump_page: rename head_mapcount() --> head_compound_mapcount()	2020-10-13 18:38:29 -07:00
debug_page_ref.c
debug_vm_pgtable.c	mm/debug_vm_pgtable: remove pte entry from the page table	2022-02-08 18:30:35 +01:00
dmapool.c	mm/dmapool.c: replace hard coded function name with __func__	2020-10-13 18:38:32 -07:00
early_ioremap.c
fadvise.c	mm, fadvise: improve the expensive remote LRU cache draining after FADV_DONTNEED	2020-10-13 18:38:29 -07:00
failslab.c
filemap.c	mm/filemap: add missing mem_cgroup_uncharge() to __add_to_page_cache_locked()	2021-02-10 09:29:21 +01:00
frame_vector.c	mmap locking API: convert mmap_sem comments	2020-06-09 09:39:14 -07:00
frontswap.c	mm/frontswap: mark various intentional data races	2020-08-14 19:56:56 -07:00
gup.c	mm/gup: fix try_grab_compound_head() race with split_huge_page()	2021-07-14 16:55:42 +02:00
gup_benchmark.c	mm/gup_benchmark: take the mmap lock around GUP	2020-10-18 09:27:09 -07:00
highmem.c	mm/highmem.c: clean up endif comments	2020-10-16 11:11:18 -07:00
hmm.c	mm/hmm.c: allow VM_MIXEDMAP to work with hmm_range_fault	2022-01-27 10:54:36 +01:00
huge_memory.c	mm/userfaultfd: fix uffd-wp special cases for fork()	2021-07-25 14:36:18 +02:00
hugetlb.c	hugetlb: fix huge_pmd_unshare address update	2022-06-09 10:21:27 +02:00
hugetlb_cgroup.c	hugetlb_cgroup: fix imbalanced css_get and css_put pair for shared mappings	2021-03-30 14:31:54 +02:00
hwpoison-inject.c	mm,hwpoison-inject: don't pin for hwpoison_filter	2020-10-16 11:11:16 -07:00
init-mm.c	mm/gup: prevent gup_fast from racing with COW during fork	2020-12-30 11:53:54 +01:00
internal.h	mm/thp: fix vma_address() if virtual address below file offset	2021-06-30 08:47:27 -04:00
interval_tree.c
ioremap.c	mm: move p?d_alloc_track to separate header file	2020-08-07 11:33:26 -07:00
Kconfig	mm/zsmalloc.c: drop ZSMALLOC_PGTABLE_MAPPING	2020-12-06 10:19:07 -08:00
Kconfig.debug	treewide: replace '---help---' in Kconfig files with 'help'	2020-06-14 01:57:21 +09:00
khugepaged.c	mm: khugepaged: skip huge page collapse for special files	2021-11-06 14:10:09 +01:00
kmemleak.c	mm: kmemleak: take a full lowmem check in kmemleak_*_phys()	2022-04-20 09:23:25 +02:00
ksm.c	ksm: fix potential missing rmap_item for stable_node	2021-05-19 10:13:07 +02:00
list_lru.c	mm: list_lru: set shrinker map bit when child nr_items is not zero	2020-12-06 10:19:07 -08:00
maccess.c	uaccess: add force_uaccess_{begin,end} helpers	2020-08-12 10:57:59 -07:00
madvise.c	Revert "mm: madvise: skip unmapped vma holes passed to process_madvise"	2022-04-08 14:39:54 +02:00
Makefile	mm,kmemleak-test.c: move kmemleak-test.c to samples dir	2020-10-13 18:38:27 -07:00
mapping_dirty_helpers.c
memblock.c	memblock: use kfree() to release kmalloced memblock regions	2022-03-02 11:42:57 +01:00
memcontrol.c	mm/memcontrol: return 1 from cgroup.memory __setup() handler	2022-04-08 14:40:43 +02:00
memfd.c	memfd: fix F_SEAL_WRITE after shmem huge page allocated	2022-03-08 19:09:36 +01:00
memory-failure.c	mm/hwpoison: clear MF_COUNT_INCREASED before retrying get_any_page()	2021-12-29 12:26:05 +01:00
memory.c	mm: split huge PUD on wp_huge_pud fallback	2022-07-21 21:19:59 +02:00
memory_hotplug.c	mm/memory_hotplug: use "unsigned long" for PFN in zone_for_pfn_range()	2021-09-22 12:27:59 +02:00
mempolicy.c	mm/mempolicy: fix uninit-value in mpol_rebind_policy()	2022-07-29 17:19:23 +02:00
mempool.c	mm/mempool: add 'else' to split mutually exclusive case	2020-10-13 18:38:34 -07:00
memremap.c	mm: fix memory_failure() handling of dax-namespace metadata	2021-03-04 11:38:21 +01:00
memtest.c
migrate.c	mm: fix missing cache flush for all tail pages of compound page	2022-05-15 20:00:09 +02:00
mincore.c	mm: factor find_get_incore_page out of mincore_page	2020-10-13 18:38:29 -07:00
mlock.c	mlock: fix unevictable_pgs event counts on THP	2020-09-19 13:13:38 -07:00
mm_init.c	mm: adjust vm_committed_as_batch according to vm overcommit policy	2020-08-07 11:33:26 -07:00
mmap.c	mm, hugetlb: allow for "high" userspace addresses	2022-04-27 13:53:54 +02:00
mmu_gather.c	mmap locking API: convert mmap_sem comments	2020-06-09 09:39:14 -07:00
mmu_notifier.c	mm/mmu_notifier.c: fix race in mmu_interval_notifier_remove()	2022-04-27 13:53:54 +02:00
mmzone.c	arm: remove CONFIG_ARCH_HAS_HOLES_MEMORYMODEL	2022-05-15 20:00:09 +02:00
mprotect.c	mm: don't try to NUMA-migrate COW pages that have other uses	2022-02-23 12:00:57 +01:00
mremap.c	mm/mremap: hold the rmap lock in write mode when moving page table entries.	2022-08-21 15:15:21 +02:00
msync.c	mmap locking API: use coccinelle to convert mmap_sem rwsem call sites	2020-06-09 09:39:14 -07:00
nommu.c	mm: remove alloc_vm_area	2020-10-18 09:27:10 -07:00
oom_kill.c	oom_kill.c: futex: delay the OOM reaper to allow time for proper futex cleanup	2022-04-27 13:53:54 +02:00
page-writeback.c	mm: make wait_on_page_writeback() wait for multiple pending writebacks	2021-01-12 20:18:22 +01:00
page_alloc.c	page_alloc: fix invalid watermark check on a negative value	2022-08-03 12:00:50 +02:00
page_counter.c	mm/page_counter: correct the obsolete func name in the comment of page_counter_try_charge()	2020-10-13 18:38:30 -07:00
page_ext.c	mm/page_ext.c: drop pfn_present() check when onlining	2020-04-07 10:43:40 -07:00
page_idle.c	mm/page_idle.c: skip offline pages	2020-06-08 11:05:55 -07:00
page_io.c	mm: fix unexpected zeroed page mapping with zram swap	2022-04-20 09:23:25 +02:00
page_isolation.c	mm: rename page_order() to buddy_order()	2020-10-16 11:11:19 -07:00
page_owner.c	mm: rename page_order() to buddy_order()	2020-10-16 11:11:19 -07:00
page_poison.c	mm/page_poison.c: replace bool variable with static key	2020-10-16 11:11:17 -07:00
page_reporting.c	mm: rename page_order() to buddy_order()	2020-10-16 11:11:19 -07:00
page_reporting.h	mm: introduce include/linux/pgtable.h	2020-06-09 09:39:13 -07:00
page_vma_mapped.c	mm/thp: another PVMW_SYNC fix in page_vma_mapped_walk()	2021-06-30 08:47:29 -04:00
pagewalk.c	mmap locking API: convert mmap_sem comments	2020-06-09 09:39:14 -07:00
percpu-internal.h	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-14 08:42:03 +02:00
percpu-km.c	mm: memcg/percpu: account percpu memory to memory cgroups	2020-08-12 10:57:55 -07:00
percpu-stats.c	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-14 08:42:03 +02:00
percpu-vm.c	mm: memcg/percpu: account percpu memory to memory cgroups	2020-08-12 10:57:55 -07:00
percpu.c	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-14 08:42:03 +02:00
pgalloc-track.h	mm: move p?d_alloc_track to separate header file	2020-08-07 11:33:26 -07:00
pgtable-generic.c	mm/thp: fix __split_huge_pmd_locked() on shmem migration entry	2021-06-30 08:47:26 -04:00
process_vm_access.c	mm/process_vm_access.c: include compat.h	2021-01-19 18:27:21 +01:00
ptdump.c	mm: ptdump: fix build failure	2021-04-21 13:00:57 +02:00
readahead.c	mm: use limited read-ahead to satisfy read	2020-10-17 13:49:08 -06:00
rmap.c	mm: fix race between MADV_FREE reclaim and blkdev direct IO read	2022-04-13 21:01:03 +02:00
rodata_test.c	mm/rodata_test.c: fix missing function declaration	2020-08-21 09:52:53 -07:00
shmem.c	shmem: fix a race between shmem_unused_huge_shrink and shmem_evict_inode	2022-01-27 10:53:44 +01:00
shuffle.c	mm: rename page_order() to buddy_order()	2020-10-16 11:11:19 -07:00
shuffle.h	mm/shuffle: remove dynamic reconfiguration	2020-08-07 11:33:29 -07:00
slab.c	mm/sl?b.c: remove ctor argument from kmem_cache_flags	2021-05-14 09:50:45 +02:00
slab.h	mm: kmemleak: slob: respect SLAB_NOLEAKTRACE flag	2021-11-26 10:39:19 +01:00
slab_common.c	mm/slub: fix redzoning for small allocations	2021-06-23 14:42:54 +02:00
slob.c	mm: memcg: convert vmstat slab counters to bytes	2020-08-07 11:33:24 -07:00
slub.c	mm/slub: add missing TID updates on slab deactivation	2022-07-12 16:32:16 +02:00
sparse-vmemmap.c	mm/sparse: only sub-section aligned range would be populated	2020-08-07 11:33:27 -07:00
sparse.c	mm/sparse: add the missing sparse_buffer_fini() in error branch	2021-05-14 09:50:45 +02:00
swap.c	mm: move call to compound_head() in release_pages()	2020-10-13 18:38:33 -07:00
swap_cgroup.c	mm: memcontrol: make swap tracking an integral part of memory control	2020-06-03 20:09:48 -07:00
swap_slots.c	mm/swap_slots.c: remove always zero and unused return value of enable_swap_slots_cache()	2020-10-13 18:38:30 -07:00
swap_state.c	mm: swap: get rid of livelock in swapin readahead	2022-03-23 09:13:27 +01:00
swapfile.c	mm/swap: fix pte_same_as_swp() not removing uffd-wp bit when compare	2021-06-23 14:42:53 +02:00
truncate.c	mm/thp: unmap_mapping_page() to fix THP truncate_cleanup_page()	2021-06-30 08:47:27 -04:00
usercopy.c	mm/usercopy: return 1 from hardened_usercopy __setup() handler	2022-04-08 14:40:43 +02:00
userfaultfd.c	mm: userfaultfd: fix missing cache flush in mcopy_atomic_pte() and __mcopy_atomic()	2022-05-15 20:00:09 +02:00
util.c	mm: Add kvrealloc()	2022-08-21 15:15:21 +02:00
vmacache.c	kernel: better document the use_mm/unuse_mm API contract	2020-06-10 19:14:18 -07:00
vmalloc.c	mm/vmalloc.c: fix potential memory leak	2021-01-19 18:27:21 +01:00
vmpressure.c	mm: vmpressure: use mem_cgroup_is_root API	2020-04-02 09:35:31 -07:00
vmscan.c	mm,vmscan: fix divide by zero in get_scan_count	2021-09-18 13:40:36 +02:00
vmstat.c	arm: remove CONFIG_ARCH_HAS_HOLES_MEMORYMODEL	2022-05-15 20:00:09 +02:00
workingset.c	XArray updates for 5.9	2020-10-20 14:39:37 -07:00
z3fold.c	mm/z3fold: use release_z3fold_page_locked() to release locked z3fold page	2021-07-14 16:56:51 +02:00
zbud.c	mm/zbud: remove redundant initialization	2020-10-13 18:38:34 -07:00
zpool.c	mm/zpool.c: delete duplicated word and fix grammar	2020-08-12 10:57:58 -07:00
zsmalloc.c	zsmalloc: fix races between asynchronous zspage free and page migration	2022-06-06 08:42:43 +02:00
zswap.c	mm/zswap: allow setting default status, compressor and allocator in Kconfig	2020-04-07 10:43:41 -07:00