linux-stable/mm
Jann Horn 275c626c13 mm/khugepaged: invoke MMU notifiers in shmem/file collapse paths
commit f268f6cf87 upstream.

Any codepath that zaps page table entries must invoke MMU notifiers to
ensure that secondary MMUs (like KVM) don't keep accessing pages which
aren't mapped anymore.  Secondary MMUs don't hold their own references to
pages that are mirrored over, so failing to notify them can lead to page
use-after-free.

I'm marking this as addressing an issue introduced in commit f3f0e1d215
("khugepaged: add support of collapse for tmpfs/shmem pages"), but most of
the security impact of this only came in commit 27e1f82731 ("khugepaged:
enable collapse pmd for pte-mapped THP"), which actually omitted flushes
for the removal of present PTEs, not just for the removal of empty page
tables.

Link: https://lkml.kernel.org/r/20221129154730.2274278-3-jannh@google.com
Link: https://lkml.kernel.org/r/20221128180252.1684965-3-jannh@google.com
Link: https://lkml.kernel.org/r/20221125213714.4115729-3-jannh@google.com
Fixes: f3f0e1d215 ("khugepaged: add support of collapse for tmpfs/shmem pages")
Signed-off-by: Jann Horn <jannh@google.com>
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Yang Shi <shy828301@gmail.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[manual backport: this code was refactored from two copies into a common
helper between 5.15 and 6.0;
pmd collapse for PTE-mapped THP was only added in 5.4;
MMU notifier API changed between 4.19 and 5.4]
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2023-01-07 12:07:10 +01:00
..
kasan kasan: avoid -Wmaybe-uninitialized warning 2019-05-08 07:19:07 +02:00
Kconfig mm: don't allow deferred pages with NEED_PER_CPU_KM 2018-05-22 16:57:57 +02:00
Kconfig.debug PM / Hibernate: allow hibernation with PAGE_POISONING_ZERO 2016-09-13 02:35:27 +02:00
Makefile Disable the __builtin_return_address() warning globally after all 2016-10-12 10:23:41 -07:00
backing-dev.c mm: bdi: initialize bdi_min_ratio when bdi is unregistered 2021-12-14 10:04:47 +01:00
balloon_compaction.c
bootmem.c mm: kmemleak: avoid using __va() on addresses that don't have a lowmem mapping 2016-10-11 15:06:33 -07:00
cleancache.c
cma.c mm/cma.c: fail if fixed declaration can't be honored 2019-08-06 18:29:37 +02:00
cma.h
cma_debug.c mm/cma_debug.c: fix the break condition in cma_maxchunk_get() 2019-06-22 08:17:12 +02:00
compaction.c mm, compaction: fix NR_ISOLATED_* stats for pfn based migration 2017-01-12 11:39:32 +01:00
debug.c mm: get rid of vmacache_flush_all() entirely 2018-09-19 22:47:17 +02:00
debug_page_ref.c
dmapool.c
early_ioremap.c mm/early_ioremap: Fix boot hang with earlyprintk=efi,keep 2018-02-25 11:05:49 +01:00
fadvise.c mm/fadvise.c: fix signed overflow UBSAN complaint 2018-09-15 09:42:57 +02:00
failslab.c
filemap.c mm: fs: initialize fsdata passed to write_begin/write_end interface 2022-11-25 17:35:43 +01:00
frame_vector.c v4l2: don't fall back to follow_pfn() if pin_user_pages_fast() fails 2022-12-08 11:15:42 +01:00
frontswap.c
gup.c gup: document and work around "COW can break either way" issue 2022-01-27 08:47:42 +01:00
highmem.c
huge_memory.c gup: document and work around "COW can break either way" issue 2022-01-27 08:47:42 +01:00
hugetlb.c mm,hugetlb: take hugetlb_lock before decrementing h->resv_huge_pages 2022-11-03 23:49:17 +09:00
hugetlb_cgroup.c mm: hugetlb: switch to css_tryget() in hugetlb_cgroup_charge_cgroup() 2019-11-25 09:51:58 +01:00
hwpoison-inject.c
init-mm.c mm: Add a user_ns owner to mm_struct and fix ptrace permission checks 2017-01-06 10:40:13 +01:00
internal.h vmscan: return NODE_RECLAIM_NOSCAN in node_reclaim() when CONFIG_NUMA is n 2019-12-05 15:35:02 +01:00
interval_tree.c
khugepaged.c mm/khugepaged: invoke MMU notifiers in shmem/file collapse paths 2023-01-07 12:07:10 +01:00
kmemcheck.c
kmemleak-test.c
kmemleak.c Revert "mm: kmemleak: take a full lowmem check in kmemleak_*_phys()" 2022-09-15 12:39:45 +02:00
ksm.c ksm: fix potential missing rmap_item for stable_node 2021-05-22 10:40:31 +02:00
list_lru.c mm/list_lru.c: fix memory leak in __memcg_init_list_lru_node 2019-06-22 08:17:18 +02:00
maccess.c uaccess: Add non-pagefault user-space write function 2020-09-12 11:47:35 +02:00
madvise.c mm: madvise(MADV_DODUMP): allow hugetlbfs pages 2018-10-10 08:53:20 +02:00
memblock.c memblock: use kfree() to release kmalloced memblock regions 2022-03-02 11:32:06 +01:00
memcontrol.c mm/memcontrol: return 1 from cgroup.memory __setup() handler 2022-04-20 09:06:41 +02:00
memory-failure.c mm: hwpoison: change PageHWPoison behavior on hugetlb pages 2021-06-30 08:49:13 -04:00
memory.c mm/khugepaged: fix GUP-fast interaction by sending IPI 2023-01-07 12:07:10 +01:00
memory_hotplug.c mm: Avoid calling build_all_zonelists_init under hotplug context 2020-08-21 11:02:11 +02:00
mempolicy.c mm/mempolicy: fix uninit-value in mpol_rebind_policy() 2022-07-29 17:05:46 +02:00
mempool.c Revert "mm, mempool: only set __GFP_NOMEMALLOC if there are free elements" 2016-07-28 16:07:41 -07:00
memtest.c
migrate.c hugetlbfs: fix races and page leaks during migration 2019-03-13 14:04:54 -07:00
mincore.c mm/mincore.c: make mincore() more conservative 2019-05-21 18:48:58 +02:00
mlock.c mm/mlock.c: change count_mm_mlocked_page_nr return type 2019-07-10 09:55:43 +02:00
mm_init.c
mmap.c mm: Fix TLB flush for not-first PFNMAP mappings in unmap_region() 2022-09-20 11:50:16 +02:00
mmu_context.c
mmu_notifier.c mm/mmu_notifier: use hlist_add_head_rcu() 2019-08-04 09:33:42 +02:00
mmzone.c
mprotect.c x86/speculation/l1tf: Disallow non privileged high MMIO PROT_NONE mappings 2018-08-15 18:14:45 +02:00
mremap.c mmmremap.c: avoid pointless invalidate_range_start/end on mremap(old_size=0) 2022-04-20 09:06:44 +02:00
msync.c
nobootmem.c mm: discard memblock data later 2017-08-24 17:12:19 -07:00
nommu.c x86/mm: split vmalloc_sync_all() 2020-04-02 17:20:26 +02:00
oom_kill.c mm, oom: do not trigger out_of_memory from the #PF 2021-11-26 11:48:40 +01:00
page-writeback.c mm: memcontrol: fix NULL pointer crash in test_clear_page_writeback() 2021-02-23 13:59:14 +01:00
page_alloc.c mm: prevent page_frag_alloc() from corrupting the memory 2022-10-26 13:15:32 +02:00
page_counter.c
page_ext.c mm/page_ext.c: fix an imbalance with kmemleak 2019-04-05 22:29:06 +02:00
page_idle.c mm/page_idle.c: fix oops because end_pfn is larger than max_pfn 2019-07-10 09:55:38 +02:00
page_io.c swap: fix swapfile read/write offset 2021-03-07 11:25:59 +01:00
page_isolation.c mm/page_isolation: fix typo: "paes" -> "pages" 2016-10-07 18:46:29 -07:00
page_owner.c mm/page_owner: don't define fields on struct page_ext by hard-coding 2016-10-07 18:46:27 -07:00
page_poison.c
pagewalk.c mm: pagewalk: fix termination condition in walk_pte_range() 2020-10-01 20:40:06 +02:00
percpu-km.c
percpu-vm.c
percpu.c percpu: stop printing kernel addresses 2019-04-27 09:34:47 +02:00
pgtable-generic.c
process_vm_access.c mm: remove write/force parameters from __get_user_pages_unlocked() 2016-10-18 14:13:37 -07:00
quicklist.c
readahead.c mm: silently skip readahead for DAX inodes 2016-08-26 17:39:35 -07:00
rmap.c mm/rmap: Fix anon_vma->degree ambiguity leading to double-reuse 2022-09-05 10:23:58 +02:00
shmem.c memfd: fix F_SEAL_WRITE after shmem huge page allocated 2022-03-08 19:00:59 +01:00
slab.c mm/slab.c: fix an infinite loop in leaks_show() 2019-06-22 08:17:13 +02:00
slab.h mm: kmemleak: slob: respect SLAB_NOLEAKTRACE flag 2021-11-26 11:48:42 +01:00
slab_common.c mm/slab: use memzero_explicit() in kzfree() 2020-06-30 15:38:44 -04:00
slob.c slub: move synchronize_sched out of slab_mutex on shrink 2017-03-22 12:43:38 +01:00
slub.c mm/slub: fix to return errno if kmalloc() fails 2022-09-28 10:55:45 +02:00
sparse-vmemmap.c treewide: replace obsolete _refok by __ref 2016-08-02 17:31:41 -04:00
sparse.c mm/memory_hotplug: set magic number to page->freelist instead of page->lru.next 2017-10-21 17:21:36 +02:00
swap.c thp: reduce usage of huge zero page's atomic counter 2016-10-07 18:46:28 -07:00
swap_cgroup.c mm, swap_cgroup: reschedule when neeed in swap_cgroup_swapoff() 2017-07-05 14:40:17 +02:00
swap_state.c mm: fix swap cache node allocation mask 2020-07-09 09:35:54 +02:00
swapfile.c swap: fix swapfile read/write offset 2021-03-07 11:25:59 +01:00
truncate.c mm: cleancache: fix corruption on missed inode invalidation 2018-12-08 13:05:09 +01:00
usercopy.c usercopy: Avoid HIGHMEM pfn warning 2019-10-17 13:42:06 -07:00
userfaultfd.c mm: userfaultfd: fix missing cache flush in mcopy_atomic_pte() and __mcopy_atomic() 2022-05-15 19:39:17 +02:00
util.c random: move randomize_page() into mm where it belongs 2022-06-25 11:45:15 +02:00
vmacache.c mm: get rid of vmacache_flush_all() entirely 2018-09-19 22:47:17 +02:00
vmalloc.c mm/vmalloc.c: don't dereference possible NULL pointer in __vunmap() 2020-06-03 08:16:47 +02:00
vmpressure.c mm: vmpressure: fix sending wrong events on underflow 2017-03-12 06:41:43 +01:00
vmscan.c mm: remove seemingly spurious reclaimability check from laptop_mode gating 2018-09-19 22:47:12 +02:00
vmstat.c mm, vmstat: drop zone->lock in /proc/pagetypeinfo 2021-06-03 08:23:27 +02:00
workingset.c mm: workingset: fix premature shadow node shrinking with cgroups 2017-04-08 09:30:36 +02:00
z3fold.c
zbud.c
zpool.c
zsmalloc.c mm/zsmalloc.c: close race window between zs_pool_dec_isolated() and zs_unregister_migration() 2021-11-26 11:48:39 +01:00
zswap.c zswap: re-check zswap_is_full() after do zswap_shrink() 2018-09-05 09:20:02 +02:00