linux-stable/arch/x86/mm
Kirill A. Shutemov 39b9552281 x86/mm: Optimize boot-time paging mode switching cost
By this point we have functioning boot-time switching between 4- and
5-level paging mode. But naive approach comes with cost.

Numbers below are for kernel build, allmodconfig, 5 times.

CONFIG_X86_5LEVEL=n:

 Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):

   17308719.892691      task-clock:u (msec)       #   26.772 CPUs utilized            ( +-  0.11% )
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
       331,993,164      page-faults:u             #    0.019 M/sec                    ( +-  0.01% )
43,614,978,867,455      cycles:u                  #    2.520 GHz                      ( +-  0.01% )
39,371,534,575,126      stalled-cycles-frontend:u #   90.27% frontend cycles idle     ( +-  0.09% )
28,363,350,152,428      instructions:u            #    0.65  insn per cycle
                                                  #    1.39  stalled cycles per insn  ( +-  0.00% )
 6,316,784,066,413      branches:u                #  364.948 M/sec                    ( +-  0.00% )
   250,808,144,781      branch-misses:u           #    3.97% of all branches          ( +-  0.01% )

     646.531974142 seconds time elapsed                                          ( +-  1.15% )

CONFIG_X86_5LEVEL=y:

 Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):

   17411536.780625      task-clock:u (msec)       #   26.426 CPUs utilized            ( +-  0.10% )
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
       331,868,663      page-faults:u             #    0.019 M/sec                    ( +-  0.01% )
43,865,909,056,301      cycles:u                  #    2.519 GHz                      ( +-  0.01% )
39,740,130,365,581      stalled-cycles-frontend:u #   90.59% frontend cycles idle     ( +-  0.05% )
28,363,358,997,959      instructions:u            #    0.65  insn per cycle
                                                  #    1.40  stalled cycles per insn  ( +-  0.00% )
 6,316,784,937,460      branches:u                #  362.793 M/sec                    ( +-  0.00% )
   251,531,919,485      branch-misses:u           #    3.98% of all branches          ( +-  0.00% )

     658.886307752 seconds time elapsed                                          ( +-  0.92% )

The patch tries to fix the performance regression by using
cpu_feature_enabled(X86_FEATURE_LA57) instead of pgtable_l5_enabled in
all hot code paths. These will statically patch the target code for
additional performance.

CONFIG_X86_5LEVEL=y + the patch:

 Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):

   17381990.268506      task-clock:u (msec)       #   26.907 CPUs utilized            ( +-  0.19% )
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
       331,862,625      page-faults:u             #    0.019 M/sec                    ( +-  0.01% )
43,697,726,320,051      cycles:u                  #    2.514 GHz                      ( +-  0.03% )
39,480,408,690,401      stalled-cycles-frontend:u #   90.35% frontend cycles idle     ( +-  0.05% )
28,363,394,221,388      instructions:u            #    0.65  insn per cycle
                                                  #    1.39  stalled cycles per insn  ( +-  0.00% )
 6,316,794,985,573      branches:u                #  363.410 M/sec                    ( +-  0.00% )
   251,013,232,547      branch-misses:u           #    3.97% of all branches          ( +-  0.01% )

     645.991174661 seconds time elapsed                                          ( +-  1.19% )

Unfortunately, this approach doesn't help with text size:

  vmlinux.before .text size:	8190319
  vmlinux.after .text size:	8200623

The .text section is increased by about 4k. Not sure if we can do anything
about this.

Signed-off-by: Kirill A. Shuemov <kirill.shutemov@linux.intel.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Borislav Petkov <bp@suse.de>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/20180216114948.68868-4-kirill.shutemov@linux.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-02-21 10:19:18 +01:00
..
amdtopology.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
cpu_entry_area.c x86/cpu_entry_area: Add debugstore entries to cpu_entry_area 2017-12-23 21:13:00 +01:00
debug_pagetables.c x86/mm/dump_pagetables: Add the EFI pagetable to the debugfs 'page_tables' directory 2018-02-13 16:00:45 +01:00
dump_pagetables.c x86/mm: Replace compile-time checks for 5-level paging with runtime-time checks 2018-02-16 10:48:49 +01:00
extable.c x86/extable: Mark exception handler functions visible 2018-01-14 20:04:16 +01:00
fault.c x86/mm: Replace compile-time checks for 5-level paging with runtime-time checks 2018-02-16 10:48:49 +01:00
highmem_32.c x86/mm: Audit and remove any unnecessary uses of module.h 2016-07-14 13:04:20 +02:00
hugetlbpage.c x86/mm: Prevent non-MAP_FIXED mapping across DEFAULT_MAP_WINDOW border 2017-11-16 11:43:11 +01:00
ident_map.c x86/mm: Replace compile-time checks for 5-level paging with runtime-time checks 2018-02-16 10:48:49 +01:00
init.c Merge branch 'x86-pti-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2018-01-05 12:23:57 -08:00
init_32.c mm: pass the vmem_altmap to arch_remove_memory and __remove_pages 2018-01-08 11:46:23 -08:00
init_64.c x86/mm: Replace compile-time checks for 5-level paging with runtime-time checks 2018-02-16 10:48:49 +01:00
iomap_32.c x86/mm: Audit and remove any unnecessary uses of module.h 2016-07-14 13:04:20 +02:00
ioremap.c x86/mm/kmmio: Fix mmiotrace for page unaligned addresses 2017-12-11 15:35:18 +01:00
kasan_init_64.c x86/mm: Optimize boot-time paging mode switching cost 2018-02-21 10:19:18 +01:00
kaslr.c x86/mm: Replace compile-time checks for 5-level paging with runtime-time checks 2018-02-16 10:48:49 +01:00
kmmio.c x86/mm/kmmio: Fix mmiotrace for page unaligned addresses 2017-12-11 15:35:18 +01:00
Makefile x86/mm/encrypt: Move page table helpers into separate translation unit 2018-02-13 15:59:48 +01:00
mem_encrypt.c x86/mm/encrypt: Move page table helpers into separate translation unit 2018-02-13 15:59:48 +01:00
mem_encrypt_boot.S x86/mm: Encrypt the initrd earlier for BSP microcode update 2018-01-16 01:50:59 +01:00
mem_encrypt_identity.c x86/mm/encrypt: Simplify sme_pgtable_calc() 2018-02-13 15:59:49 +01:00
mm_internal.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
mmap.c x86/mm: Limit mmap() of /dev/mem to valid physical addresses 2017-11-16 12:49:48 +01:00
mmio-mod.c x86: do not use print_symbol() 2018-01-05 15:23:01 +01:00
mpx.c Merge branch 'x86/mpx' into x86/asm, to pick up dependent commits 2017-11-08 10:55:48 +01:00
numa.c Merge branch 'x86/boot' into x86/mm, to avoid conflict 2017-04-11 08:56:05 +02:00
numa_32.c x86/mm/32: Set the '__vmalloc_start_set' flag in initmem_init() 2017-05-09 08:12:27 +02:00
numa_64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
numa_emulation.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
numa_internal.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pageattr-test.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pageattr.c kmemcheck: stop using GFP_NOTRACK and SLAB_NOTRACK 2017-11-15 18:21:04 -08:00
pat.c x86/mm: add a function to check if a pfn is UC/UC-/WC 2018-01-16 16:40:09 +01:00
pat_internal.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pat_rbtree.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pf_in.c x86/mm: Audit and remove any unnecessary uses of module.h 2016-07-14 13:04:20 +02:00
pf_in.h
pgtable.c Merge branch 'x86-pti-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2017-12-29 17:02:49 -08:00
pgtable_32.c x86/cpu_entry_area: Move it out of the fixmap 2017-12-22 20:13:05 +01:00
physaddr.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
physaddr.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pkeys.c x86/fpu: Rename fpu::fpstate_active to fpu::initialized 2017-09-26 09:43:36 +02:00
pti.c x86/mm/pti: Remove dead logic in pti_user_pagetable_walk*() 2018-01-08 17:42:13 +01:00
setup_nx.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
srat.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
testmmiotrace.c Annotate hardware config module parameters in arch/x86/mm/ 2017-04-04 16:54:21 +01:00
tlb.c x86/mm: Replace compile-time checks for 5-level paging with runtime-time checks 2018-02-16 10:48:49 +01:00