Commit Graph

15 Commits

Author SHA1 Message Date
Quentin Perret cd195bc477 kbuild: split adjust_autoksyms.sh in two parts
In order to prepare the ground for a build-time optimization, split
adjust_autoksyms.sh into two scripts: one that generates autoksyms.h
based on all currently available information (whitelist, and .mod
files), and the other to inspect the diff between two versions of
autoksyms.h and trigger appropriate rebuilds.

Acked-by: Nicolas Pitre <nico@fluxnic.net>
Tested-by: Matthias Maennich <maennich@google.com>
Reviewed-by: Matthias Maennich <maennich@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
2020-03-03 20:49:21 +09:00
Quentin Perret 1518c633df kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS
CONFIG_TRIM_UNUSED_KSYMS currently removes all unused exported symbols
from ksymtab. This works really well when using in-tree drivers, but
cannot be used in its current form if some of them are out-of-tree.

Indeed, even if the list of symbols required by out-of-tree drivers is
known at compile time, the only solution today to guarantee these don't
get trimmed is to set CONFIG_TRIM_UNUSED_KSYMS=n. This not only wastes
space, but also makes it difficult to control the ABI usable by vendor
modules in distribution kernels such as Android. Being able to control
the kernel ABI surface is particularly useful to ship a unique Generic
Kernel Image (GKI) for all vendors, which is a first step in the
direction of getting all vendors to contribute their code upstream.

As such, attempt to improve the situation by enabling users to specify a
symbol 'whitelist' at compile time. Any symbol specified in this
whitelist will be kept exported when CONFIG_TRIM_UNUSED_KSYMS is set,
even if it has no in-tree user. The whitelist is defined as a simple
text file, listing symbols, one per line.

Acked-by: Jessica Yu <jeyu@kernel.org>
Acked-by: Nicolas Pitre <nico@fluxnic.net>
Tested-by: Matthias Maennich <maennich@google.com>
Reviewed-by: Matthias Maennich <maennich@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
2020-03-03 20:49:21 +09:00
Masahiro Yamada 60ae1b194b kbuild: remove the first line of *.mod files
The current format of *.mod is like this:

  line 1: directory path to the .ko file
  line 2: a list of objects linked into this module
  line 3: unresolved symbols (only when CONFIG_TRIM_UNUSED_KSYMS=y)

Now that *.mod and *.ko are created in the same directory, the line 1
provides no valuable information. It can be derived by replacing the
extension .mod with .ko. In fact, nobody uses the first line any more.

Cut down the first line.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2019-07-18 02:19:31 +09:00
Masahiro Yamada b7dca6dd1e kbuild: create *.mod with full directory path and remove MODVERDIR
While descending directories, Kbuild produces objects for modules,
but do not link final *.ko files; it is done in the modpost.

To keep track of modules, Kbuild creates a *.mod file in $(MODVERDIR)
for every module it is building. Some post-processing steps read the
necessary information from *.mod files. This avoids descending into
directories again. This mechanism was introduced in 2003 or so.

Later, commit 551559e13a ("kbuild: implement modules.order") added
modules.order. So, we can simply read it out to know all the modules
with directory paths. This is easier than parsing the first line of
*.mod files.

$(MODVERDIR) has a flat directory structure, that is, *.mod files
are named only with base names. This is based on the assumption that
the module name is unique across the tree. This assumption is really
fragile.

Stephen Rothwell reported a race condition caused by a module name
conflict:

  https://lkml.org/lkml/2019/5/13/991

In parallel building, two different threads could write to the same
$(MODVERDIR)/*.mod simultaneously.

Non-unique module names are the source of all kind of troubles, hence
commit 3a48a91901 ("kbuild: check uniqueness of module names")
introduced a new checker script.

However, it is still fragile in the build system point of view because
this race happens before scripts/modules-check.sh is invoked. If it
happens again, the modpost will emit unclear error messages.

To fix this issue completely, create *.mod with full directory path
so that two threads never attempt to write to the same file.

$(MODVERDIR) is no longer needed.

Since modules with directory paths are listed in modules.order, Kbuild
is still able to find *.mod files without additional descending.

I also killed cmd_secanalysis; scripts/mod/sumversion.c computes MD4 hash
for modules with MODULE_VERSION(). When CONFIG_DEBUG_SECTION_MISMATCH=y,
it occurs not only in the modpost stage, but also during directory
descending, where sumversion.c may parse stale *.mod files. It would emit
'No such file or directory' warning when an object consisting a module is
renamed, or when a single-obj module is turned into a multi-obj module or
vice versa.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Nicolas Pitre <nico@fluxnic.net>
2019-07-18 02:19:31 +09:00
Thomas Gleixner d2912cb15b treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500
Based on 2 normalized pattern(s):

  this program is free software you can redistribute it and or modify
  it under the terms of the gnu general public license version 2 as
  published by the free software foundation

  this program is free software you can redistribute it and or modify
  it under the terms of the gnu general public license version 2 as
  published by the free software foundation #

extracted by the scancode license scanner the SPDX license identifier

  GPL-2.0-only

has been chosen to replace the boilerplate/reference in 4122 file(s).

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Enrico Weigelt <info@metux.net>
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Allison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-06-19 17:09:55 +02:00
Masahiro Yamada 94cf8acc38 kbuild: source include/config/auto.conf instead of ${KCONFIG_CONFIG}
As commit 423a8155fa ("kbuild: Fix reading of .config in
link-vmlinux.sh") addressed, some shells fail to perform '.' if
${KCONFIG_CONFIG} does not contain a slash at all.

Instead, we can source include/config/auto.conf, which obviously
contain slashes, and we do not expect its file path overridden by
a user. Perhaps, the performance might be slightly better since
unset CONFIG options are stripped from include/config/auto.conf.

scripts/setlocalversion already works this way.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2019-03-14 02:39:11 +09:00
Masahiro Yamada 704db5433f kbuild: remove CONFIG_HAVE_UNDERSCORE_SYMBOL_PREFIX
CONFIG_HAVE_UNDERSCORE_SYMBOL_PREFIX was selected by BLACKFIN, METAG.
They were removed by commit 4ba66a9760 ("arch: remove blackfin port"),
commit bb6fb6dfcc ("metag: Remove arch/metag/"), respectively.

No more architecture enables CONFIG_HAVE_UNDERSCORE_SYMBOL_PREFIX.
Clean up the rest of scripts, and remove the Kconfig entry.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Sam Ravnborg <sam@ravnborg.org>
2018-05-17 22:44:57 +09:00
Linus Torvalds 3b24b83763 Kbuild updates for v4.17
- add a shell script to get Clang version
 
 - improve portability of build scripts
 
 - drop always-enabled CONFIG_THIN_ARCHIVE and remove unused code
 
 - rename built-in.o which is now thin archive to built-in.a
 
 - process clean/build targets one by one to get along with -j option
 
 - simplify ld-option
 
 - improve building with CONFIG_TRIM_UNUSED_KSYMS
 
 - define KBUILD_MODNAME even for objects shared among multiple modules
 
 - avoid linking multiple instances of same objects from composite objects
 
 - move <linux/compiler_types.h> to c_flags to include it only for C files
 
 - clean-up various Makefiles
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQIcBAABAgAGBQJaw6eWAAoJED2LAQed4NsGrK8QAJmbYg83TTNoOgQRK/7Lg+sj
 KL1+RGFxmdHRVOqG5n18L7Q4LmTD19tUFNQImrQTTrKrbH2vbMSTF2PfzdmDRwMz
 R5vW5+wsagfhSttOce/GR4p9+bM9XEclzEa3liqNVQxijOFXmkV14pn0x5anYfeB
 ABthxFFHcVn3exP/q3lmq048x1yNE71wUU5WQIWf6V/ZKf+++wQU8r7HpnATWYeO
 vtf8gZq+xyLLjhxoJF6n6olSZXI7Yhz4jV2G68/VroS312AUFWPogK+cSshWGlSw
 zGixM1q55oj9CXjZ37nR6pTzQhSZLf/uHX5beatlpeoJ4Hho6HlIABvx2oEnat7b
 o5RW64RB0gVJqlYZdKxL29HNrovr9tlWPTaIPGFRvWDpl3c1w+rMKXE+5hwu8jMJ
 2jgxd5FZCgBaDsAKojmeQR7PAo2ffAdUO0Dj/SuAaMOpuHWHcnJk9kIN2PUrC+Sf
 d/H2soT9x+60KbQmtCEo5VfEN8bvNP3+ZSnadEG/gRN2IIa5FZAUQykU+i50gAvj
 tuKAokdRGZHvXM+buYFBfN6RbhVCXzBF/fAG7r37QVR2u1zaUszmgFOUqERhTQfm
 RNnyeAs9G9rljtna/AD7cIOdKTg8oETPISxt8Y6EzNMpI8PhF0aGoxso3yD19oH1
 M+fq55RigsR48Kic40hY
 =N5BL
 -----END PGP SIGNATURE-----

Merge tag 'kbuild-v4.17' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild

Pull Kbuild updates from Masahiro Yamada:

 - add a shell script to get Clang version

 - improve portability of build scripts

 - drop always-enabled CONFIG_THIN_ARCHIVE and remove unused code

 - rename built-in.o which is now thin archive to built-in.a

 - process clean/build targets one by one to get along with -j option

 - simplify ld-option

 - improve building with CONFIG_TRIM_UNUSED_KSYMS

 - define KBUILD_MODNAME even for objects shared among multiple modules

 - avoid linking multiple instances of same objects from composite
   objects

 - move <linux/compiler_types.h> to c_flags to include it only for C
   files

 - clean-up various Makefiles

* tag 'kbuild-v4.17' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild: (29 commits)
  kbuild: get <linux/compiler_types.h> out of <linux/kconfig.h>
  kbuild: clean up link rule of composite modules
  kbuild: clean up archive rule of built-in.a
  kbuild: remove partial section mismatch detection for built-in.a
  net: liquidio: clean up Makefile for simpler composite object handling
  lib: zstd: clean up Makefile for simpler composite object handling
  kbuild: link $(real-obj-y) instead of $(obj-y) into built-in.a
  kbuild: rename real-objs-y/m to real-obj-y/m
  kbuild: move modname and modname-multi close to modname_flags
  kbuild: simplify modname calculation
  kbuild: fix modname for composite modules
  kbuild: define KBUILD_MODNAME even if multiple modules share objects
  kbuild: remove unnecessary $(subst $(obj)/, , ...) in modname-multi
  kbuild: Use ls(1) instead of stat(1) to obtain file size
  kbuild: link vmlinux only once for CONFIG_TRIM_UNUSED_KSYMS
  kbuild: move include/config/ksym/* to include/ksym/*
  kbuild: move CONFIG_TRIM_UNUSED_KSYMS code unneeded for external module
  kbuild: restore autoksyms.h touch to the top Makefile
  kbuild: move 'scripts' target below
  kbuild: remove wrong 'touch' in adjust_autoksyms.sh
  ...
2018-04-03 15:51:22 -07:00
Masahiro Yamada fbfa9be990 kbuild: move include/config/ksym/* to include/ksym/*
The idea of using fixdep was inspired by Kconfig, but autoksyms
belongs to a different group.  So, I want to move those touched
files under include/config/ksym/ to include/ksym/.

The directory include/ksym/ can be removed by 'make clean' because
it is meaningless for the external module building.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Nicolas Pitre <nico@linaro.org>
2018-03-26 02:01:23 +09:00
Masahiro Yamada baa16684b0 kbuild: remove wrong 'touch' in adjust_autoksyms.sh
The comment mentions it creates autoksyms.h in case it is missing,
but the actual code touches it when it does exists.

The build system creates it anyway because <linux/export.h> and
<asm-generic/export.h> need it.

The code would not have worked as intended, and people have not
noticed it.  This is a proof that we can simply remove it.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Nicolas Pitre <nico@linaro.org>
2018-03-26 02:01:22 +09:00
Michael Forney 1fe7d2bb24 kbuild: Improve portability of some sed invocations
* Use BREs where EREs aren't necessary.
* Pass -E instead of -r to use EREs. This will be standardized in the
  next POSIX revision[0]. GNU sed supports this since 4.2 (May 2009),
  and busybox since 1.22.0 (Jan 2014).
* Use the [:space:] character class instead of ` \t` in bracket
  expressions. In bracket expressions, POSIX says that <backslash> loses
  its special meaning, so a conforming implementation cannot expand \t
  to <tab>[1].
* In BREs, use interval expressions (\{n,m\}) instead of non-standard
  features like \+ and \?.
* Use a loop instead of -s flag.

There are still plenty of other cases of non-standard sed invocations
(use of ERE features in BREs, in-place editing), but this fixes some
core ones.

[0] http://austingroupbugs.net/view.php?id=528
[1] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03_05

Signed-off-by: Michael Forney <forney@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-03-26 02:01:18 +09:00
Nicolas Pitre 825d487583 kbuild: make scripts/adjust_autoksyms.sh robust against timestamp races
Some filesystems have timestamps with coarse precision that may allow
for a recently built object file to have the same timestamp as the
updated time on one of its dependency files. When that happens, the
object file doesn't get rebuilt as it should.

This is especially the case on filesystems that don't have sub-second
time precision, such as ext3 or Ext4 with 128B inodes.

Let's prevent that by making sure updated dependency files have a newer
timestamp than the first file we created (i.e. autoksyms.h.tmpnew).

Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com>
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Tested-by: Thomas Lindroth <thomas.lindroth@gmail.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-03-21 18:18:10 +09:00
Nicolas Pitre d073472ac2 kbuild: fix scripts/adjust_autoksyms.sh* for the no modules case
When CONFIG_TRIM_UNUSED_KSYMS=y and no modules are actually selected,
the adjust_autoksyms.sh script fails with:

sed: can't read .tmp_versions/*.mod: No such file or directory

Let's cope with that case gracefully.

Signed-off-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-12-11 22:30:10 +01:00
Nicolas Pitre a7c65b9729 kbuild: fix adjust_autoksyms.sh for modules that need only one symbol
When only one symbol was listed and therefore the line didn't contain
any space to separate multiple symbols, that symbol got ignored.

Reported-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-05-10 17:25:29 +02:00
Nicolas Pitre 23121ca2b5 kbuild: create/adjust generated/autoksyms.h
Given the list of exported symbols needed by all modules, we can create
a header file containing preprocessor defines for each of those symbols.
Also, when some symbols are added and/or removed from the list, we can
update the time on the corresponding files used as build dependencies for
those symbols. And finally, if any symbol did change state, the
corresponding source files must be rebuilt.

The insertion or removal of an EXPORT_SYMBOL() entry within a module may
create or remove the need for another exported symbol.  This is why this
operation has to be repeated until the list of needed exported symbols
becomes stable. Only then the final kernel and modules link take place.

Signed-off-by: Nicolas Pitre <nico@linaro.org>
Acked-by: Rusty Russell <rusty@rustcorp.com.au>
2016-03-29 16:30:57 -04:00