In the follow-up of commit fb3041d61f ("kbuild: fix SIGPIPE error
message for AR=gcc-ar and AR=llvm-ar"), Kees Cook pointed out that
tools should _not_ catch their own SIGPIPEs [1] [2].
Based on his feedback, LLVM was fixed [3].
However, Python's default behavior is to show noisy bracktrace when
SIGPIPE is sent. So, scripts written in Python are basically in the
same situation as the buggy llvm tools.
Example:
$ make -s allnoconfig
$ make -s allmodconfig
$ scripts/diffconfig .config.old .config | head -n1
-ALIX n
Traceback (most recent call last):
File "/home/masahiro/linux/scripts/diffconfig", line 132, in <module>
main()
File "/home/masahiro/linux/scripts/diffconfig", line 130, in main
print_config("+", config, None, b[config])
File "/home/masahiro/linux/scripts/diffconfig", line 64, in print_config
print("+%s %s" % (config, new_value))
BrokenPipeError: [Errno 32] Broken pipe
Python documentation [4] notes how to make scripts die immediately and
silently:
"""
Piping output of your program to tools like head(1) will cause a
SIGPIPE signal to be sent to your process when the receiver of its
standard output closes early. This results in an exception like
BrokenPipeError: [Errno 32] Broken pipe. To handle this case,
wrap your entry point to catch this exception as follows:
import os
import sys
def main():
try:
# simulate large output (your code replaces this loop)
for x in range(10000):
print("y")
# flush output here to force SIGPIPE to be triggered
# while inside this try block.
sys.stdout.flush()
except BrokenPipeError:
# Python flushes standard streams on exit; redirect remaining output
# to devnull to avoid another BrokenPipeError at shutdown
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno())
sys.exit(1) # Python exits with error code 1 on EPIPE
if __name__ == '__main__':
main()
Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid
BrokenPipeError. Doing that would cause your program to exit
unexpectedly whenever any socket connection is interrupted while
your program is still writing to it.
"""
Currently, tools/perf/scripts/python/intel-pt-events.py seems to be the
only script that fixes the issue that way.
tools/perf/scripts/python/compaction-times.py uses another approach
signal.signal(signal.SIGPIPE, signal.SIG_DFL) but the Python
documentation clearly says "Don't do it".
I cannot fix all Python scripts since there are so many.
I fixed some in the scripts/ directory.
[1]: https://lore.kernel.org/all/202211161056.1B9611A@keescook/
[2]: https://github.com/llvm/llvm-project/issues/59037
[3]: 4787efa380
[4]: https://docs.python.org/3/library/signal.html#note-on-sigpipe
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
When parsing Kconfig files to find symbol definitions and references,
lines after a 'help' line are skipped until a new config definition
starts.
However, Kconfig statements can actually be after a help section, as
long as these have shallower indentation. These are skipped by the
parser.
This means that symbols referenced in this kind of statements are
ignored by this function and thus are not considered undefined
references in case the symbol is not defined.
Remove the 'skip' logic entirely, as it is not needed if we just use the
STMT regex to find the end of help lines.
However, this means that keywords that appear as part of the help
message (i.e. with the same indentation as the help lines) it will be
considered as a reference/definition. This can happen now as well, but
only with REGEX_KCONFIG_DEF lines. Also, the keyword must have a SYMBOL
after it, which probably means that someone referenced a config in the
help so it seems like a bonus :)
The real solution is to keep track of the indentation when a the first
help line in encountered and then handle DEF and STMT lines only if the
indentation is shallower.
Signed-off-by: Ariel Marcovitch <arielmarcovitch@gmail.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
As opposed to the --diff option, --commit can get ref names instead of
commit hashes.
When using the --commit option, the script resets the working directory
to the commit before the given ref, by adding '~' to the end of the ref.
However, the 'HEAD' ref is relative, and so when the working directory
is reset to 'HEAD~', 'HEAD' points to what was 'HEAD~'. Then when the
script resets to 'HEAD' it actually stays in the same commit. In this
case, the script won't report any cases because there is no diff between
the cases of the two refs.
Prevent the user from using HEAD refs.
A better solution might be to resolve the refs before doing the
reset, but for now just disallow such refs.
Signed-off-by: Ariel Marcovitch <arielmarcovitch@gmail.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
It seems like the implementation of the --ignore option is broken.
In check_symbols_helper, when going through the list of files, a file is
added to the list of source files to check if it matches the ignore
pattern. Instead, as stated in the comment below this condition, the
file should be added if it doesn't match the pattern.
This means that when providing an ignore pattern, the only files that
will be checked will be the ones we want the ignore, in addition to the
Kconfig files that don't match the pattern (the check in
parse_kconfig_files is done right)
Signed-off-by: Ariel Marcovitch <arielmarcovitch@gmail.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Based on 1 normalized pattern(s):
licensed under the terms of the gnu gpl license version 2
extracted by the scancode license scanner the SPDX license identifier
GPL-2.0-only
has been chosen to replace the boilerplate/reference in 62 file(s).
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Allison Randal <allison@lohutok.net>
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Richard Fontana <rfontana@redhat.com>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190527070033.929121379@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Support the new imply statement in Kconfig. The imply statement has
been added by commit 237e3ad0f1 ("Kconfig: Introduce the "imply"
keyword") and is a weak version of a select, but the target symbol can
still be turned off.
Signed-off-by: Andreas Ziegler <andreas.ziegler@fau.de>
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Don't sort the list of string-similar Kconfig symbols alphabetically to
preserve the correct order of string similarity.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Support git's "^" syntax for diffing two commits, for instance via
"--diff HEAD^^^..HEAD".
Signed-off-by: Michael Ellermann <mpe@ellerman.id.au>
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Splitting a command string could lead to unintended arguments. Use an
argument list in the execute() function instead.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
'symbol' and 'feature' are used synonymously to refer to Kconfig symbols
(configs, menus, etc.). Use the term 'symbol' to have a consistent
terminology and to make the code more comprehensible.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Fix pylint and pep8 warnings to have a consistent syntax and style.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Replace the deprecated OptionParser with ArgumentParser, as recommended
by pylint.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Python 2 is slowly dying, so port the script to Python 3.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Use subprocess and set shell to False to avoid potential shell
injections.
Reported-by: Bernd Dietzel <tcpip@t-online.de>
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Only print the ANSI colour escape codes if stdout is a TTY. Useful if
redirecting output to a file or piping to another script.
Also add a new option, --no-color, if the user wants to disable colour
output for whatever reason.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Acked-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Fix a typo in the help message for the -d parameter by removing one 'm'.
Signed-off-by: Andreas Ziegler <andreas.ziegler@fau.de>
Acked-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Add support to find string-similar symbols. When option --sim SYM is
specified, checkkconfigsymbols.py will print at most 10 symbols defined
in Kconfig that are string similar to SYM in the following format:
Similar symbols: $COMMA_SEPARATED_LIST_OF_SYMBOLS
Note, if no similar symbols are found it is indicated as follows:
Similar symbols: no similar symbols found
Since the implemented functionality is also useful when searching the
entire source or when diffing two commits, a list of similar symbols is
printed unconditionally with the other data. In order to make the
output more readable, the format now looks as follows:
$UNDEFINED_SYMBOL
Referencing files: $COMMA_SEPARATED_LIST_OF_FILES
Similar symbols: $COMMA_SEPARATED_LIST_OF_SYMBOLS
[Optional with '--find']
Commits changing symbol:
- $COMMIT_1_HASH ("$COMMIT_1_MESSAGE")
- $COMMIT_2_HASH ("$COMMIT_2_MESSAGE")
or
- no commit found
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Distribute the parsing of source and Kconfig files on all available
cores to speed up processing.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Until now, checkkonfigsymbols.py did not check default statements for
references on missing Kconfig symbols (i.e., undefined Kconfig options).
Hence, add support to parse and check the Kconfig default statement.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Color output to make it more readable. Symbols will be printed yellow,
relevant commits (see --find) red.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Acked-by: Stefan Hengelein <stefan.hengelein@fau.de>
Acked-by: Andreas Ruprecht <andreas.ruprecht@fau.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Add option -f/--find to find relevant commits when using the --diff
option. --find is useful in case a user wants to check commits that
potentially cause a Kconfig symbol to be missing. This is done via 'git
log -G $SYMBOL' (i.e., to get a list of commits that change $SYMBOL).
The relevant commits are printed below the "SYMBOL\tFILES" line,
followed by an empty line to increase readability.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Acked-by: Stefan Hengelein <stefan.hengelein@fau.de>
Acked-by: Andreas Ruprecht <andreas.ruprecht@fau.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Some more recent distributions set the default interpreter to python3,
causing the script to break since it's written for python2.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Sometimes a user might be interested to filter certain reports (e.g.,
the many defconfigs). Now, this can be achieved by specifying a Python
regex with -i / --ignore.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Commit b1a3f24348 ("checkkconfigsymbols.py: make it Git aware")
mistakenly removed to print undefined Kconfig symbols in alphabetical
order. Furthermore, the script does not print anything anymore when the
entire tree is checked (i.e., when no commit is specified).
This patch restores the sorted output and adds the missing print for the
default case. Additionally, the file lists are now sorted as well which
(a) makes it easier to read and (b) makes the output deterministic.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
The script now supports to check a specified commit or a specified range
of commits (i.e., commit1..commit2). Developers and maintainers are
encouraged to use this functionality before sending or merging patches
to avoid potential bugs and to keep the code, documentation, etc. clean.
This patch adds the following options to the script:
-c COMMIT, --commit=COMMIT
Check if the specified commit (hash) introduces
undefined Kconfig symbols.
-d DIFF, --diff=DIFF
Diff undefined symbols between two commits. The input
format bases on Git log's 'commmit1..commit2'.
--force Reset current Git tree even when it's dirty.
Note that the first two options require to 'git reset --hard' the user's
Git tree. This hard reset is necessary to keep the script fast, but it
can lead to the loss of uncommitted data. Hence, the script aborts in
case it is executed in a dirty tree. It won't abort if '--force' is
passed.
If neither -c nor -d is specified, the script defaults to check the
entire local tree (i.e., the previous behavior).
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Recent changes to the build system of tools suggest to filter reports
for the entire tools directory. Various C preprocessor identifiers are
prefixed with CONFIG_ but are NOT defined in Kconfig but in Makefiles in
the tools directory. Such identifiers are false positives for most static
analysis tools (i.e., scripts/checkkconfigsymbols.py) since the CONFIG_
prefix and the _MODULE suffix is reserved for Kconfig features in CPP
and Make syntax.
Signed-off-by: Valentin Rothberg <Valentin.Rothberg@lip6.fr>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch improves the detection of defects by updating the
regular expression to find Kconfig identifiers in the source
code, and fixes some cases of false positives. The following
changes are made:
- improve regex to find Kconfig identifiers in the source
- exclude .log files from analysis
- improve filtering of false positives (e.g, CONFIG_XXX)
- change output format from (feature:\tlist) to (feature\tlist)
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
The scripts/checkkconfigsymbols.sh script searches Kconfig features
in the source code that are not defined in Kconfig. Such identifiers
always evaluate to false and are the source of various kinds of bugs.
However, the shell script is slow and it does not detect such broken
references in Kbuild and Kconfig files (e.g., ``depends on UNDEFINED´´).
Furthermore, it generates false positives. The script is also hard to
read and understand, and is thereby difficult to maintain.
This patch replaces the shell script with an implementation in Python,
which:
(a) detects the same bugs, but does not report previous false positives
(b) additionally detects broken references in Kconfig and all
non-Kconfig files, such as Kbuild, .[cSh], .txt, .sh, defconfig, etc.
(c) is up to 75 times faster than the shell script
(d) only checks files under version control
The new script reduces the runtime on my machine (i7-2620M, 8GB RAM, SSD)
from 3m47s to 0m3s, and reports 938 broken references in Linux v3.17-rc1;
419 additional reports of which 16 are located in Kconfig files,
287 in defconfigs, 63 in ./Documentation, 1 in Kbuild.
Moreover, we intentionally include references in comments, which have been
ignored until now. Such comments may be leftovers of features that have
been removed or renamed in Kconfig (e.g., ``#endif /* CONFIG_MPC52xx */´´).
These references can be misleading and should be removed or replaced.
Note that the output format changed from (file list <tab> feature) to
(feature <tab> file list) as it simplifies the detection of the Kconfig
feature for long file lists.
Signed-off-by: Valentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: Stefan Hengelein <stefan.hengelein@fau.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Masahiro Yamada