Commit Graph

46 Commits

Author SHA1 Message Date
Mauro Carvalho Chehab f5b3553b50 scripts: coccicheck: fix troubles on non-English builds
When LANG is not set to English, the logic which checks the
number of CPUs fail, as the messages can be localized, and
the logic at:

    THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd "[:digit:]")

will not get the number of threads per core.

This causes the script to not run properly, as it will produce
a warning:

	$ make coccicheck COCCI=$PWD/scripts/coccinelle/misc/add_namespace.cocci MODE=report drivers/media/
	./scripts/coccicheck: linha 93: [: número excessivo de argumentos

Fix it by forcing LANG=C when calling lscpu.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
2021-05-18 11:09:59 +02:00
Sumera Priyadarsini d8f6e5c6c8 scripts: coccicheck: Correct usage of make coccicheck
The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the
error:
        ./scripts/coccicheck: line 65: -1: shift count out of range

This happens because every time the C variable is specified,
the shell arguments need to be "shifted" in order to take only
the last argument, which is the C file to test. These shell arguments
mostly comprise flags that have been set in the Makefile. However,
when coccicheck is specified in the make command as a rule, the
number of shell arguments is zero, thus passing the invalid value -1
to the shift command, resulting in an error.

Modify coccicheck to print correct usage of make coccicheck so as to
avoid the error.

Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
2020-12-24 12:59:43 +01:00
Masahiro Yamada 33114c4359 kbuild: do not use scripts/ld-version.sh for checking spatch version
scripts/ld-version.sh was, as its file name implies, originally intended
for the GNU ld version, but is (ab)used for the spatch version too.

Use 'sort -CV' for the version comparison, then coccicheck does not need
to use scripts/ld-version.sh. Fix nsdeps as well.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
2020-12-12 18:31:29 +01:00
Sumera Priyadarsini c5864560d9 scripts: coccicheck: Change default condition for parallelism
Currently, Coccinelle uses at most one thread per core by default in
machines with more than 2 hyperthreads. However, for systems with only 4
hyperthreads, this does not improve performance.

Modify coccicheck to use all available threads in machines with
upto 4 hyperthreads.

Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
2020-10-12 10:37:56 +02:00
Sumera Priyadarsini 2f324dd8ab scripts: coccicheck: Add quotes to improve portability
While fetching the number of threads per core with lscpu,
the [:digit:] set is used for translation of digits from 0-9.
However, using [:digit:] instead of "[:digit:]" does not seem
to work uniformly for some shell types and configurations
(such as zsh).

Therefore, modify coccicheck to use double quotes around the
[:digit:] set for uniformity and better portability.

Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
2020-10-12 10:32:40 +02:00
Sumera Priyadarsini e16a7c47d5 scripts: coccicheck: Change default value for parallelism
By default, coccicheck utilizes all available threads to implement
parallelisation. However, when all available threads are used,
a decrease in performance is noted. The elapsed time is  minimum
when at most one thread per core is used.

For example, on benchmarking the semantic patch kfree.cocci for
usb/serial using hyperfine, the outputs obtained for J=5 and J=2
are 1.32 and 1.90 times faster than those for J=10 and J=9
respectively for two separate runs. For the larger drivers/staging
directory, minimium elapsed time is obtained for J=3 which is 1.86
times faster than that for J=12. The optimal J value does not
exceed 6 in any of the test runs. The benchmarks are run on a machine
with 6 cores, with 2 threads per core, i.e, 12 hyperthreads in all.

To improve performance, modify coccicheck to use at most only
one thread per core by default.

Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
2020-09-27 22:52:59 +02:00
Sumera Priyadarsini a19d135834 scripts: coccicheck: Improve error feedback when coccicheck fails
Currently, coccicheck fails with only the message "coccicheck failed"
and the error code for the failure. To obtain the error logs,
one needs to specify a debug file using the DEBUG_FILE option.

Modify coccicheck to display error logs when it crashes unless
DEBUG_FILE is set, in which case, the error logs are stored in
the specified debug file.

Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
2020-09-13 22:13:01 +02:00
Sumera Priyadarsini 4b2bd20c35 scripts: coccicheck: Add chain mode to list of modes
This patch adds chain mode to the list of available modes in coccicheck.

Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
2020-08-21 22:34:49 +02:00
Denis Efremov 512ddf7d7d coccicheck: return proper error code on fail
If coccicheck fails, it should return an error code distinct from zero
to signal about an internal problem. Current code instead of exiting with
the tool's error code returns the error code of 'echo "coccicheck failed"'
which is almost always equals to zero, thus failing the original intention
of alerting about a problem. This patch fixes the code.

Found by Linux Driver Verification project (linuxtesting.org).

Signed-off-by: Denis Efremov <efremov@linux.com>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-08-14 08:58:56 +09:00
Julia Lawall 1640eea35e Coccinelle: coccicheck: fix typo
Correct spelling of "coccinelle".

Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-02-02 00:21:47 +09:00
Masahiro Yamada d7059ca014 coccinelle: fix parallel build with CHECK=scripts/coccicheck
The command "make -j8 C=1 CHECK=scripts/coccicheck" produces
lots of "coccicheck failed" error messages.

Julia Lawall explained the Coccinelle behavior as follows:
"The problem on the Coccinelle side is that it uses a subdirectory
with the name of the semantic patch to store standard output and
standard error for the different threads.  I didn't want to use a
name with the pid, so that one could easily find this information
while Coccinelle is running.  Normally the subdirectory is cleaned
up when Coccinelle completes, so there is only one of them at a time.
Maybe it is best to just add the pid.  There is the risk that these
subdirectories will accumulate if Coccinelle crashes in a way such
that they don't get cleaned up, but Coccinelle could print a warning
if it detects this case, rather than failing."

When scripts/coccicheck is used as CHECK tool and -j option is given
to Make, the whole of build process runs in parallel.  So, multiple
processes try to get access to the same subdirectory.

I notice spatch creates the subdirectory only when it runs in parallel
(i.e. --jobs <N> is given and <N> is greater than 1).

Setting NPROC=1 is a reasonable solution; spatch does not create the
subdirectory.  Besides, ONLINE=1 mode takes a single file input for
each spatch invocation, so there is no reason to parallelize it in
the first place.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Julia Lawall <Julia.Lawall@lip6.fr>
2017-11-23 23:12:03 +09:00
Linus Torvalds 2ce079f04d Kbuild misc updates for v4.15
- Clean up and fix RPM package build
 
 - Fix a warning in DEB package build
 
 - Improve coccicheck script
 
 - Improve some semantic patches
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQIcBAABAgAGBQJaDxwTAAoJED2LAQed4NsGtJAQAK1VnxVqcGznoy0CXg7vA93D
 iIstHN1AR1uFJgTEFU7PwKDd7zJ78R1lN0BeZat70RS4pv8YMvCFpjPajMbF0KHz
 zjRARuy+MKbWQVBGg7BjFaFcQqLrYLasIfp3nXDOBoyTxvaqvV/4HtyKISxYnY1n
 3vSoVm/JacwT9KZ3496eV8hZcO2pbpo4PvT5T/WGKzHCnHRjv40kCMj8a+BDLWWD
 LJ5CHXQRmzMTBR8J7OMVbaUl8JgJb/Gl6yAKnfNXVqaxzSMunWMC0cUEXjJ3kxEf
 wmthF99p4Og8Fe+eT5yUzODwPdH/JDqULaMKo1BSuQbB0ck0BRavlRvvTlv7ptKk
 5eoHoxop0GCNgsbhcXhpTQTDLqkGBtLVUD6pjUVGF5mtHIJ1m4GnNJ+4+cf453lr
 szhg0mOATydkOTQYg1oeWACXfoagbSieQQsQx5LIX3HTx3I+RW9FGEB7TltEGuoL
 cE+ijhIZZmFo7wHjcdW2FPV0A+qRYRPW//O2t7Zu/S/uRkrgTmpvIKPtoAfj7y+4
 c74LcsWyVB1TCMtbc2oJUyY6ltDuwTEA9i/FCr+GgaG3OHbmw7YEYewbhV9m4+Yr
 t3jNwJ+6TuxWX23RQPFgalAlf8XMr5MM0cF89Io0BZEIaoWWIGw1YiMxzL0Su61h
 eauy22XUBzZ2ewkcuq3v
 =7aYW
 -----END PGP SIGNATURE-----

Merge tag 'kbuild-misc-v4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild

Pull Kbuild misc updates from Masahiro Yamada:

 - Clean up and fix RPM package build

 - Fix a warning in DEB package build

 - Improve coccicheck script

 - Improve some semantic patches

* tag 'kbuild-misc-v4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
  docs: dev-tools: coccinelle: delete out of date wiki reference
  coccinelle: orplus: reorganize to improve performance
  coccinelle: use exists to improve efficiency
  builddeb: Pass the kernel:debarch substvar to dpkg-genchanges
  Coccinelle: use false positive annotation
  coccinelle: fix verbose message about .cocci file being run
  coccinelle: grep Options and Requires fields more precisely
  Coccinelle: make DEBUG_FILE option more useful
  coccinelle: api: detect identical chip data arrays
  coccinelle: Improve setup_timer.cocci matching
  Coccinelle: setup_timer: improve messages from setup_timer
  kbuild: rpm-pkg: do not force -jN in submake
  kbuild: rpm-pkg: keep spec file until make mrproper
  kbuild: rpm-pkg: fix jobserver unavailable warning
  kbuild: rpm-pkg: replace $RPM_BUILD_ROOT with %{buildroot}
  kbuild: rpm-pkg: fix build error when CONFIG_MODULES is disabled
  kbuild: rpm-pkg: refactor mkspec with here doc
  kbuild: rpm-pkg: clean up mkspec
  kbuild: rpm-pkg: install vmlinux.bz2 unconditionally
  kbuild: rpm-pkg: remove ppc64 specific image handling
2017-11-17 17:51:33 -08:00
Masahiro Yamada cd1af7cfbb coccinelle: fix verbose message about .cocci file being run
If you run coccicheck with V=1 and COCCI=, you will see a strange
path to the semantic patch file.  For example, run the following:

$ make V=1 COCCI=scripts/coccinelle/free/kfree.cocci coccicheck
  [ snip ]
 The semantic patch that makes this report is available
 in scriptcoccinelle/free/kfree.cocci.

Notice "s/" was dropped from "scripts/coccinelle/free/kfree.cocci".

When running coccicheck without O=, $srctree is expanded to ".", which
represents one arbitrary character in the regular expression.  Using
sed is not a good choice here.  Strip $srctree/ simply without sed.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
2017-11-14 23:19:04 +09:00
Masahiro Yamada e0be348e4d coccinelle: grep Options and Requires fields more precisely
Currently, the required version for badzero.cocci is picked up from
its "Comments:" line since it contains the word "Requires".

Surprisingly, ld-version.sh can extract the version number from the
string "Requires Coccinelle version 1.0.0-rc20 or later", but this
expectation is fragile.  Fix the .cocci file.  I removed "-rc20"
because ld-version.sh cannot handle it.

Make the coccicheck script to see exact patterns for "Options:" and
"Requires:" in order to avoid accidental matching to what just happens
to appear in comment lines.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
2017-11-14 23:19:04 +09:00
Julia Lawall 9ed07ada0e Coccinelle: make DEBUG_FILE option more useful
Make coccicheck checked for the existence of DEBUG_FILE on each semantic
patch, and bailed if it already existed.  This meant that DEBUG_FILE was
useless for checking more than one semantic patch at a time.  Now the check
is moved to the start of make coccicheck, and the 2> is changed to a 2>> to
append to the file on each semantic patch.  Furthermore, the spatch command
that is run for each semantic patch is also added to the DEBUG_FILE, to
make clear what each stdout trace corresponds to.

Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2017-11-14 23:19:03 +09:00
Greg Kroah-Hartman b24413180f License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.

By default all files without license information are under the default
license of the kernel, which is GPL version 2.

Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier.  The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.

This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.

How this work was done:

Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
 - file had no licensing information it it.
 - file was a */uapi/* one with no licensing information in it,
 - file was a */uapi/* one with existing licensing information,

Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.

The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne.  Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.

The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed.  Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.

Criteria used to select files for SPDX license identifier tagging was:
 - Files considered eligible had to be source code files.
 - Make and config files were included as candidates if they contained >5
   lines of source
 - File already had some variant of a license header in it (even if <5
   lines).

All documentation files were explicitly excluded.

The following heuristics were used to determine which SPDX license
identifiers to apply.

 - when both scanners couldn't find any license traces, file was
   considered to have no license information in it, and the top level
   COPYING file license applied.

   For non */uapi/* files that summary was:

   SPDX license identifier                            # files
   ---------------------------------------------------|-------
   GPL-2.0                                              11139

   and resulted in the first patch in this series.

   If that file was a */uapi/* path one, it was "GPL-2.0 WITH
   Linux-syscall-note" otherwise it was "GPL-2.0".  Results of that was:

   SPDX license identifier                            # files
   ---------------------------------------------------|-------
   GPL-2.0 WITH Linux-syscall-note                        930

   and resulted in the second patch in this series.

 - if a file had some form of licensing information in it, and was one
   of the */uapi/* ones, it was denoted with the Linux-syscall-note if
   any GPL family license was found in the file or had no licensing in
   it (per prior point).  Results summary:

   SPDX license identifier                            # files
   ---------------------------------------------------|------
   GPL-2.0 WITH Linux-syscall-note                       270
   GPL-2.0+ WITH Linux-syscall-note                      169
   ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause)    21
   ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)    17
   LGPL-2.1+ WITH Linux-syscall-note                      15
   GPL-1.0+ WITH Linux-syscall-note                       14
   ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause)    5
   LGPL-2.0+ WITH Linux-syscall-note                       4
   LGPL-2.1 WITH Linux-syscall-note                        3
   ((GPL-2.0 WITH Linux-syscall-note) OR MIT)              3
   ((GPL-2.0 WITH Linux-syscall-note) AND MIT)             1

   and that resulted in the third patch in this series.

 - when the two scanners agreed on the detected license(s), that became
   the concluded license(s).

 - when there was disagreement between the two scanners (one detected a
   license but the other didn't, or they both detected different
   licenses) a manual inspection of the file occurred.

 - In most cases a manual inspection of the information in the file
   resulted in a clear resolution of the license that should apply (and
   which scanner probably needed to revisit its heuristics).

 - When it was not immediately clear, the license identifier was
   confirmed with lawyers working with the Linux Foundation.

 - If there was any question as to the appropriate license identifier,
   the file was flagged for further research and to be revisited later
   in time.

In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.

Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights.  The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.

Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.

In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.

Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
 - a full scancode scan run, collecting the matched texts, detected
   license ids and scores
 - reviewing anything where there was a license detected (about 500+
   files) to ensure that the applied SPDX license was correct
 - reviewing anything where there was no detection but the patch license
   was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
   SPDX license was correct

This produced a worksheet with 20 files needing minor correction.  This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.

These .csv files were then reviewed by Greg.  Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected.  This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.)  Finally Greg ran the script using the .csv files to
generate the patches.

Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-02 11:10:55 +01:00
Markus Elfring 1e01892e7a scripts/coccicheck: Update reference for the corresponding documentation
Use the current name (in a comment at the beginning of this script) for
the file which was converted to the documentation format "reStructuredText"
in August 2016.

Fixes: 4b9033a334 ("docs: sphinxify coccinelle.txt and add it to dev-tools")
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-10-11 09:50:43 +02:00
Luis R. Rodriguez c100d537b9 coccicheck: refer to Documentation/coccinelle.txt and wiki
Refer to the Documentation/coccinelle.txt and supplemental documentation
on the wiki:

https://bottest.wiki.kernel.org/coccicheck

This page shall always refer to the linux-next iteration of scripts/coccicheck.

v4: only refer to the wiki as supplemental documentation, and also
    update Documentation/coccinelle.txt.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22 12:13:39 +02:00
Luis R. Rodriguez a9e064c004 coccicheck: add support for requring a coccinelle version
Enable Coccinelle SmPL patches to require a specific version of
Coccinelle. In the event that the version does not match we just
inform the user, if the user asked to go through all SmPL patches
we just inform them of the need for a new version of coccinelle for
the SmPL patch and continue on with the rest.

This uses the simple kernel scripts/ld-version.sh to create a weight
on the version provided by spatch. The -dirty attribute is ignored if
supplied, the benefit of scripts/ld-version.sh is it has a long history
and well tested.

While at it, document the // Options stuff as well.

v4: Document // Options and // Requires as well on
    Documentation/coccinelle.txt.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22 12:13:39 +02:00
Luis R. Rodriguez 5c384dba97 coccicheck: replace --very-quiet with --quiet when debugging
When debugging (using --profile or --show-trying) you want to
avoid supressing output,  use --quiet instead. While at it, extend
documentation for SPFLAGS use.

For instance one can use:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck DEBUG_FILE="poo.err" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c

Expand Documentation/coccinelle.txt as well.

v4: expand Documentation/coccinelle.txt
v3: rebased, resolve conflicts, expand Documentation/coccinelle.txt
v2: use egrep instead of the *"=--option"* check, this doesn't work for
    disjunctions.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22 12:13:39 +02:00
Luis R. Rodriguez be1fa90066 coccicheck: add support for DEBUG_FILE
Enable to capture stderr via a DEBUG_FILE variable passed to
coccicheck. You can now do:

$ rm -f cocci.err
$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ make coccicheck MODE=report DEBUG_FILE=cocci.err
...
$ cat cocci.err

This will be come more useful once we add support to
use more things which would go into stderr, such as
profiling. That will be done separately in another
commit.

Expand Documentation/coccinelle.txt with details.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22 12:13:39 +02:00
Luis R. Rodriguez c930a1b23b coccicheck: enable parmap support
Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported and use it
only if your number of processors detected is > 1.

If parmap is enabled also enable the load balancing to be dynamic, so
that if a thread finishes early we keep feeding it.

stderr is currently sent to /dev/null, addressing a way to capture
that will be addressed next.

If --jobs is not supported we fallback to the old mechanism.
We expect to deprecate the old mechanism as soon as we can get
confirmation all users are ready.

While at it propagate back into the shell script any coccinelle error
code. When used in serialized mode where all cocci files are run this
also stops processing if an error has occured. This lets us handle some
errors in coccinelle cocci files and if they bail out we should inspect
the errors. This will be more useful later to help annotate coccinelle
version dependency requirements. This will let you run only SmPL files
that your system supports.

Extend Documentation/coccinelle.txt as well.

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

v4:

o expand Documentation/coccinelle.txt to reflect parmap support info
o update commit log to reflect what we actually do now with stderr
o split out DEBUG_FILE use into another patch
o detect number of CPUs and if its 1 then skip parmap support,
  note that if you still support parmap, but have 1 CPU you will
  also go through the new branches, so the old complex multithreaded process
  is skipped as well.

v3:

o move USE_JOBS to avoid being overriden

v2:

o redirect coccinelle stderr to /dev/null by default and
  only if DEBUG_FILE is used do we pass it to a file
o fix typo of paramap/parmap

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22 12:13:39 +02:00
Luis R. Rodriguez 8e826ad52b coccicheck: make SPFLAGS more useful
SPFLAGS is set early, it means that any heuristics done on
coccicheck cannot be overridden currently. Move SPFLAGS
after OPTIONS and set this at the end. This lets you override
any heuristics as coccinelle treats conflicts by only listening
to the last option that makes sense.

v3: this patch was added in the v3 series
v4: Update Documentation/coccinelle.txt explaining how
    SPFLAGS works as well.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22 12:13:39 +02:00
Luis R. Rodriguez 13d9486533 coccicheck: move spatch binary check up
This has no functional changes. This is being done
to enable us to later use spatch binary for some
flag checking for certain features early on.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22 12:13:39 +02:00
Deepa Dinamani 7a2358b381 coccicheck: Allow for overriding spatch flags
Documentation/coccinelle.txt suggests using the SPFLAGS
make variable to pass additional options to spatch.

Reorder the way SPFLAGS is added to FLAGS, to allow
for options in the SPFLAGS to override the default
--very-quiet option.

Similarly, rearrage the FLAGS for org or report mode.
This allows for overriding of the default --no-show-diff
option through SPFLAGS.

Signed-off-by: Deepa Dinamani <deepa.kernel@gmail.com>
Cc: Gilles Muller <Gilles.Muller@lip6.fr>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Acked-by: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-06-20 22:08:13 +02:00
Kees Cook 2552a39fac coccicheck: Fix missing 0 index in kill loop
By default, "seq" counts from 1, but processes were starting counting
from 0, so when interrupted, coccicheck would leave the 0th process
running.

Signed-off-by: Kees Cook <keescook@chromium.org>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
2016-05-19 09:56:59 +02:00
Andrzej Hajda 5b16910857 Coccinelle: fix incorrect -include option transformation
kbuild/gcc uses -include option to include files and -I to provide paths
for #include <> directive. In case of spatch latter option should be
prefixed with two -.

Signed-off-by: Andrzej Hajda <a.hajda@samsung.com>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2015-10-26 21:32:09 +01:00
Nicolas Palix ec97946ed0 Coccinelle: Update information about the minimal version required
The naming convention of options has changed one year ago.
The options have been recently updated in the cocci file
and in scripts/coccicheck. This patch also adds this information
in the documentation.

Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-07-03 22:58:20 +02:00
Nicolas Palix 93f1446849 Coccinelle: Update the options used to the new option scheme
spatch has changed its option scheme.
E.g., --no_show_diff is now --no-show-diff

This patch updates:
 - scripts/coccicheck
 - Semantic patches under scripts/coccinelle/

Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-07-03 22:58:13 +02:00
Kees Cook 90d06a4683 coccicheck: span checks across CPUs
This adds parallelism by default to the "coccicheck" target using
spatch's "-max" and "-index" arguments.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-07-03 15:26:29 +02:00
Nicolas Palix 1f0a6742dd Coccinelle: Make 'report' the default mode
It appears that the 'report' mode is the one always
provided by the semantic patches included in the kernel.
It is thus more natural to select it by default.

The 'chain' mode is however kept and the 'patch' mode
is still the first tried in that case.

Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-06-14 15:26:47 +02:00
Nicolas Palix bad6a4092e Coccinelle: Fix patch output when coccicheck is used with M= and C=
When the M variable is used, the -patch option should be given
to spatch. This patch fixes the case where C is used.

Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-04-08 15:42:03 +02:00
Nicolas Palix ed621cc4a5 Coccinelle: Add support to the SPFLAGS variable
The SPFLAGS variable allows to pass additional options
to spatch, e.g. -use_glimpse.

Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-04-08 15:42:03 +02:00
Nicolas Palix 9e39555077 Coccinelle: Cleanup the setting of the FLAGS and OPTIONS variables
The FLAGS variable is factorized independently of the ONLINE mode.
The OPTIONS variable is now based on LINUXINCLUDE and explicit
includes are thus removed.

The format of the -I option differs between gcc and spatch.
The COCCIINCLUDE is used to adapt the format. This rewritting
needs bash.

Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-04-08 15:42:02 +02:00
Nicolas Palix 35d88a3871 Coccinelle: Restore coccicheck verbosity in ONLINE mode (C=1 or C=2)
A recent patch have introduce the VERBOSE variable and comments
now depend on it. However, the message printed for each cocci file
such not be printed when the ONLINE mode is active, whatever is
the value of VERBOSE.

Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-04-08 15:42:02 +02:00
Bernd Schubert 5303265a48 coccicheck: Allow to show the executed command line
On my system one of the tests failed with
"Fatal error: exception Failure("No OCaml compiler found! Install either ocamlopt or ocamlopt.opt")".

Investigating such issues is easier if the executed command line is
being shown.

Signed-off-by: Bernd Schubert <bernd.schubert@itwm.fraunhofer.de>
CC: Julia Lawall <Julia.Lawall@lip6.fr>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-02-22 14:25:44 +01:00
Bernd Schubert 26e5672091 coccicheck: Allow the user to give a V= (verbose) argument
Do not run with verbosity on/off depending on the ONLINE variable,
which gets set with C=1 or C=2, but allow the user to set the
verbosity using kernel default make V= paramemter.
Verbosity is off by default now.

Signed-off-by: Bernd Schubert <bernd.schubert@itwm.fraunhofer.de>
CC: Julia Lawall <Julia.Lawall@lip6.fr>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-02-22 11:39:13 +01:00
Nicolas Palix c05cd6ddb6 coccicheck: Add the rep+ctxt mode
This adds a 'rep+ctxt' mode which prints the warning
message followed by the context.

Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2012-10-11 15:11:12 +02:00
Greg Dietsche 42f1c01b79 coccicheck: change handling of C={1,2} when M= is set
This patch reverts a portion of d0bc1fb4 so that coccicheck will
work properly when C=1 or C=2.

Reported-and-tested-by: Brice Goglin <Brice.Goglin@inria.fr>
Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2012-02-24 23:50:19 +01:00
Greg Dietsche d0bc1fb467 coccicheck: add M= option to control which dir is processed
Examples:
	make coccicheck M=drivers/net/wireless/
	make coccicheck SUBDIRS=drivers/net/wireless/

Version 2:
	fix patch file names when using M=
	tell coccinelle where the include files are

Version 3:
	Add second include option to support out of tree development
	Fix error message

Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2012-01-14 22:25:40 +01:00
Nicolas Palix 062c1825a5 Coccinelle: Add contextual message
Change the message displayed to the user according
to the current mode used.

Fix trailing white-space and spelling

Signed-off-by: Nicolas Palix <npalix.work@gmail.com>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-10-28 00:32:23 +02:00
Nicolas Palix 03ee0c42a7 Coccinelle: Use the -no_show_diff option for org and report mode
This allows to write the semantic patches with code sharing
for the matching parts.

Signed-off-by: Nicolas Palix <npalix.work@gmail.com>
Signed-off-by: Julia Lawall <julia@diku.dk>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-10-13 14:26:06 +02:00
Nicolas Palix 2c1160c874 Coccinelle: Add a new mode named 'chain'
spatch now returns -1 when a virtual rule (given with
-D on the command line) is not defined in the semantic patch.

Using this spatch feature, coccicheck is now
tries several modes by default, in the order:
	patch, report, context, org

Signed-off-by: Nicolas Palix <npalix.work@gmail.com>
Signed-off-by: Julia Lawall <julia@diku.dk>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-10-13 14:26:06 +02:00
Nicolas Palix 3c90841767 Coccinelle: Improve user information with a new kind of comment
Improve user information with a new kind of comment
about semantic patch output.

Fix spelling.

Signed-off-by: Nicolas Palix <npalix.work@gmail.com>
Signed-off-by: Julia Lawall <julia@diku.dk>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-10-13 14:26:06 +02:00
Nicolas Palix 1e9dea2a60 Add support for the C variable in the coccicheck script
This patch makes it possible to use the Coccinelle checker with the C
variable of the build system. To check only newly edited code, the
following command may be used:

'make C={1,2} CHECK="scripts/coccicheck"'

This runs every semantic patch in scripts/coccinelle by default. The
COCCI variable may additionally be used to only apply a single
semantic patch.

Signed-off-by: Nicolas Palix <npalix@diku.dk>
Signed-off-by: Julia Lawall <julia@diku.dk>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-06-28 16:44:53 +02:00
Nicolas Palix 74425eee71 Add a target to use the Coccinelle checker
A 'coccicheck' target is added. It can be called with four different
modes. Each one generates a different kind of output, i.e. context,
patch, org, report, according to the corresponding mode to be
activated.

The new target calls the 'coccicheck' front-end in the 'scripts'
directory with the MODE argument. Every SmPL file in the
subdirectories of 'scripts/coccinelle' is then given to the front-end
and applied to the entire source tree.

The four modes behave as follows:

'report' generates a list in the following format:
  file:line:column-column: message

'patch' proposes a fix, when possible.

'context' highlights lines of interest and their context in a
diff-like style. Lines of interest are indicated with '-'.

'org' generates a report in the Org mode format of Emacs.

Signed-off-by: Nicolas Palix <npalix@diku.dk>
Signed-off-by: Julia Lawall <julia@diku.dk>
Acked-by: Sam Ravnborg <sam@ravnborg.org>
Acked-by: Joerg Roedel <joerg.roedel@amd.com>
Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-06-12 00:00:29 +02:00