2021-02-02 06:06:04 +00:00
|
|
|
#!/usr/bin/env python3
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
# SPDX-License-Identifier: GPL-2.0
|
|
|
|
#
|
|
|
|
# Copyright (C) Google LLC, 2018
|
|
|
|
#
|
|
|
|
# Author: Tom Roeder <tmroeder@google.com>
|
|
|
|
#
|
|
|
|
"""A tool for generating compile_commands.json in the Linux kernel."""
|
|
|
|
|
|
|
|
import argparse
|
|
|
|
import json
|
|
|
|
import logging
|
|
|
|
import os
|
|
|
|
import re
|
2020-08-22 14:56:15 +00:00
|
|
|
import subprocess
|
2021-09-08 03:28:48 +00:00
|
|
|
import sys
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
|
|
|
_DEFAULT_OUTPUT = 'compile_commands.json'
|
|
|
|
_DEFAULT_LOG_LEVEL = 'WARNING'
|
|
|
|
|
|
|
|
_FILENAME_PATTERN = r'^\..*\.cmd$'
|
2023-07-19 03:19:12 +00:00
|
|
|
_LINE_PATTERN = r'^savedcmd_[^ ]*\.o := (.* )([^ ]*\.[cS]) *(;|$)'
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
_VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
|
gen_compile_commands: prune some directories
If directories are passed to gen_compile_commands.py, os.walk() traverses
all the subdirectories to search for .cmd files, but we know some of them
are not worth traversing.
Use the 'topdown' parameter of os.walk to prune them.
Documentation about the 'topdown' option of os.walk:
When topdown is True, the caller can modify the dirnames list
in-place (perhaps using del or slice assignment), and walk() will
only recurse into the subdirectories whose names remain in dirnames;
this can be used to prune the search, impose a specific order of
visiting, or even to inform walk() about directories the caller
creates or renames before it resumes walk() again. Modifying
dirnames when topdown is False has no effect on the behavior of
the walk, because in bottom-up mode the directories in dirnames
are generated before dirpath itself is generated.
This commit prunes four directories, .git, Documentation, include, and
tools.
The first three do not contain any C files, so skipping them makes this
script work slightly faster. My main motivation is the last one, tools/
directory.
Commit 6ca4c6d25949 ("gen_compile_commands: do not support .cmd files
under tools/ directory") stopped supporting the tools/ directory.
The current code no longer picks up .cmd files from the tools/
directory.
If you run:
./scripts/clang-tools/gen_compile_commands.py --log_level=INFO
then, you will see several "File ... not found" log messages.
This is expected, and I do not want to support the tools/ directory.
However, without an explicit comment "do not support tools/", somebody
might try to get it back. Clarify this.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Acked-by: Nathan Chancellor <nathan@kernel.org>
2021-02-11 16:11:54 +00:00
|
|
|
# The tools/ directory adopts a different build system, and produces .cmd
|
|
|
|
# files in a different format. Do not support it.
|
|
|
|
_EXCLUDE_DIRS = ['.git', 'Documentation', 'include', 'tools']
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
|
|
|
def parse_arguments():
|
|
|
|
"""Sets up and parses command-line arguments.
|
|
|
|
|
|
|
|
Returns:
|
|
|
|
log_level: A logging level to filter log output.
|
2020-08-22 14:56:12 +00:00
|
|
|
directory: The work directory where the objects were built.
|
2020-08-22 14:56:15 +00:00
|
|
|
ar: Command used for parsing .a archives.
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
output: Where to write the compile-commands JSON file.
|
2020-08-22 14:56:15 +00:00
|
|
|
paths: The list of files/directories to handle to find .cmd files.
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
"""
|
|
|
|
usage = 'Creates a compile_commands.json database from kernel .cmd files'
|
|
|
|
parser = argparse.ArgumentParser(description=usage)
|
|
|
|
|
2020-08-22 14:56:12 +00:00
|
|
|
directory_help = ('specify the output directory used for the kernel build '
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
'(defaults to the working directory)')
|
2020-08-22 14:56:13 +00:00
|
|
|
parser.add_argument('-d', '--directory', type=str, default='.',
|
|
|
|
help=directory_help)
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
2020-08-22 14:56:13 +00:00
|
|
|
output_help = ('path to the output command database (defaults to ' +
|
|
|
|
_DEFAULT_OUTPUT + ')')
|
|
|
|
parser.add_argument('-o', '--output', type=str, default=_DEFAULT_OUTPUT,
|
|
|
|
help=output_help)
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
gen_compile_commands: use choices for --log_levels option
Use 'choices' to check if the given parameter is valid.
I also simplified the help message because, with 'choices', --help
shows the list of valid parameters:
--log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
I started the help message with a lower case, "the level of log ..."
in order to be consistent with the -h option:
-h, --help show this help message and exit
The message "show this help ..." comes from the ArgumentParser library
code, and I do not know how to change it. So, I changed our code.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-22 14:56:10 +00:00
|
|
|
log_level_help = ('the level of log messages to produce (defaults to ' +
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
_DEFAULT_LOG_LEVEL + ')')
|
gen_compile_commands: use choices for --log_levels option
Use 'choices' to check if the given parameter is valid.
I also simplified the help message because, with 'choices', --help
shows the list of valid parameters:
--log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
I started the help message with a lower case, "the level of log ..."
in order to be consistent with the -h option:
-h, --help show this help message and exit
The message "show this help ..." comes from the ArgumentParser library
code, and I do not know how to change it. So, I changed our code.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
2020-08-22 14:56:10 +00:00
|
|
|
parser.add_argument('--log_level', choices=_VALID_LOG_LEVELS,
|
|
|
|
default=_DEFAULT_LOG_LEVEL, help=log_level_help)
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
2020-08-22 14:56:15 +00:00
|
|
|
ar_help = 'command used for parsing .a archives'
|
|
|
|
parser.add_argument('-a', '--ar', type=str, default='llvm-ar', help=ar_help)
|
|
|
|
|
|
|
|
paths_help = ('directories to search or files to parse '
|
|
|
|
'(files should be *.o, *.a, or modules.order). '
|
|
|
|
'If nothing is specified, the current directory is searched')
|
|
|
|
parser.add_argument('paths', type=str, nargs='*', help=paths_help)
|
|
|
|
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
args = parser.parse_args()
|
|
|
|
|
2020-08-22 14:56:13 +00:00
|
|
|
return (args.log_level,
|
|
|
|
os.path.abspath(args.directory),
|
2020-08-22 14:56:14 +00:00
|
|
|
args.output,
|
2020-08-22 14:56:15 +00:00
|
|
|
args.ar,
|
|
|
|
args.paths if len(args.paths) > 0 else [args.directory])
|
2020-08-22 14:56:14 +00:00
|
|
|
|
|
|
|
|
|
|
|
def cmdfiles_in_dir(directory):
|
|
|
|
"""Generate the iterator of .cmd files found under the directory.
|
|
|
|
|
|
|
|
Walk under the given directory, and yield every .cmd file found.
|
|
|
|
|
|
|
|
Args:
|
|
|
|
directory: The directory to search for .cmd files.
|
|
|
|
|
|
|
|
Yields:
|
|
|
|
The path to a .cmd file.
|
|
|
|
"""
|
|
|
|
|
|
|
|
filename_matcher = re.compile(_FILENAME_PATTERN)
|
gen_compile_commands: prune some directories
If directories are passed to gen_compile_commands.py, os.walk() traverses
all the subdirectories to search for .cmd files, but we know some of them
are not worth traversing.
Use the 'topdown' parameter of os.walk to prune them.
Documentation about the 'topdown' option of os.walk:
When topdown is True, the caller can modify the dirnames list
in-place (perhaps using del or slice assignment), and walk() will
only recurse into the subdirectories whose names remain in dirnames;
this can be used to prune the search, impose a specific order of
visiting, or even to inform walk() about directories the caller
creates or renames before it resumes walk() again. Modifying
dirnames when topdown is False has no effect on the behavior of
the walk, because in bottom-up mode the directories in dirnames
are generated before dirpath itself is generated.
This commit prunes four directories, .git, Documentation, include, and
tools.
The first three do not contain any C files, so skipping them makes this
script work slightly faster. My main motivation is the last one, tools/
directory.
Commit 6ca4c6d25949 ("gen_compile_commands: do not support .cmd files
under tools/ directory") stopped supporting the tools/ directory.
The current code no longer picks up .cmd files from the tools/
directory.
If you run:
./scripts/clang-tools/gen_compile_commands.py --log_level=INFO
then, you will see several "File ... not found" log messages.
This is expected, and I do not want to support the tools/ directory.
However, without an explicit comment "do not support tools/", somebody
might try to get it back. Clarify this.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Acked-by: Nathan Chancellor <nathan@kernel.org>
2021-02-11 16:11:54 +00:00
|
|
|
exclude_dirs = [ os.path.join(directory, d) for d in _EXCLUDE_DIRS ]
|
|
|
|
|
|
|
|
for dirpath, dirnames, filenames in os.walk(directory, topdown=True):
|
|
|
|
# Prune unwanted directories.
|
|
|
|
if dirpath in exclude_dirs:
|
|
|
|
dirnames[:] = []
|
|
|
|
continue
|
2020-08-22 14:56:14 +00:00
|
|
|
|
|
|
|
for filename in filenames:
|
|
|
|
if filename_matcher.match(filename):
|
|
|
|
yield os.path.join(dirpath, filename)
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
|
|
|
|
2020-08-22 14:56:15 +00:00
|
|
|
def to_cmdfile(path):
|
|
|
|
"""Return the path of .cmd file used for the given build artifact
|
|
|
|
|
|
|
|
Args:
|
|
|
|
Path: file path
|
|
|
|
|
|
|
|
Returns:
|
|
|
|
The path to .cmd file
|
|
|
|
"""
|
|
|
|
dir, base = os.path.split(path)
|
|
|
|
return os.path.join(dir, '.' + base + '.cmd')
|
|
|
|
|
|
|
|
|
|
|
|
def cmdfiles_for_a(archive, ar):
|
|
|
|
"""Generate the iterator of .cmd files associated with the archive.
|
|
|
|
|
|
|
|
Parse the given archive, and yield every .cmd file used to build it.
|
|
|
|
|
|
|
|
Args:
|
|
|
|
archive: The archive to parse
|
|
|
|
|
|
|
|
Yields:
|
|
|
|
The path to every .cmd file found
|
|
|
|
"""
|
|
|
|
for obj in subprocess.check_output([ar, '-t', archive]).decode().split():
|
|
|
|
yield to_cmdfile(obj)
|
|
|
|
|
|
|
|
|
|
|
|
def cmdfiles_for_modorder(modorder):
|
|
|
|
"""Generate the iterator of .cmd files associated with the modules.order.
|
|
|
|
|
|
|
|
Parse the given modules.order, and yield every .cmd file used to build the
|
|
|
|
contained modules.
|
|
|
|
|
|
|
|
Args:
|
|
|
|
modorder: The modules.order file to parse
|
|
|
|
|
|
|
|
Yields:
|
|
|
|
The path to every .cmd file found
|
|
|
|
"""
|
|
|
|
with open(modorder) as f:
|
|
|
|
for line in f:
|
2022-12-11 13:04:07 +00:00
|
|
|
obj = line.rstrip()
|
|
|
|
base, ext = os.path.splitext(obj)
|
|
|
|
if ext != '.o':
|
|
|
|
sys.exit('{}: module path must end with .o'.format(obj))
|
2020-08-22 14:56:15 +00:00
|
|
|
mod = base + '.mod'
|
2022-06-28 01:23:53 +00:00
|
|
|
# Read from *.mod, to get a list of objects that compose the module.
|
2020-08-22 14:56:15 +00:00
|
|
|
with open(mod) as m:
|
2022-06-28 01:23:53 +00:00
|
|
|
for mod_line in m:
|
|
|
|
yield to_cmdfile(mod_line.rstrip())
|
2020-08-22 14:56:15 +00:00
|
|
|
|
|
|
|
|
2020-08-22 14:56:11 +00:00
|
|
|
def process_line(root_directory, command_prefix, file_path):
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
"""Extracts information from a .cmd line and creates an entry from it.
|
|
|
|
|
|
|
|
Args:
|
|
|
|
root_directory: The directory that was searched for .cmd files. Usually
|
|
|
|
used directly in the "directory" entry in compile_commands.json.
|
|
|
|
command_prefix: The extracted command line, up to the last element.
|
2020-08-22 14:56:11 +00:00
|
|
|
file_path: The .c file from the end of the extracted command.
|
|
|
|
Usually relative to root_directory, but sometimes absolute.
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
|
|
|
Returns:
|
|
|
|
An entry to append to compile_commands.
|
|
|
|
|
|
|
|
Raises:
|
2020-08-22 14:56:11 +00:00
|
|
|
ValueError: Could not find the extracted file based on file_path and
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
root_directory or file_directory.
|
|
|
|
"""
|
|
|
|
# The .cmd files are intended to be included directly by Make, so they
|
|
|
|
# escape the pound sign '#', either as '\#' or '$(pound)' (depending on the
|
|
|
|
# kernel version). The compile_commands.json file is not interepreted
|
|
|
|
# by Make, so this code replaces the escaped version with '#'.
|
|
|
|
prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#')
|
|
|
|
|
2020-08-22 14:56:11 +00:00
|
|
|
# Use os.path.abspath() to normalize the path resolving '.' and '..' .
|
|
|
|
abs_path = os.path.abspath(os.path.join(root_directory, file_path))
|
|
|
|
if not os.path.exists(abs_path):
|
|
|
|
raise ValueError('File %s not found' % abs_path)
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
return {
|
2020-08-22 14:56:11 +00:00
|
|
|
'directory': root_directory,
|
|
|
|
'file': abs_path,
|
|
|
|
'command': prefix + file_path,
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
def main():
|
|
|
|
"""Walks through the directory and finds and parses .cmd files."""
|
2020-08-22 14:56:15 +00:00
|
|
|
log_level, directory, output, ar, paths = parse_arguments()
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
|
|
|
level = getattr(logging, log_level)
|
|
|
|
logging.basicConfig(format='%(levelname)s: %(message)s', level=level)
|
|
|
|
|
|
|
|
line_matcher = re.compile(_LINE_PATTERN)
|
|
|
|
|
|
|
|
compile_commands = []
|
|
|
|
|
2020-08-22 14:56:14 +00:00
|
|
|
for path in paths:
|
2020-08-22 14:56:15 +00:00
|
|
|
# If 'path' is a directory, handle all .cmd files under it.
|
|
|
|
# Otherwise, handle .cmd files associated with the file.
|
2022-09-24 18:19:14 +00:00
|
|
|
# built-in objects are linked via vmlinux.a
|
2020-08-22 14:56:15 +00:00
|
|
|
# Modules are listed in modules.order.
|
|
|
|
if os.path.isdir(path):
|
|
|
|
cmdfiles = cmdfiles_in_dir(path)
|
|
|
|
elif path.endswith('.a'):
|
|
|
|
cmdfiles = cmdfiles_for_a(path, ar)
|
|
|
|
elif path.endswith('modules.order'):
|
|
|
|
cmdfiles = cmdfiles_for_modorder(path)
|
|
|
|
else:
|
|
|
|
sys.exit('{}: unknown file type'.format(path))
|
2020-08-22 14:56:14 +00:00
|
|
|
|
|
|
|
for cmdfile in cmdfiles:
|
|
|
|
with open(cmdfile, 'rt') as f:
|
2020-08-22 14:56:09 +00:00
|
|
|
result = line_matcher.match(f.readline())
|
|
|
|
if result:
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
try:
|
2020-08-22 14:56:14 +00:00
|
|
|
entry = process_line(directory, result.group(1),
|
|
|
|
result.group(2))
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
compile_commands.append(entry)
|
|
|
|
except ValueError as err:
|
|
|
|
logging.info('Could not add line from %s: %s',
|
2020-08-22 14:56:14 +00:00
|
|
|
cmdfile, err)
|
scripts: add a tool to produce a compile_commands.json file
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-12-18 22:49:07 +00:00
|
|
|
|
|
|
|
with open(output, 'wt') as f:
|
|
|
|
json.dump(compile_commands, f, indent=2, sort_keys=True)
|
|
|
|
|
|
|
|
|
|
|
|
if __name__ == '__main__':
|
|
|
|
main()
|