If I provide CLANG_DATABASE_PATH in Doxyfile do I still have to provide INPUT and INCLUDE_PATH? - clang

I am using Doxygen for documentation of my project. I have a compile_commands.json file describing the source code of my project inside the directory C:\dev\project_dir. I set the following variables:
CLANG_ASSISTED_PARSING = YES
CLANG_DATABASE_PATH = C:\dev\project_dir
So how does this work? Do I also have to set the variables INPUT and INCLUDE_PATH? It seems that all the files and the instructions to compile them, including where to get header files from, are written in the compilation database.
And if I do have to set the variables INPUT and INCLUDE_PATH also, what should I set them to? The compilation database lists the source and header files of the project, which are scattered among multiple different directories. How should I proceed in this situation.

I found the answer.
So I set the following variables in the Doxyfile.
CLANG_ASSISTED_PARSING = YES
CLANG_DATABASE_PATH = C:\dev\project_dir
And I set the following variables as blank:
INPUT =
INCLUDE_PATH =
INPUT specifies the paths to source code *.c *.cpp files and/or directories to be processed. INCLUDE_PATH specifies the paths to header code *.h *.hpp files and/or directories to be processed.
The CLANG_ASSISTED_PARSING = YES enables using clang compiler as the parser instead of the default doxygen parser. So if INPUT and INCLUDE_PATH are not set, then it gets the source code files and header code files from the compilation database itself. The CLANG_DATABASE_PATH specifies the directory in which the compilation database is stored. It grabs the file named compile_commands.json by default from that directory, implying that the name of the compilation database is fixed. If you name your compilation database JSON file anything else other than compile_commands.json doxygen won't be able to find it.
So if a clang compilation database JSON file is used all the *.c *.cpp files that are being compiled are placed in the INPUT. And all the header code files are placed in the INCLUDE_PATH. The clang used by doxygen parses the JSON, and every time it encounters a -I compiler flag it recognizes that file as a header file, adding it to the INCLUDE_PATH. This means that setting INPUT and INCLUDE_PATH are not mandatory. So if the compilation database is properly formatted, and all the header files are explicitly marked with the -I, only setting the CLANG_DATABASE_PATH is sufficient.
But there is a certain situation when the INCLUDE_PATH also needs to be set explicitly. For example if you have a source code file, which includes a header file, which includes another header file inside of it.
first.h
int one(int);
second.h
#include "first.h"
int two(int);
code.cpp
#include "second.h"
int main(void) {}
And the command in the compilation database is such:
clang -I path/to/second.h -c code.cpp
So in this case doxygen would read that file, and it would internally set the following variables as such:
INPUT = code.cpp
INCLUDE_PATH = path/to/second.h
This means that although doxygen will index second.h, it will miss first.h since that header file isn't explicitly provided in the -I compilation database. That would be an error. So we need to list it explicitly in the doxyfile, an an additional include path.
INPUT =
INCLUDE_PATH = path\to\first.h
CLANG_ASSISTED_PARSING = YES
CLANG_DATABASE_PATH = C:\dev\project_dir

Related

Reading the content of directory declared with `actions.declare_directory`

Imagine I have a java_binary target triggered by a custom rule that generates source code and places the generated sources under a directory, let's call it "root".
So after the code generation we will have something like this:
// bazel-bin/...../src/com/example/root
root:
-> Foo.java
-> Bar.java
-> utils
-> Baz.java
Now, I have another target, a java_library, that depends on the previously generated sources, so it depends on the custom rule.
My custom rule definition currently looks something like this:
def _code_generator(ctx):
outputDir = ctx.actions.declare_directory("root")
files = [
ctx.actions.declare_file("root/Foo.java"),
ctx.actions.declare_file("root/Bar.java"),
ctx.actions.declare_file("root/utils/Baz.java"),
// and many,
// many other files
]
outputs = []
outputs.append(outputDir)
outputs.extend(files)
ctx.actions.run(
executable = // executable pointing to the java_binary
outputs = outputs
// ....
)
This works. But as you can see, every anticipated file that is to be generated, is hard-coded in the rule definition. This makes it very fragile, should the code generation produce a different set of files in the future (which it will).
(Without specifying each of the files, as shown above, Bazel will fail the build saying that the files have no generating action)
So I was wondering, is there a way to read the content of the root directory and automatically, somehow, declare each of the files as an output?
What I tried:
The documentation of declare_directory says:
The contents of the directory are not directly accessible from Starlark, but can be expanded in an action command with Args.add_all().
And add_all says:
[...] Each directory File item is replaced by all Files recursively contained in that directory.
This sounds like there could be a way to get access to the individual files in the directory, but I am not sure how.
I tried:
outputDir = ctx.actions.declare_directory("root")
//...
args = ctx.actions.args()
args.add_all(outputDir)
with the intention to access the individual files later from args, but the build fails with: "Error in add_all: expected value of type sequence or depset for values, got File".
Any other ideas on how to implement the rule, so that I don't have to hard-code each and every file that will be generated?

How to write Bazel rules that work with external repositories?

The Bazel Starlark API does strange things with files in external repositories. I have the following Starlark snippet:
print(ctx.genfiles_dir)
print(ctx.genfiles_dir.path)
print(output_filename)
ret = ctx.new_file(ctx.genfiles_dir, output_filename)
print(ret.path)
It is creating the following output:
DEBUG: build_defs.bzl:292:5: <derived root>
DEBUG: build_defs.bzl:293:5: bazel-out/k8-fastbuild/genfiles
DEBUG: build_defs.bzl:294:5: google/protobuf/descriptor.upb.c
DEBUG: build_defs.bzl:296:5: bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/google/protobuf/descriptor.upb.c
That extra external/com_google_protobuf comes seemingly out of nowhere, and it makes my rule fail:
I tell protoc to generate into ctx.genfiles_dir.path (which is bazel-out/k8-fastbuild/genfiles).
So protoc generates bazel-out/k8-fastbuild/genfiles/google/protobuf/descriptor.upb.c
Bazel fails because I didn't generate bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/google/protobuf/descriptor.upb.c
Likewise, when I try to call file.short_path on a source file from an external repository, I get a result like ../com_google_protobuf/google/protobuf/descriptor.proto. This seems quite unhelpful, so I just wrote some manual code to strip off the leading ../com_google_protobuf/.
Am I missing something? How can I write this rule in a way that doesn't feel like I'm fighting Bazel the whole time?
Am I missing something?
The basic problem, as you already realized, is that you have two path "namespaces" the one that protoc sees (i.e. import paths) and the one that bazel sees (i.e. the path you pass to declare_file().
2 things to note:
1) All paths declared with declare_file() get the path <bin dir>/<package path incl. workspace>/<path you passed to declare_file()>
2) All actions are executed from <bin dir> (unless output_to_genfils=True in which case this switches to <gen dir> as in your example.
Trying to solve the exact same problem you encountered, I resorted to stripping the known path from the output_file's path to determine which directory to pass as p:
# This code is run from the context of the external protobuf dependency
proto_path = "google/a/b.proto"
output_file = ctx.actions.declare_file(proto_path)
# output_file.path would be `<gen_dir>/external/protobuf/google/a/b.proto`
# Strip the known proto_path from output_file.path
protoc_prefix = output_file.path[:-len(proto_path)]
print(protoc_prefix) # Prints: <gen_dir>/external/protobuf
command = "{protoc} {proto_paths} {cpp_out} {plugin} {plugin_options} {proto_file}".format(
...
cpp_out = "--cpp_out=" + protoc_prefix,
...
)
Alternatives
You may also be able to construct the same path with ctx.bin_dir, ctx.label.workspace_name, ctx.label.package, and ctx.label.name.
Misc.
proto_library recently gained an attribute strip_import_prefix. When used, the above is not correct, as all dependent files are symlinked into a new directory from which they have the relative paths declared with strip_import_prefix.
The path format is:
<bin dir>/<repo>/<package>/_virtual_base/<label name>/<path `import`ed in .proto files>
i.e.
<bin dir>/external/protobuf/_virtual_base/b_proto/google/a/b.proto
Assuming you are building an external repo called protobuf, which contains a BUILD file at its root with a target named b_proto, which in turn, relies on a proto_library wrapping google/a/b.proto AND uses the strip_import_prefix attribute.

How to get the path where the library is installed

I am working in Linux and I have a library written in Fortran 90 (written by 3rd party), that is reading from a file in the current working directory. I would like to be able to call the resulting library from other folders, hence I need to read the path where the library is installed. How can I know the path to the compiled library within the Fortran code?
I need to store in a variable the path within the code.
For who knows python, I want to achieve the same as
import os
os.path.dirname(os.path.abspath(__file__))
but in f90 (see Get location of the .py source file)
Using the suggestions in the comment I have done the following:
export DATAPATH=`pwd`
make
in the Makefile
ifort -O3 -fpic -fpp -DDATAPATH -c mysource.f90
in mysource.f90
subroutine mysub
character(len=100)::dpath
#ifdef DATAPATH
dpath=DATAPATH
#endif
open(10,file=trim(dpath)//'initialise.dat')
....
....
the problem is that at compile time I get
mysource.f90(42): error #6054: A CHARACTER data type is required in this context. [1]
dpath=1
----------^
compilation aborted for mysource.f90 (code 1)
If you wish you can fix the path at compile time. Something like
gfortran -cpp mylib.f90 -DPREFIX=\"/usr/local/\"
open(newunit=u,file=PREFIX//'mylib/initialise.dat')
You must than make sure the library is indeed installed in that place PREFIX/mylib/
You can create an environment variable containing the path of your data. This variable can be set by hand, in your .bashrc or .bash_profile or in the system /etc/profile.d/ or /etc/bash.bashrc, there are manyways and they depend if the library is just for one user or for all users of some large computer.
For example
export MYLIB_PATH='/usr/local/mylib'
Then you can read the variable in Fortran as
CALL get_environment_variable("MYLIB_PATH", mylib_path, status=stat)
and the path is now in variable mylib_path. You can check the success by checking if stat==0.
This is not the only possible approach. You can also have a configuration file for your library in your home directory:
mkdir $HOME/.config/mylib/
echo "/usr/local/mylib" > $HOME/.config/mylib/path
and then you can try to read the path from this file if the environment variable was not set
if (stat/=0) then
CALL get_environment_variable("HOME", home_dir)
open(newunit=path_unit, file=home_dir//'/.config/mylib/path',status='old',action='read',iostat=stat)
if (stat/=0) complain
read(path_unit,'(a)',iostat=stat) mylib_path
if (stat/=0) complain
end if
So when you compiled with -DDATAPATH you have not passed the variable DATAPATH into your code only declared a symbol called DATAPATH as being true, so ifort will substitute DATAPATH as 1. What you need to do is pass it as a value:
-DDATAPATH="$(DATAPATH)"
For the compilation to work.

GNU m4 macros auto generated file

I download latex package on which I want do some changes, but in this packege exist file include.m4 and I don't know what it does and how it was generated. Here its lines:
m4_changequote([[, ]])m4_dnl
m4_dnl
m4_define([[m4_FILE_INIT]], [[m4_dnl
%
% This is automaticaly generated file, do not edit it.
%
]])m4_dnl
m4_dnl
m4_define([[m4_FILE_ID]], [[m4_dnl
m4_patsubst([[$1]], [[\$Date::? \([0-9]+\)-\([0-9]+\)-\([0-9]+\).*]], [[\1/\2/\3]])m4_dnl
v[[]]m4_ESKDX_VERSION]])m4_dnl
m4_dnl
m4_define([[m4_FILE_DATE]], [[m4_dnl
m4_patsubst([[$1]], [[\$Date::? \([0-9]+\)-\([0-9]+\)-\([0-9]+\).*]], [[\1/\2/\3]])]])m4_dnl
m4_dnl
Can you explain with which tool it was generated?
Thk. So this file is not autogenerated? ANd can you help me understand these lines from Makefile:
M4FLAGS = -P -Dm4_ESKDX_INIT="m4_include($(TOP_DIR)/include.m4)" \
-Dm4_ESKDX_VERSION=$(VERSION) -Dm4_ESKDX_DATE=$(RELEASE_DATE)
And rule:
%.def: %.def.in $(M4DEPS)
m4 $(M4FLAGS) $< >$#
%.sty: %.sty.in $(M4DEPS)
m4 $(M4FLAGS) $< >$#
%.cls: %.cls.in $(M4DEPS)
m4 $(M4FLAGS) $< >$#
As I can see GNU m4 options '-D' substitutes macro m4_ESKDX_INIT in .sty .cls files to m4_include(../include.m4) and then options '-P' first expands file include.m4 and furthemore expands macros in include.m4.
This is a macro for the GNU m4 macro processor. This file is designed to be used with the -P or --prefix-builtins commandline option. The m4_ part will be stripped away when m4 evaluates this file. This file doesn't do anything itself, it just defines three macros (FILE_INIT, FILE_ID and FILE_DATE) which presumably will be used in another step. You might want to look in the other files for references to this one. The basic idea will be to load this file before running another file through m4 and it will replace those macros as it goes.
The message about automatically generated is supposed to end up in the final file as a comment. As we can see in the rules in the Makefile, each of the .def, .sty and .cls files are generated from an equivalently named .in file (so result.cls will be built from result.cls.in. by evaluating the macros in these files and replacing them with the equivalents.
So, to modify these files, you will want to edit the .in files.

Remove the path in objcopy symbol names

I need to include a binary program in my project. I use objcopy to create an object file from a binary file. The object file can be linked in my program. objcopy creates appropriate symbols to access the binary data.
Example
objcopy -I binary -O elf32-littlearm --binary-architecture arm D:\Src\data.jpg data.o
The generated symbols are:
_binary_D__Src_data_jpg_end
_binary_D__Src_data_jpg_size
_binary_D__Src_data_jpg_start
The problem is that the symbols include the path to the binary file D__Src_. This may help when binary files are included from different location. But it bothers me that the symbols changes when I get the file from a different location. Since this shall run on several build stations, the path can't be stripped with the --redefine-sym option.
How do I get rid of the path in the symbol name?
I solved this problem by using this switch in objcopy:
--prefix-sections=abc
This gives a way to uniquely identify the data in your binary object file (ex. binary.o)
In your linker script you can then define your own labels around where you include the binary.o. Since you are no longer referencing anything in binary.o the binary will be thrown out by the linker if you use -gc-sections switch. The section in binary.o will now be abc.data. Use KEEP in your linker script to tell the linker not to throw out binary.o. Your linker script will contain the following:
__binary_start__ = .;
KEEP(*(abc.data))
binary.o
*(abc.data)
. = ALIGN(4);
__binary_end__ = .;
The switch --localize-symbols works for me.

Resources