I'm using PRE_TARGETDEPS to generate source files, and I'm adding the generated source files to SOURCES for compilation.
The output of my generator obviously doesn't exist at the time qmake is run, so qmake outputs WARNING: Failure to find: for each of the to-be-created source files.
How can I quell this warning, since I know my PRE_TARGETDEPS is going to produce those files?
Or, is there a better way to generate intermediate files using qmake?
Example
Here's a complete test.pro file that exhibits the problem:
TEMPLATE = lib
preprocess.commands += cat test.cc.in | sed 's/a/0/g' > test.0.cc ;
preprocess.commands += cat test.cc.in | sed 's/a/1/g' > test.1.cc ;
preprocess.depends = test.cc.in
QMAKE_EXTRA_TARGETS += preprocess
PRE_TARGETDEPS += preprocess
SOURCES = test.0.cc test.1.cc
Place this in an empty folder, and also create an empty test.cc.in file. Run qmake, and you'll see these warnings:
WARNING: Failure to find: test.0.cc
WARNING: Failure to find: test.1.cc
How can I quell this warning
From my reading of the qmake code, it looks like you can:
either have qmake ignore any filenames that don't exist - in which case they won't get built by your later steps
or have it write these warnings
I don't think either of these would be satisfactory for you.
Here's my reasoning.... I had a hunt for the text Failure to find in a Qt distribution I had to hand: qt4.8.1.
It appeared 3 times in in qmake/generators/makefile.cpp. The two blocks of code look like this:
QStringList
MakefileGenerator::findFilesInVPATH(QStringList l, uchar flags, const QString &vpath_var)
{
....
debug_msg(1, "%s:%d Failure to find %s in vpath (%s)",
__FILE__, __LINE__,
val.toLatin1().constData(), vpath.join("::").toLatin1().constData());
if(flags & VPATH_RemoveMissingFiles)
remove_file = true;
else if(flags & VPATH_WarnMissingFiles)
warn_msg(WarnLogic, "Failure to find: %s", val.toLatin1().constData());
....
else if(flags & VPATH_WarnMissingFiles)
warn_msg(WarnLogic, "Failure to find: %s", val.toLatin1().constData());
and this is called with:
l = findFilesInVPATH(l, (comp.flags & Compiler::CompilerRemoveNoExist) ?
VPATH_RemoveMissingFiles : VPATH_WarnMissingFiles, "VPATH_" + comp.variable_in);
So the flags parameter passed in to the first block will be either RemoveMissingFiles or WarnMissingFiles, depending on comp.flags & Compiler::CompilerRemoveNoExist.
Or, is there a better way to generate intermediate files using qmake?
I'm not sure that it's better - i.e. it's certainly complex - but this is what is done where I work...
In the .pro file, a system call is done, that:
generates the required files,
and then writes out their names to stdout.
Here's an example from the .pro, to show how it would be called:
SOURCES += $$system( python my_script_name.py )
You can of course pass arguments in to the python script, if you like
Things to note/limitations:
This means that the python script gets run whenever you run qmake, but not during individual make invocations
Each invocation of python really slows down our qmake steps - taking roughly twice as long as running qmake without launching python - but you could always use a different scripting language
This would fix your problem, in that by the time qmake processes the SOURCES value, the files have been created by the script.
Related
Imagine I have a java_binary target triggered by a custom rule that generates source code and places the generated sources under a directory, let's call it "root".
So after the code generation we will have something like this:
// bazel-bin/...../src/com/example/root
root:
-> Foo.java
-> Bar.java
-> utils
-> Baz.java
Now, I have another target, a java_library, that depends on the previously generated sources, so it depends on the custom rule.
My custom rule definition currently looks something like this:
def _code_generator(ctx):
outputDir = ctx.actions.declare_directory("root")
files = [
ctx.actions.declare_file("root/Foo.java"),
ctx.actions.declare_file("root/Bar.java"),
ctx.actions.declare_file("root/utils/Baz.java"),
// and many,
// many other files
]
outputs = []
outputs.append(outputDir)
outputs.extend(files)
ctx.actions.run(
executable = // executable pointing to the java_binary
outputs = outputs
// ....
)
This works. But as you can see, every anticipated file that is to be generated, is hard-coded in the rule definition. This makes it very fragile, should the code generation produce a different set of files in the future (which it will).
(Without specifying each of the files, as shown above, Bazel will fail the build saying that the files have no generating action)
So I was wondering, is there a way to read the content of the root directory and automatically, somehow, declare each of the files as an output?
What I tried:
The documentation of declare_directory says:
The contents of the directory are not directly accessible from Starlark, but can be expanded in an action command with Args.add_all().
And add_all says:
[...] Each directory File item is replaced by all Files recursively contained in that directory.
This sounds like there could be a way to get access to the individual files in the directory, but I am not sure how.
I tried:
outputDir = ctx.actions.declare_directory("root")
//...
args = ctx.actions.args()
args.add_all(outputDir)
with the intention to access the individual files later from args, but the build fails with: "Error in add_all: expected value of type sequence or depset for values, got File".
Any other ideas on how to implement the rule, so that I don't have to hard-code each and every file that will be generated?
The Bazel Starlark API does strange things with files in external repositories. I have the following Starlark snippet:
print(ctx.genfiles_dir)
print(ctx.genfiles_dir.path)
print(output_filename)
ret = ctx.new_file(ctx.genfiles_dir, output_filename)
print(ret.path)
It is creating the following output:
DEBUG: build_defs.bzl:292:5: <derived root>
DEBUG: build_defs.bzl:293:5: bazel-out/k8-fastbuild/genfiles
DEBUG: build_defs.bzl:294:5: google/protobuf/descriptor.upb.c
DEBUG: build_defs.bzl:296:5: bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/google/protobuf/descriptor.upb.c
That extra external/com_google_protobuf comes seemingly out of nowhere, and it makes my rule fail:
I tell protoc to generate into ctx.genfiles_dir.path (which is bazel-out/k8-fastbuild/genfiles).
So protoc generates bazel-out/k8-fastbuild/genfiles/google/protobuf/descriptor.upb.c
Bazel fails because I didn't generate bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/google/protobuf/descriptor.upb.c
Likewise, when I try to call file.short_path on a source file from an external repository, I get a result like ../com_google_protobuf/google/protobuf/descriptor.proto. This seems quite unhelpful, so I just wrote some manual code to strip off the leading ../com_google_protobuf/.
Am I missing something? How can I write this rule in a way that doesn't feel like I'm fighting Bazel the whole time?
Am I missing something?
The basic problem, as you already realized, is that you have two path "namespaces" the one that protoc sees (i.e. import paths) and the one that bazel sees (i.e. the path you pass to declare_file().
2 things to note:
1) All paths declared with declare_file() get the path <bin dir>/<package path incl. workspace>/<path you passed to declare_file()>
2) All actions are executed from <bin dir> (unless output_to_genfils=True in which case this switches to <gen dir> as in your example.
Trying to solve the exact same problem you encountered, I resorted to stripping the known path from the output_file's path to determine which directory to pass as p:
# This code is run from the context of the external protobuf dependency
proto_path = "google/a/b.proto"
output_file = ctx.actions.declare_file(proto_path)
# output_file.path would be `<gen_dir>/external/protobuf/google/a/b.proto`
# Strip the known proto_path from output_file.path
protoc_prefix = output_file.path[:-len(proto_path)]
print(protoc_prefix) # Prints: <gen_dir>/external/protobuf
command = "{protoc} {proto_paths} {cpp_out} {plugin} {plugin_options} {proto_file}".format(
...
cpp_out = "--cpp_out=" + protoc_prefix,
...
)
Alternatives
You may also be able to construct the same path with ctx.bin_dir, ctx.label.workspace_name, ctx.label.package, and ctx.label.name.
Misc.
proto_library recently gained an attribute strip_import_prefix. When used, the above is not correct, as all dependent files are symlinked into a new directory from which they have the relative paths declared with strip_import_prefix.
The path format is:
<bin dir>/<repo>/<package>/_virtual_base/<label name>/<path `import`ed in .proto files>
i.e.
<bin dir>/external/protobuf/_virtual_base/b_proto/google/a/b.proto
Assuming you are building an external repo called protobuf, which contains a BUILD file at its root with a target named b_proto, which in turn, relies on a proto_library wrapping google/a/b.proto AND uses the strip_import_prefix attribute.
Goal of my Makefile is to create in the end a static library *.a out of Fortran77 files and some *.c's + *.h's whereas a specific part of the headers have to be precompiled with a special company internal precompiler which is provided via executable and all you have to hand over is the pathname+filename.
Let's call the Precompiler CPreComp.
The files needing the precompilation *_l.h .
So I want first to collect all the headers I need to precompile and then hand it over to a script which does some magic (env variables blubb blubb) and calls the precompiler.
Here you go with my Makefile:
SHELL=/usr/bin/bash
.SHELLFLAGS:= -ec
SOURCE_PATH = ./src
CPRECOMP = ./tools/cprecomp.exe
DO_CPreComp = $(SOURCE_PATH)/do_cprec
HDREXT = .h
PREC_HEADERS = $(foreach d, $(SOURCE_PATH), $(wildcard $(addprefix $(d)/*, $(HDREXT))))
.PHONY: all prereq
all:: \
prereq \
lib.a
prereq: chmod 777 $(DO_CPreComp)
echo $(PREC_HEADERS) >> makefileTellMeWhatYouHaveSoFar.txt
lib.a: \
obj/file1.o \
obj/file2.o
ar -r lib.a $?
obj/file1.o:
# do some fortran precompiling stuff here for a specific file
obj/file2.o: $(SOURCE_PATH)/*.h precomp_path/*.h $(SOURCE_PATH)/file2.c precomp_path/%_l.h
cc -c -g file2.c
precomp_path/%_l.h : DatabaseForPreComp.txt
precomp_path/%_l.h :
$(foreach i , $(PREC_HEADERS) , $(DO_CPreComp) $(i) $(CPRECOMP);)
So that is my Makefile, the script for the DO_CPreComp looks as follows:
#!/bin/bash
filename="(basename "$1")"
dir="$(dirname "$1")"
cprecomptool="$2"
echo ${dir} ${filename} ${cprecomptool} >> scriptTellMeWhatYouKnow.txt
"${cprecomptool}" "precomp_path/${filename}.1" >&cprecomp.err
cp "precomp_path/${filename}.1" "precomp_path/${filename}"
So according to the makefileTellMeWhatYouHaveSoFar.txt I collect all the headers, obviously also the ones not specified with _l.h . This has space for improvement but the precompiler is smart enough to skip the files which are not suitable. So makefileTellMeWhatYouHaveSoFar.txt looks like that:
header1.h header2.h header2_l.h headerx_l.h headery_l.h headerz.h
The Error tells me:
path_to_here/do_cprec : line xy: $2: unbound variable
make[2]: *** [precomp_path/%_l.h] Error 1
make[1]: *** [lib.a] Error 2
scriptTellMeWhatYouKnow.txt shows me the script knows nothing and it is not even created. If I modify cprecomptool and directly add it in the script hardcoded the scriptTellMeWhatYouKnow.txt shows me the argument $(CPRECOMP) twice as file name and path name and the hardcoded precompiler. And ofc it ends up with Segmentation fault, so the header name was never handed over.
Additionally:
If I do not call the script in the second foreach but let $(i) be printed out with echo in another file it is empty.
Perhaps I am just too blind. And please if you are able to help me , explain it to me for dumb people, such that for the next time I stumble over a problem I am smarter because I know what I am doing. :)
OK, now that the main issue is solved, let's have a look at make coding styles. The make way of accomplishing what you want is not exactly using foreach in recipes. There are several drawbacks with this approach like, for instance, the fact that make cannot run parallel jobs, while it is extremely good at this. And on modern multi-core architectures, it can really make a difference. Or the fact that things are always redone while they are potentially up to date.
Assuming the result of the pre-compilation of foo_l.h file is a foo.h (we will look at other options later), the make way is more something like:
SOURCE_PATH := ./src
CPRECOMP := ./tools/cprecomp.exe
DO_CPreComp := $(SOURCE_PATH)/do_cprec
HDREXT := .h
PREC_HEADERS := $(wildcard $(addsuffix /*_l.$(HDREXT),$(SOURCE_PATH)))
PRECOMPILED_HEADERS := $(patsubst %_l.h,%.h,$(PREC_HEADERS))
$(PRECOMPILED_HEADERS): %_l.h: %.h DatabaseForPreComp.txt
$(DO_CPreComp) $# $(CPRECOMP)
($# expands as the target). This is a static pattern rule. With this coding style only the headers that need to be pre-compiled (because they are older than their prerequisites) are re-built. And if you run make in parallel mode (make -j4 for 4 jobs in parallel) you should see a nice speed-up factor on a multi-core processor.
But what if the pre-compilation modifies the foo_l.h file itself? In this case you need another dummy (empty) file to keep track of when a file has been pre-compiled:
SOURCE_PATH := ./src
CPRECOMP := ./tools/cprecomp.exe
DO_CPreComp := $(SOURCE_PATH)/do_cprec
HDREXT := .h
PREC_HEADERS := $(wildcard $(addsuffix /*_l.$(HDREXT),$(SOURCE_PATH)))
PREC_TAGS := $(patsubst %,%.done,$(PREC_HEADERS))
$(PREC_TAGS): %.done: % DatabaseForPreComp.txt
$(DO_CPreComp) $< $(CPRECOMP) && \
touch $#
($< expands as the first prerequisite). The trick here is that the foo_l.h.done empty file is a marker. Its last modification time records the last time foo_l.h has been pre-compiled. If foo_l.h or DatabaseForPreComp.txt has changed since, then foo_l.h.done is out of date and make re-builds it, that is, pre-compiles foo_l.h and then touch foo_l.h.done to update its last modification time. Of course, if you use this, you must tell make that some other targets depend on $(PREC_TAGS).
With the help of #Renaud Pacalet I was able to find a solution.
In the comments you can read further try & errors.
I am using GNU Make 3.82 Built for x86_64-redhat-linux-gnu . Seems like the foreach does not like the space behind the i or furthermore takes the space as part of the variable.
# ... like beforehand check out in the question
PREC_HEADERS=$(shell find $(SOURCE_PATH) -iname '*_l.h')
# nothing changed here in between...
$(foreach i,$(PREC_HEADERS),$(DO_CPC) $i $(CPC);)
This has the advantage that I only precompile the headers which have the _l.h - ending. Having the brackets $(i) around the $i or not, doesn't make a change. What really changed everything was the space behind the first i .
Good luck!
My .pro file has extra stuff in it:
win32 {
OUT_PWD_SHELL = $$replace(OUT_PWD, /, \\)
autoversion.target = $$OUT_PWD\\autoversioninfo.h
autoversion.depends = FORCE
autoversion.commands = $$PWD/../../AutoBuildVersion.exe $$replace(PWD, /, \\) $$OUT_PWD_SHELL
QMAKE_EXTRA_TARGETS += autoversion
PRE_TARGETDEPS += $$OUT_PWD\\autoversioninfo.h
}
This fails to work as expected because in the generated makefile DESTDIR_TARGET has a new dependency added that starts d:\ but the rule generated for autoversion starts with D:/. I can improve this slightly by replacing all / with \, but the case sensitivity still breaks it and the target is not built.
If I remove the full path from autoversion.target and PRE_TARGETDEPS then it solves that problem, but then when calculating dependencies, the rule for the cpp file that includes the generated header changes to give an explicit path to the header in the dependencies, and that path points to the source directory and not the output directory where the generated file is produced. This causes make to barf and not produce the generated file.
I don't know why qmake changes the case handling of the drive, it is very irritating, but how do I get this all to work correctly?
There is no good solution. The best I came up with is to use a phony target that always runs to generate the header file. On the downside this slows the build when the header file already exists, but on the upside, it allows the build to complete.
win32 {
OUT_PWD_SHELL = $$replace(OUT_PWD, /, \\)
gen_autoversion.target = GENERATE_AUTOVERSIONINFO
gen_autoversion.commands = $$PWD/../../AutoBuildVersion.exe $$replace(PWD, /, \\) $$OUT_PWD_SHELL
QMAKE_EXTRA_TARGETS += gen_autoversion
PRE_TARGETDEPS += GENERATE_AUTOVERSIONINFO
}
I am not sure that PRE_TARGETDEPS is actually needed here.
I have a relatively complicated suite of OMake files designed for cross-compiling on a specific platform. My source is in C++.
I'm building from Windows and I need to pass to the compiler include directories which have spaces in their names. The way that the includes string which is inserted in the command line to compile files is created is by the line:
public.PREFIXED_INCLUDES = $`(addprefix $(INCLUDES_OPT), $(set $(absname $(INCLUDES))))
At some other point in the OMake files I have a line like:
INCLUDES += $(dir "$(LIBRARY_LOCATION)/Path with spaces/include")
In the middle of the command line this expands to:
-IC:\Library location with spaces\Path with spaces\include
I want it to expand to:
-I"C:\Library location with spaces\Path with spaces\include"
I don't want to change anything but the "INCLUDES += ..." line if possible, although modifying something else in that file is also fine. I don't want to have to do something like change the definition of PREFIXED_INCLUDES, as that's in a suite of OMake files which are part of an SDK which may change beneath me. Is this possible? If so, how can I do it? If not, in what ways can I make sure that includes with spaces in them are quoted by modifying little makefile code (hopefully one line)?
The standard library function quote adds escaped quotes around its argument, so it should do the job:
INCLUDES += $(quote $(dir "$(LIBRARY_LOCATION)/Path with spaces/include"))
If needed, see quote in Omake manual.
In case someone else is having the same problem, I thought I'd share the solution I eventually went with, having never figured out how to surround with quotes. Instead of putting quotes around a name with spaces in it I ended up converting the path to the short (8.3) version. I did this via a a simple JScript file called shorten.js and a one line OMake function.
The script:
// Get Access to the file system.
var FileSystemObject = WScript.CreateObject("Scripting.FileSystemObject");
// Get the short path.
var shortPath = FileSystemObject.GetFolder(WScript.Arguments(0)).ShortPath;
// Output short path.
WScript.StdOut.Write(shortPath);
The function:
ShortDirectoryPath(longPath) =
return $(dir $(shell cscript /Nologo $(dir ./tools/shorten.js) "$(absname $(longPath))"))
So now I just use a line like the following for includes:
INCLUDES += $(ShortDirectoryPath $(dir "$(LIBRARY_LOCATION)/Path with spaces/include"))