I have a somewhat complex SCons build script that an some point does the following two steps:
# 1: builds unit tests (googletest, shell executable)
compile_tests = some_environment.Program(executable_path, test_sources)
# 2: runs unit tests (call earlier compiled program)
run_tests = other_environment.Command(
source = executable_path,
action = executable_path + ' --gtest_output=xml:' + test_results_path,
target = test_results_path
)
Depends(run_tests, compile_tests)
This is working fine if I run scons with this build script on its own.
If I, however, invoke it via environment.SConscript() from another SConstruct file one directory level up, then step 1 adjusts the path to the project's location while step 2 doesn't. See this output:
scons: Building targets ...
g++ -o Nuclex.Game.Native/obj/gcc-7-amd64-release/NuclexGameNativeTests -z defs -Bsymbolic Nuclex.Game.Native/obj/gcc-7-amd64-release/Tests/Timing/ClockTests.o -LNuclex.Game.Native/obj/gcc-7-amd64-release -LReferences/googletest/gcc-7-amd64-release -lNuclexGameNativeStatic -lgoogletest -lgoogletest_main -lpthread
obj/gcc-7-amd64-release/NuclexGameNativeTests --gtest_output=xml:bin/gcc-7-amd64-release/gtest-results.xml
sh: obj/gcc-7-amd64-release/NuclexGameNativeTests: No such file or directory
Line 2 builds the executable into Nuclex.Game.Native/obj/gcc-7-amd64-release/ while line 3 tries to call it in obj/gcc-7-amd64-release/, forgetting the project directory.
Should I use another way to invoke my unit test executable? Or can I query the SCons environment for its base path?
Update: reproduction case, place https://pastebin.com/W08yZuF9 as SConstruct in root directory, create subdirectory somelib and place https://pastebin.com/eiP63Yxh as SConstruct therein, also create main.cpp with a "Hello World" or other dummy program.
A SCons Action (the action parameter in the Command) will use the SCons variables to substitute sources and targets in correctly, taking into account VariantDirs and SConscript directories automatically. You can find more info on these source and target substitutions here: https://scons.org/doc/HTML/scons-man.html#variable_substitution
There is a section which explains using this in regards to SConscript and VariantDirs:
SConscript('src/SConscript', variant_dir='sub/dir')
$SOURCE => sub/dir/file.x
${SOURCE.srcpath} => src/file.x
${SOURCE.srcdir} => src
So in your example I think you want to replace executable_path with $SOURCE in the action string:
# 2: runs unit tests (call earlier compiled program)
run_tests = other_environment.Command(
source = executable_path,
action = '$SOURCE --gtest_output=xml:$TARGET',
target = test_results_path
)
Related
I have set up bazel to build a number of CLI tools that perform various database maintenance tasks. Each one is a py_binary or cc_binary target that is called from the command line with the path to some data file: it processes that file and stores the results in a database.
Now, I need to create a dependent package that contains data files and shell scripts that call these CLI tools to perform application-specific database operations.
However, there doesn't seem to be a way to depend on the existing py_binary or cc_binary targets from a new package that only contains sh_binary targets and data files. Trying to do so results in an error like:
ERROR: /workspace/shbin/BUILD.bazel:5:12: in deps attribute of sh_binary rule //shbin:run: py_binary rule '//pybin:counter' is misplaced here (expected sh_library)
Is there a way to call/depend on an existing bazel binary target from a shell script using sh_binary?
I have implemented a full example here:
https://github.com/psigen/bazel-mixed-binaries
Notes:
I cannot use py_library and cc_library instead of py_binary and cc_binary. This is because (a) I need to call mixes of the two languages to process my data files and (b) these tools are from an upstream repository where they are already designed as CLI tools.
I also cannot put all the data files into the CLI tool packages -- there are multiple application-specific packages and they cannot be mixed.
You can either create a genrule to run these tools as part of the build, or create a sh_binary that depends on the tools via the data attribute and runs them them.
The genrule approach
This is the easier way and lets you run the tools as part of the build.
genrule(
name = "foo",
tools = [
"//tool_a:py",
"//tool_b:cc",
],
srcs = [
"//source:file1",
":file2",
],
outs = [
"output_file1",
"output_file2",
],
cmd = "$(location //tool_a:py) --input=$(location //source:file1) --output=$(location output_file1) && $(location //tool_b:cc) < $(location :file2) > $(location output_file2)",
)
The sh_binary approach
This is more complicated, but lets you run the sh_binary either as part of the build (if it is in a genrule.tools, similar to the previous approach) or after the build (from under bazel-bin).
In the sh_binary you have to data-depend on the tools:
sh_binary(
name = "foo",
srcs = ["my_shbin.sh"],
data = [
"//tool_a:py",
"//tool_b:cc",
],
)
Then, in the sh_binary you have to use the so-called "Bash runfiles library" built into Bazel to look up the runtime-path of the binaries. This library's documentation is in its source file.
The idea is:
the sh_binary has to depend on a specific target
you have to copy-paste some boilerplate code to the top of the sh_binary (reason is described here)
then you can use the rlocation function to look up the runtime-path of the binaries
For example your my_shbin.sh may look like this:
#!/bin/bash
# --- begin runfiles.bash initialization ---
...
# --- end runfiles.bash initialization ---
path=$(rlocation "__main__/tool_a/py")
if [[ ! -f "${path:-}" ]]; then
echo >&2 "ERROR: could not look up the Python tool path"
exit 1
fi
$path --input=$1 --output=$2
The __main__ in the rlocation path argument is the name of the workspace. Since your WORKSPACE file does not have a "workspace" rule in, which would define the workspace's name, Bazel will use the default workspace name, which is __main__.
An easier approach for me is to add the cc_binary as a dependency in the data section. In prefix/BUILD
cc_binary(name = "foo", ...)
sh_test(name = "foo_test", srcs = ["foo_test.sh"], data = [":foo"])
Inside foo_test.sh, the working directory is different, so you need to find the right prefix for the binary
#! /usr/bin/env bash
executable=prefix/foo
$executable ...
A clean way to do this is to use args and $(location):
Contents of BUILD:
py_binary(
name = "counter",
srcs = ["counter.py"],
main = "counter.py",
)
sh_binary(
name = "run",
srcs = ["run.sh"],
data = [":counter"],
args = ["$(location :counter)"],
)
Contents of counter.py (your tool):
print("This is the counter tool.")
Contents of run.sh (your bash script):
#!/bin/bash
set -eEuo pipefail
counter="$1"
shift
echo "This is the bash script, about to call the counter tool."
"$counter"
And here's a demo showing the bash script calling the Python tool:
$ bazel run //example:run 2>/dev/null
This is the bash script, about to call the counter tool.
This is the counter tool.
It's also worth mentioning this note (from the docs):
The arguments are not passed when you run the target outside of bazel (for example, by manually executing the binary in bazel-bin/).
The docs for Bazel's cc_binary rule say:
Implicit output targets
<name>.stripped (only built if explicitly requested): A stripped version of the binary. strip -g is run on the binary to remove debug symbols. Additional strip options can be provided on the command line using --stripopt=-foo. This output is only built if explicitly requested.
How do I "explicitly request" that this stripped binary get built? Is there something I need to put in my cc_binary declaration in my BUILD file? I can't figure it out from the docs (or the Bazel source).
Okay I think I figured out how.
If my BUILD file has this:
cc_binary(
name = "mytool",
srcs = ["mytool.c"]
)
... then from the command line I can build the stripped binary with this:
bazel build //:mytool.stripped
or, the more common scenario, if I have another BUILD rule that needs the stripped binary as one of its inputs, I can just refer to it by that same label, :mytool.stripped. Here is sort of a weird contrived example:
genrule(
name = "mygenrule",
outs = ["genrule.out"],
srcs = [":tool1.stripped"],
# run tool1.stripped, sends its output to genrule.out:
cmd = "$(SRCS) > $#"
)
I'm looking for a good recipe to run "checks" or "verify" steps in Bazel, like go vet, gofmt, pylint, cppcheck. These steps don't create any output file. The only thing that matters is the return code (like a test).
Right now I'm using the following recipe:
sh_test(
name = "verify-pylint",
srcs = ["verify-pylint.sh"],
data = ["//:all-srcs"],
)
And verify-pylint.sh looks like this:
find . -name '*.py' | xargs pylint
This has two problems:
The verify logic is split between the shell script and the BUILD file. Ideally I would like to have both in the same place (in the BUILD file)
Anytime one of the source file changes (in //:all-srcs), bazel test verify-pylint re-runs pylint on every single file (and that can be expensive/slow).
What is the idiomatic way in bazel to run these steps?
There are more than one solutions.
The cleanest way is to do the verification at build time: you create a genrule for each file (or batch of files) you want to verify, and if verification succeeds, the genrule outputs something, if it fails, then the rule outputs nothing, which automatically fails the build as well.
Since success of verification depends on the file's contents, and the same input should yield the same output, the genrules should produce an output file that's dependent on the contents of the input(s). The most convenient thing is to write the digest of the file(s) to the output if verification succeeded, and no output if verification fails.
To make the verifier reusable, you could create a Skylark macro and use it in all your packages.
To put this all together, you'd write something like the following.
Contents of //tools:py_verify_test.bzl:
def py_verify_test(name, srcs, visibility = None):
rules = {"%s-file%d" % (name, hash(s)): s for s in srcs}
for rulename, src in rules.items():
native.genrule(
name = rulename,
srcs = [s],
outs = ["%s.md5" % rulename],
cmd = "$(location //tools:py_verifier) $< && md5sum $< > $#",
tools = ["//tools:py_verifier"],
visibility = ["//visibility:private"],
)
native.sh_test(
name = name,
srcs = ["//tools:build_test.sh"],
data = rules.keys(),
visibility = visibility,
)
Contents of //tools:build_test.sh:
#!/bin/true
# If the test rule's dependencies could be built,
# then all files were successfully verified at
# build time, so this test can merely return true.
Contents of //tools:BUILD:
# I just use sh_binary as an example, this could
# be a more complicated rule of course.
sh_binary(
name = "py_verifier",
srcs = ["py_verifier.sh"],
visibility = ["//visibility:public"],
)
Contents of any package that wants to verify files:
load("//tools:py_verify_test.bzl", "py_verify_test")
py_verify_test(
name = "verify",
srcs = glob(["**/*.py"]),
)
A simple solution.
In your BUILD file:
load(":gofmt.bzl", "gofmt_test")
gofmt_test(
name = "format_test",
srcs = glob(["*.go"]),
)
In gofmt.bzl:
def gofmt_test(name, srcs):
cmd = """
export TMPDIR=.
out=$$(gofmt -d $(SRCS))
if [ -n "$$out" ]; then
echo "gmfmt failed:"
echo "$$out"
exit 1
fi
touch $#
"""
native.genrule(
name = name,
cmd = cmd,
srcs = srcs,
outs = [name + ".out"],
tools = ["gofmt.sh"],
)
Some remarks:
If your wrapper script grows, you should put it in a separate .sh file.
In the genrule command, we need $$ instead $ due to escaping (see documentation)
gofmt_test is actually not a test and will run with bazel build :all. If you really need a test, see Laszlo's example and call sh_test.
I call touch to create a file because genrule requires an output to succeed.
export TMPDIR=. is needed because by default the sandbox prevents writing in other directories.
To cache results for each file (and avoid rechecking a file that hasn't changed), you'll need to create multiple actions. See Laszlo's for loop.
To simplify the code, we could provide a generic rule. Maybe this is something we should put in a standard library.
I added
version.target = version.h
version.commands = bash generate-version.sh
QMAKE_EXTRA_TARGETS += version
PRE_TARGETDEPS += version.h
to the project, but it attempts to run "generate-version.sh" in destination directory:
make: Leaving directory `.../qqq-build-desktop'
make: Entering directory `.../qqq-build-desktop'
Makefile:236: warning: overriding commands for target `version.h'
Makefile:233: warning: ignoring old commands for target `version.h'
bash generate-version.sh
bash: generate-version.sh: No such file or directory
make: Leaving directory `.../qqq-build-desktop'
There is $$DESTDIR, but I don't see $$SRCDIR. How to refer to the project directory in qmake (or how to rewrite this)?
My first thought is to try to rewrite
version.commands = bash generate-version.sh
so as not to have to invoke a shell script. Perhaps you can combine all of the statements into one line:
version.commands = echo \'char VERSION[]=\"1.0\";\' > version.h && ls && echo Done
If you are stuck with invoking the script, probably PWD or OUT_PWD are what you are looking for. From the qmake Variable Reference
PWD
This variable contains the full path leading to the directory where the qmake project file (project.pro) is located.
OUT_PWD
This variable contains the full path leading to the directory where qmake places the generated Makefile.
The one caveat that is not mentioned in the documentation is that if you are doing a recursive qmake, PWD refers to where the top level .pro file was read from. Thus if you run qmake -r from {proj-root}, when sub/sub/sub/dir-proj.pro is finally read in, PWD will still point to {proj-root}.
Assuming that generate-version.sh is in the same directory as your top level .pro file, you might try:
version.commands = bash $$PWD/generate-version.sh
I found a better and cleaner solution
version.target = version.h
version.commands = bash ${QMAKE_VAR__PRO_FILE_PWD_}/generate-version.sh
QMAKE_EXTRA_TARGETS += version
The variable _PRO_FILE_PWD_ is documented since qt 4.5 and contains the path to the directory containing the project file in use (Contains the .pro file)
But to access this variable for QMAKE_EXTRA_TARGETS, QMAKE_VAR_ must be appended.
PWD
Specifies the full path leading to the directory containing the
current file being parsed. This can be useful to refer to files within
the source tree when writing project files to support shadow builds.
I use (Linux and g++)
DEFINES += SVN_VERSION=\\\"\""`svnversion $$PWD`\""\\\"
DEFINES += COMPILE_DATE=\\\"\""`date`\""\\\"
DEFINES += SW_VERSION=\\\"\"0.5\"\\\"
which defines the macro SVNVERSON to be the svn version.
To access it from C++:
QString svnVersion = SVN_VERSION;
QString swVersion = SW_VERSION;
Explanation: On the shell I want to see this call:
-DSVN_VERSION=\""`svnversion /path/to/my/source`"\"
As you see some escapes are necessary on shell level. In the .pro-file it then has to be escaped twice.
This works and is easy to understand.
version.commands = ( cd $${PWD}; generate-version.sh )
I've been using sbox with a Make-based codebase with no problems. Now
I'm using a scons-based codebase, and am getting some odd problems.
It seems that within scratchbox, scons can't find g++. For example, it
attempts to execute things like:
o hello hello.c
When it should be doing:
g++ -o hello hello.c
So presumably its g++ string variable is empty. g++ is present and in
the PATH - "which g++" produces /scratchbox/compilers/bin/g++.
The same source builds fine outside of scratchbox, so it shouldn't be
a problem with scons or the codebase. There are no special environment
variables set outside of scratchbox when it works.
If I symbolically link /usr/bin/g++ to /scratchbox/compilers/bin/g++,
it gets a bit further (produces the correct-looking g++ commands) but
then upon executing them produces:
sb_gcc_wrapper (g++):
/scratchbox/compilers/arm-linux-cs2007q3-51sb3/bin/sbox-arm-none-linux-gnueabi-g++:
No such file or directory
The file listed is present.
PATH contains /scratchbox/compilers/bin, SBOX_REDIRECT_FROM_DIRS contains /usr/bin and SBOX_REDIRECT_TO_DIRS contains /scratchbox/compilers/bin, so I think it should be able to find it.
Any suggestions would be appreciated!
Thanks,
Ray
Edit: Perhaps related - it also can't find pkg-config unless I prepend the full path within the scons file
scons does not propagate the PATH environment variable, so testing e.g. 'which g++' doesn't help much.
Either set the compilers directly, e.g.
env['CXX'] = '/scratchbox/compilers/bin/g++'
Build your own explicit PATH
path = ['/scratchbox/compilers/bin/','/bin', '/usr/bin', '/sbin','/usr/sbin']
env = Environment(ENV = {'PATH' : path})
Or use the PATH env variable from your shell
import os
env = Environment(ENV = {'PATH' : os.environ['PATH']})