Bazel: why lint failed for variable reference? - bazel

I had this genrule in BUILD file, but bazel build failed for the error of:
in cmd attribute of genrule rule //example:create_version_pom: $(BUILD_TAG) not defined
genrule(
name = "create_version_pom",
srcs = ["pom_template.xml"],
outs = ["version_pom.xml"],
cmd = "sed 's/BUILD_TAG/$(BUILD_TAG)/g' $< > $#",
)
What's the reason, and how to fix it please?

The cmd attribute of genrule will do variable expansion for Bazel build variables before the command is executed. The $< and $# variables for the input file and the output file are some of the pre-defined variables. Variables can be defined with --define, e.g.:
$ cat BUILD
genrule(
name = "genfoo",
outs = ["foo"],
cmd = "echo $(bar) > $#",
)
$ bazel build foo --define=bar=123
INFO: Analyzed target //:foo (5 packages loaded, 8 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
bazel-bin/foo
INFO: Elapsed time: 0.310s, Critical Path: 0.01s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
$ cat bazel-bin/foo
123
So to have $(BUILD_TAG) work in the genrule, you'll want to pass
--define=BUILD_TAG=the_build_tag
Unless it's that you want BUILD_TAG replaced with literally $(BUILD_TAG), in which case the $ needs to be escaped with another $: $$(BUILD_TAG).
See
https://docs.bazel.build/versions/main/be/general.html#genrule.cmd
https://docs.bazel.build/versions/main/be/make-variables.html
Note that Bazel also has a mechanism for "build stamping" for bringing information like build time and version numbers into the build:
https://docs.bazel.build/versions/main/user-manual.html#workspace_status
https://docs.bazel.build/versions/main/command-line-reference.html#flag--embed_label
Using --workspace_status_command and --embed_label are a little more complicated though.

Related

Erlang Hello World using Bazel as a build system

I want to build an Erlang hello world example using rules_erlang on Ubuntu 22.04.
My setup looks like this:
BUILD.bazel
load("#rules_erlang//:erlang_app.bzl", "erlang_app", "test_erlang_app")
load("#rules_erlang//:xref.bzl", "xref")
load("#rules_erlang//:dialyze.bzl", "dialyze", "plt")
load("#rules_erlang//:ct.bzl", "ct_suite", "assert_suites")
APP_NAME = "hello_world"
APP_VERSION = "0.1.0"
erlang_app(
app_name = APP_NAME,
app_version = APP_VERSION,
)
src/hello_world.erl
-module(hello_world).
-compile(export_all).
hello() ->
io:format("hello world~n").
WORKSPACE.bazel
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "bazel_skylib",
sha256 = "af87959afe497dc8dfd4c6cb66e1279cb98ccc84284619ebfec27d9c09a903de",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.2.0/bazel-skylib-1.2.0.tar.gz",
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.2.0/bazel-skylib-1.2.0.tar.gz",
],
)
load("#bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
bazel_skylib_workspace()
http_archive(
name = "rules_erlang",
sha256 = "5e59800ecc786d5375951028c959c6e6275c94eff2a52f5d53ccb1ad8b2ea20a",
strip_prefix = "rules_erlang-3.8.4",
urls = ["https://github.com/rabbitmq/rules_erlang/archive/refs/tags/3.8.4.zip"],
)
load(
"#rules_erlang//:rules_erlang.bzl",
"erlang_config",
"rules_erlang_dependencies",
)
erlang_config()
rules_erlang_dependencies()
load("#erlang_config//:defaults.bzl", "register_defaults")
register_defaults()
The code can also found here.
When I execute bazel build //... I get
ERROR:
/home/vertexwahn/.cache/bazel/_bazel_vertexwahn/b5f945f94177a8ffa6ac0f7108dfc1cd/external/erlang_config/external/BUILD.bazel:12:16:
Validating otp at /usr failed: (Exit 1): bash failed: error executing
command /bin/bash -c ... (remaining 1 argument skipped)
Use --sandbox_debug to see verbose messages from the sandbox and
retain the sandbox build root for debugging Erlang version mismatch
(Expected UNKNOWN, found 24.2.1)
Any hints to get this working are welcome!
To run it, you can declare a shell rule in your BUILD.bazel, for instance:
load("#rules_erlang//:shell.bzl", "shell")
shell(
name = "repl",
deps = [":erlang_app"],
extra_erl_args = ["-eval", "'hello_world:hello()'"],
)
and then you could bazel run :repl.
Or, you could use an escript rule if you change hello_world.erl so that it exports main/1 instead of hello:
load("#rules_erlang//:escript.bzl", "escript_archive")
escript_archive(
name = "hello_world",
app = ":erlang_app",
)
hello_world.erl:
-module(hello_world).
-export([main/1]).
main(_) ->
io:format("hello world~n").
and run with bazel run :hello_world
bazel build //... --sandbox_debug
gave me
compile: warnings being treated as errors
hello_world.erl:2:2: export_all flag enabled - all functions will be exported
With -export([hello/0]). instead of -compile(export_all). it works
-module(hello_world).
-export([hello/0]).
hello() ->
io:format("hello world~n").
❯ bazel build //...
INFO: Analyzed 3 targets (0 packages loaded, 0 targets configured).
INFO: Found 3 targets...
INFO: Elapsed time: 0,326s, Critical Path: 0,25s
INFO: 2 processes: 1 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 2 total actions

How bazel genrule srcs works under the hood

Well, I'm trying to figure out how srcs works for genrule, I have the following:
genrule(
name = "flutter_build_android",
srcs = [
"//:genfiles"
],
outs = ["android/mobile.apk"],
cmd_bash = "ls -ltr && flutter build apk > $#",
tags = ["local"]
)
//:genfiles is a filegroup using glob:
filegroup(
name = "genfiles",
srcs = glob(["lib/**","assets/**", "pubspec.yaml"])
)
When executing my genrule what I expect is only files under ://genfiles label should be returned, but it is returning all folders under my root project:
What's happening is that the genrule is tagged with local, and local means
precludes the action or test from being remotely cached, remotely executed, or run inside the sandbox
https://bazel.build/reference/be/common-definitions#common.tags
Sandboxing is what prevents an action from seeing files that aren't declared as dependencies (i.e. in srcs here). Without the sandbox, the action sees everything.
$ tree
.
├── BUILD
├── file1
├── file2
├── file3
└── WORKSPACE
0 directories, 5 files
$ cat BUILD
genrule(
name = "gen_foo",
outs = ["foo"],
srcs = ["file1"],
cmd = "echo ----- ; ls ; echo ----- ; wc -l $< > $#",
# tags = ["local"],
)
$ bazel build foo
INFO: Analyzed target //:foo (5 packages loaded, 9 targets configured).
INFO: Found 1 target...
INFO: From Executing genrule //:gen_foo:
-----
bazel-out
external
file1
-----
Target //:foo up-to-date:
bazel-bin/foo
INFO: Elapsed time: 0.379s, Critical Path: 0.02s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
# edit BUILD to restore the local tag
$ cat BUILD
genrule(
name = "gen_foo",
outs = ["foo"],
srcs = ["file1"],
cmd = "echo ----- ; ls ; echo ----- ; wc -l $< > $#",
tags = ["local"],
)
$ bazel build foo
INFO: Analyzed target //:foo (5 packages loaded, 9 targets configured).
INFO: Found 1 target...
INFO: From Executing genrule //:gen_foo:
-----
BUILD
WORKSPACE
bazel-out
external
file1
file2
file3
local-spawn-runner.8714966150718292736
-----
Target //:foo up-to-date:
bazel-bin/foo
INFO: Elapsed time: 0.350s, Critical Path: 0.02s
INFO: 2 processes: 1 internal, 1 local.
INFO: Build completed successfully, 2 total actions

Bazel: map set(files) -> set(targets)

According to https://docs.bazel.build/versions/master/query-how-to.html#What_build_rule_contains_file_ja,
fullname=$(bazel query path/to/file/bar.java)
bazel query "attr('srcs', $fullname, ${fullname//:*/}:*)"
will tell me which target bar.java belongs to.
How can I get the set of targets that multiple files belong to? I.e. map set(files) -> set(targets). I could do this in serial, but each bazel call is fairly expensive and slow---I want to get it done in one call.
Context: I'd like to do this (build targets pertaining to a few files):
git diff HEAD~ | xargs bazel query "get targets for set(files)" | xargs bazel build
I feel like this capability must already exist, but I haven't been able to find it.
Using query will work, but you can also use aquery for a more direct approach.
https://docs.bazel.build/versions/master/aquery.html
BUILD:
genrule(
name = "gen1",
srcs = ["a"],
outs = ["gen1.out"],
cmd = "echo foo > $#",
)
pkg/BUILD:
genrule(
name = "gen2",
srcs = ["b"],
outs = ["gen2.out"],
cmd = "echo foo > $#",
)
$ bazel aquery "inputs('a|pkg/b', ...)" --include_artifacts=false --include_commandline=false
INFO: Analyzed 2 targets (6 packages loaded, 10 targets configured).
INFO: Found 2 targets...
action 'Executing genrule //pkg:gen2'
Mnemonic: Genrule
Target: //pkg:gen2
Configuration: k8-fastbuild
ActionKey: 8d7d05620bfd8303aa66488e0cd6586d8e978197126cdb41c5fc8c49c81988ef
action 'Executing genrule //:gen1'
Mnemonic: Genrule
Target: //:gen1
Configuration: k8-fastbuild
ActionKey: d4c76a6b6913ce5d887829dbc00d101c1cf5b0ff5240ed13ea328c26e4b41960
INFO: Elapsed time: 0.198s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions
inputs (like attr) accepts a regular expression, so you can "or" the files together with |. Then just filter for Target:, or use one of the other outputs (--output=(proto|textproto|jsonproto))
This has the advantages that:
you don't need to figure out the labels of the files first (since attr in query operates on labels).
aquery queries after the analysis phase, so your results are more accurate because it accounts for configuration (flags, etc).
this will work for any attribute, since it's querying the inputs of all actions, which is at a lower level than rules
On the other hand, since aquery runs loading + analysis, it might take longer than query, since query just runs the loading phase.

Bazel: Referencing a file inside an output directory in a genrule

I am trying to reference an output of a rule that is nested inside an output directory of another rule in a genrule.
For example, I use rules_foreign_cc to build boost:
boost_build(
name = "boost",
lib_source = "#boost//:all",
linkopts = [
"-lpthread",
],
shared_libraries = [
"libboost_chrono.so.1.72.0",
"libboost_program_options.so.1.72.0",
"libboost_filesystem.so.1.72.0",
"libboost_system.so.1.72.0",
"libboost_thread.so.1.72.0",
"libboost_timer.so.1.72.0",
],
user_options = [
"cxxstd=17",
"--with-chrono",
"--with-filesystem",
"--with-program_options",
"--with-system",
"--with-thread",
"--with-timer",
"-j4",
],
)
And when I build it, I see the outputs:
bazel build //:boost
INFO: Invocation ID: 36440de3-15f2-4ca0-8802-0a95f75ed926
INFO: Analyzed target //:boost (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:boost up-to-date:
bazel-bin/boost/include
bazel-bin/boost/lib/libboost_chrono.so.1.72.0
bazel-bin/boost/lib/libboost_program_options.so.1.72.0
bazel-bin/boost/lib/libboost_filesystem.so.1.72.0
bazel-bin/boost/lib/libboost_system.so.1.72.0
bazel-bin/boost/lib/libboost_thread.so.1.72.0
bazel-bin/boost/lib/libboost_timer.so.1.72.0
bazel-bin/copy_boost/boost
bazel-bin/boost/logs/BuildBoost_script.sh
bazel-bin/boost/logs/BuildBoost.log
bazel-bin/boost/logs/wrapper_script.sh
INFO: Elapsed time: 0.758s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
Boost works properly, I can reference it in cc_library targets and binaries run fine.
Now, I would like to reference one of the outputs in a genrule. The file I want to reference is nested inside the boost/lib/ directory. I would expect something like: $(location :boost/lib/libboost_program_options.so.1.72.0), but that doesn't work.
What's the proper way to reference the outputs in the directory?
This is what I did, hope it helps.
Inside your genrule, create a variable that hold the output array of your boost rule, use $(locations) instead of $(location). Then loop through the array and find the path you want to use later.
BOOST_OUTPUTS=( $(locations boost) )
LIBBOOST_CHRONO_SO=
for i in "$${{BOOST_OUTPUTS[#]}}"
do
if [[ "$$i" == *"libboost_chrono.so.1.72.0"* ]]; then
LIBBOOST_CHRONO_SO="$$i"
fi
done
# Do something with LIBBOOST_CHRONO_SO
Just note that $$ and {{ and }} are for escaping special characters.

How to see Bazel build output?

How do you view stdout of bazel build as it happens?
I want to see all the logs written to stdout during a bazel build.
No one of these allows it to show the ls command before after it has failed
$ bazel build --show_progress --worker_verbose --verbose_failures --verbose_explanations=true -s --test_output=streamed :build
genrule(
name = "build",
cmd = "ls && sleep 60 && exit 1",
)
$ bazel build --show_progress --worker_verbose --verbose_failures --verbose_explanations=true -s --test_output=streamed :build
WARNING: --verbose_explanations has no effect when --explain=<file> is not enabled
INFO: Analyzed target //:build (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
SUBCOMMAND: # //:build [action 'Executing genrule //:build']
(cd /private/var/tmp/_bazel_kevinsimper/f9e6a72c146c5ad83b84a8ebf539f8b2/execroot/__main__ && \
exec env - \
PATH=/usr/local/sbin \
/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; ls && sleep 60 && exit 1')
ERROR: /Users/kevinsimper/testproject/BUILD:1:1: Executing genrule //:build failed (Exit 1)
BUILD
TESTFILE
Target //:build failed to build
INFO: Elapsed time: 60.256s, Critical Path: 60.04s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
There's no way to stream action stdout/stderr while it's executing, unless it's a test while using the --test_output=streamed flag.
You can follow an underhanded approach. If your build includes small number of lengthy actions, it is feasible to snoop for output in bazel-out/_tmp/actions/std{err,out}-* as it happens. This works for me with Bazel 3.1.0.

Resources