a project I'm working on -- Envoy proxy -- uses Bazel and tcmalloc. I'd like to configure it to use the debug version of tcmalloc when compiling for debug and fastbuild, and use the optimized one for optimized builds.
There are other conditions as well, e.g. a command-line flag passed to bazel to turn off tcmalloc completely, using this logic:
https://github.com/envoyproxy/envoy/blob/7d2e84d3d0f8a4ffbf4257c450b3e5a6d93d4697/bazel/envoy_build_system.bzl#L166
def tcmalloc_external_dep(repository):
return select({
repository + "//bazel:disable_tcmalloc": None,
"//conditions:default": envoy_external_dep_path("tcmalloc_and_profiler"),
})
I have PR out (https://github.com/envoyproxy/envoy/pull/5424) failing continuous integration which changes the logic (https://github.com/envoyproxy/envoy/blob/1ed5aba5894ce519181edbdaee3f52c2971befaf/bazel/envoy_build_system.bzl#L156) to:
def tcmalloc_external_dep(repository):
return select({
repository + "//bazel:disable_tcmalloc": None,
repository + "//bazel:dbg_build": envoy_external_dep_path("tcmalloc_debug"),
"//conditions:default": envoy_external_dep_path("tcmalloc_and_profiler"),
})
However this does not work as we allow disabling tcmalloc on debug builds (which we do in continuous-integration scripts when running tsan). This runs afoul of bazel which evidently expects the conditions to be mutually exclusive, when I want "first matching rule wins" in this case. I get this error:
ERROR: /home/jmarantz/git4/envoy/test/common/network/BUILD:58:1: Illegal ambiguous match on configurable attribute "malloc" in //test/common/network:dns_impl_test:
//bazel:disable_tcmalloc
//bazel:dbg_build
Multiple matches are not allowed unless one is unambiguously more specialized.
ERROR: Analysis of target '//test/common/network:dns_impl_test' failed; build aborted:
/home/jmarantz/git4/envoy/test/common/network/BUILD:58:1: Illegal ambiguous match on configurable attribute "malloc" in //test/common/network:dns_impl_test:
//bazel:disable_tcmalloc
//bazel:dbg_build
What's the best way to solve this? Can I use a Python conditional on the bazel command-line settings? Can I use AND or OR operators in the conditional expressions to make them mutually exclusive? Or is there another approach I could use?
Not an answer, but perhaps I can give you some ideas:
As of now, you can simulate and and or by nesting selects or refactoring your config_settings.
There is a proposal for some changes to add flexibility here:
https://github.com/bazelbuild/proposals/blob/master/designs/2018-11-09-config-setting-chaining.md
You might also find some useful ideas in Skylib.
https://github.com/bazelbuild/bazel-skylib
Yup you can chain select using https://github.com/bazelbuild/bazel-skylib/blob/master/lib/selects.bzl#L80. You can also write your own feature flag rule that can be used in the select and that has artibrary logic in it, see https://source.bazel.build/bazel/+/0faef9148362a5234df3507441dadb0f32ade59a:tools/cpp/compiler_flag.bzl for example, it's a rule that can be used in selects and that gets the current C++ toolchain and inspects its state and returns its compiler value. You'll have to follow the thread a bit to see all the pieces. I'll ask for better docs for this.
Related
I've set up my bazel crosstool so that I can specifically select the compiler that I want: gcc9, gcc10, ..., clang12, clang13... This works great.
bazel build --compiler=clang13 //:target
I'm scratching my head wondering how I achieve this with platforms! It seems to want to select whatever compiler you specify for the given platform, and if you want to change it, you have to edit the file!
In particular, if I want my compiler to be used by dependencies, whatever I do needs to be compatible with, for example, absl, and grpc.
Is there anyway to coerce toolchain selection via --config, --define, or other flags?
# In Workspace
register_toolchains("//toolchains:gcc12",
"//toolchains:clang13",
"//toolchains:clang14",
...)
# But how do I tell it that I want clang13, or clang14???
bazel build --platform=linux_x86 //:target
Here are two ideas that could help you:
do not use register_toolchains() to make all toolchains known to bazel, but use https://bazel.build/reference/command-line-reference#flag--extra_toolchains (maybe based on a --config via the .bazelrc). This lets bazel only know about one compiler toolchain available for resolution. Of course with this approach you can't use different compiler toolchains for different targets.
make use of the constraint_setting()s defined here. https://bazel.build/configure/windows#clang references how this is done:
platform(
name = "x64_windows-clang-cl",
constraint_values = [
"#platforms//cpu:x86_64",
"#platforms//os:windows",
"#bazel_tools//tools/cpp:clang-cl",
],
)
I have a repo which uses bazel to build a bunch of Python code. I would like to introduce various flavors of static analysis into the build and have the build fail if these static analyses throw errors. What is the best way to do this?
For example, I'd like to declare something like:
py_library_with_static_analysis(
name = "foo",
srcs = ["foo.py"],
)
py_library_with_static_analysis(
name = "bar",
srcs = ["bar.py"],
deps = [":foo"],
)
In a build file and have it error out if there are mypy/flake/etc errors in foo.py. I would like to be able to do this gradually, converting libraries/binaries to static analysis one target at a time. I'm not sure if I should do this via a new rule, a macro, an aspect or something else.
Essentially, I think I'm asking how to run an additional command while building a py_binary/py_library and fail if that command fails.
I could create my own version of a py_library rule and have it run static analysis within the implementation but that seems like something which is really easy to get wrong (my guess is that native.py_library is quite complex?) and there doesn't seem to be a way to instantiate a native.py_library within a custom rule.
I've also played around with macros a bit, but haven't been able to get that to work either. I think my issue there is that a macro doesn't actually specify new commands, only new targets and I can't figure out how to make the static analysis target get force built along with the py_library/py_binary I'm interested in.
A macro that adds implicit test targets is not such a bad idea: The test targets will be picked up automatically when you run bazel test //..., which you could do in a gating CI to prevent imperfect code from merging.
Bazel supports a BUILD prelude (which is underdocumented) that you could use to replace all py_binary, py_library, and even py_test with your test-adding wrapper macros with minimal changes to existing code.
If you somehow fail the build instead it will make it harder to quickly prototype things. Sometimes you want to just quickly try something out, and you don't care about any pydoc violations yet.
In case you do want to fail the build, you might be able to use the Validations Output Group of a rule that you implement to wrap or replace your py_libraries.
I am writing a custom rule that takes inputs from cc_library, cc_binary, apple_static_library, and a few other platform-specific rules. I'd like to view each API given to me via referencing ctx.attr.foo inside the custom rule's implementation function.
There is a list of providers here https://docs.bazel.build/versions/master/skylark/lib/skylark-provider.html but it doesn't say which rules are using them.
Is there a best practice for viewing what these rules are providing me, or does it require going through the source for each one?
This is how you get all providers and output groups from a target:
bazel cquery my_target --output=starlark --starlark:expr="providers(target)"
You can get a list of providers for a given target with dir. Something like this is helpful for debugging:
def _print_attrs_impl(ctx):
for target in ctx.attr.targets:
print('%s: %s' % (target.label, dir(target)))
Printing from inside a rule you're developing is often helpful too, to verify targets are actually what you expect them to be.
You can also apply dir to the providers themselves, to see what fields they have.
Let's say I have a rule like this.
foo(
name = "helloworld",
myarray = [
":bar",
"//path/to:qux",
],
)
In this case, myarray is static.
However, I want it to be given by cli, like
bazel run //:helloworld --myarray=":bar,//path/to:qux,:baz,:another"
How is this possible?
Thanks
To get exactly what you're asking for, Bazel would need to support LABEL_LIST in Starlark-defined command line flags, which are documented here:
https://docs.bazel.build/versions/2.1.0/skylark/lib/config.html
and here: https://docs.bazel.build/versions/2.1.0/skylark/config.html
Unfortunately that's not implemented at the moment.
If you don't actually need a list of labels (i.e., to create dependencies between targets), then maybe STRING_LIST will work for you.
If you do need a list of labels, and the different possible values are known, then you can use --define, config_setting(), and select():
https://docs.bazel.build/versions/2.1.0/configurable-attributes.html
The question is, what are you really after. Passing variable, array into the bazel build/run isn't really possible, well not as such and not (mostly) without (very likely unwanted) side effects. Aren't you perhaps really just looking into passing arguments directly to what is being run by the run? I.e. pass it to the executable itself, not bazel?
There are few ways you could sneak stuff in (you'd also in most cases need to come up with a syntax to pass data on CLI and unpack the array in a rule), but many come with relatively substantial price.
You can define your array in a bzl file and load it from where the rule uses it. You can then dump the bzl content rewriting your build/run configuration (also making it obvious, traceable) and load the bits from the rule (only affecting the rule loading and using the variable). E.g, BUILD file:
load(":myarray.bzl", "myarray")
foo(
name = "helloworld",
myarray = myarray,
],
)
And you can then call your build:
$ echo 'myarray=[":bar", "//path/to:qux", ":baz", ":another"]' > myarray.bzl
$ bazel run //:helloworld
Which you can of course put in a single wrapper script. If this really needs to be a bazel array, this one is probably the cleanest way to do that.
--workspace_status_command: you can collection information about your environment, add either or both of the resulting files (depending on whether the inputs are meant to invalidate the rule results or not, you could use volatile or stable status files) as a dependency of your rule and process the incoming file in the what is being executed by the rule (at which point one would wonder why not pass it to as its command line arguments directly). If using stable status file, also each other rule depending on it is invalidated by any change.
You can do similar thing by using --action_env. From within the executable/tool/script underpinning the rule, you can directly access defined environmental variable. However, this also means environment of each rule is affected (not just the one you're targeting); and again, why would it parse the information from environment and not accept arguments on the command line.
There is also --define, but you would not really get direct access it's value as much as you could select() a choice out of possible options.
I was wondering if its possible for platform-specific default Bazel build flags.
For example, we want to use --workspace_status_command but this must be a shell script on Linux and must point towards a batch script for Windows.
Is there a way we can write in the tools/bazel.rc file something like...
if platform=WINDOWS build: --workspace_status_command=status_command.bat
if platform=LINUX build: --workspace_status_command=status_command.sh
We could generate a .bazelrc file by having the users run a script before building, but it would be cleaner/nicer if this was not neccessary.
Yes, kind of. You can specify config-specific bazelrc entries, which you can select by passing --config=<configname>.
For example your bazelrc could look like:
build:linux --cpu=k8
build:linux --workspace_status_command=/path/to/command.sh
build:windows --cpu=x64_windows
build:windows --workspace_status_command=c:/path/to/command.bat
And you'd build like so:
bazel build --config=linux //path/to:target
or:
bazel build --config=windows //path/to:target
You have to be careful not to mix semantically conflicting --config flags (Bazel doesn't prevent you from that). Though it will work, the results may be unpredictable when the configs tinker with the same flags.
Passing --config to all commands is tricky, it depends on developers remembering to do this, or controlling the places where Bazel is called.
I think a better answer would be to teach the version control system how to produce the values, like by putting a git-bazel-stamp script on the $PATH/%PATH% so that git bazel-stamp works.
Then we need workspace_status_command to allow commands from the PATH rather than a path on disk.
Proper way to do this is to wrap your cc_library with a custom macro, and pass hardcoded flags to copts. For full reference, look at envoy_library.bzl.
In short, your steps:
Define a macro to wrap cc_library:
def my_cc_library(
name,
copts=[],
**kwargs):
cc_library(name, copts=copts + my_flags(), **kwargs)
Define my_flags() macro as following:
config_setting(
name = "windows_x86_64",
values = {"cpu": "x64_windows"},
)
config_setting(
name = "linux_k8",
values = {"cpu": "k8"},
)
def my_flags():
x64_windows_options = ["/W4"]
k8_options = ["-Wall"]
return select({
":windows_x86_64": x64_windows_options,
":linux_k8": k8_options,
"//conditions:default": [],
})
How it works:
Depending on --cpu flag value my_flags() will return different flags.
This value is resolved automatically based on a platform. On Windows, it's x64_windows, and on Linux it's k8.
Then, your macro my_cc_library will supply this flags to every target in a project.
A better way of doing this has been added since you asked--sometime in 2019.
If you add
common --enable_platform_specific_config to your .bazelrc, then --config=windows will automatically apply on windows hosts, --config=macos on mac, --config=linux on linux, etc.
You can then add lines to your .bazelrc like:
build:windows --windows-flags
build:linux --linux-flags
There is one downside, though. This works based on the host rather than the target. So if you're cross-compiling, e.g. to mobile, and want different flags there, you'll have to go with a solution like envoy's (see other answer), or (probably better) add transitions into your graph targets. (See discussion here and here. "Flagless builds" are still under development, but there are usable hacks in the meantime.) You could also use the temporary platform_mappings API.
References:
Commit that added this functionality.
Where it appears in the Bazel docs.