I need to set some build options every time I invoke bazel for a specific target. For example, bazel build --collect_code_coverage //:target. How can I avoid providing the build options at the command line explicitly, so that bazel build //:target implicitly has the build option --collect_code_coverage applied?
The closest solution I found was using the bazelrc file, but it does not allow me to configure build options at a target level.
This is doable with user defined transitions, there are some performance considerations here that are worth reading over. It's also relatively complicated.
# transition.bzl
def _platform_transition_impl(settings, attr):
_ignore = settings
return {"//command_line_option:collect_code_coverage": True}
coverage_transition = transition(
implementation = _coverage_transition_impl,
inputs = [],
outputs = ["//command_line_option:collect_code_coverage"],
)
def _coverage_impl(ctx):
executable = ctx.actions.declare_file(ctx.attr.name)
ctx.actions.symlink(output = executable, target_file = ctx.file.target)
return [DefaultInfo(executable = executable)]
build_with_collect_coverage = rule(
implementation = _coverage_impl,
attrs = {
"target": attr.label(mandatory = True, allow_single_file = True, cfg = coverage_transition),
"_allowlist_function_transition": attr.label(
default = "#bazel_tools//tools/allowlists/function_transition_allowlist",
),
},
)
Then add the transition rule to BUILD.bazel e.g.
load(":transition.bzl", "build_with_collect_coverage")
cc_binary(
name = "target"
#...
)
build_with_collect_coverage(
name = "target_with_coverage_collection",
target = ":target",
)
Note: While this is a generic solution that is incredibly powerful and supports any language you might simply be able to get away with using the features attribute e.g.
cc_binary(
name = "target",
#...
features = ["coverage"],
)
Related
I am having a simple shell script that executes a prebuilt binary. The build file looks like this:
filegroup(
name = "generator_srcs",
srcs = glob([
"configuration/**",
"features/**",
"plugins/**"
]) + [
"commonapi-generator-linux-x86_64.ini",
"artifacts.xml"
],
)
filegroup(
name = "generator_binary",
srcs = ["commonapi-generator-linux-x86_64"],
data = [":generator_srcs"],
)
sh_binary(
name = "generator",
srcs = ["#//tools:generator.sh"],
data = [":generator_binary"],
toolchains = ["#//toolchain:jre_toolchain_definition"],
args = ["$(location :generator_binary)", "$(JAVA)"],
)
However the prebuilt binary depends on a specific Java Runtime Environment. Therefore I simply defined a custom java_runtime that fits the requirements of the binary. The corresponding build file looks like this:
java_runtime(
name = "jre8u181-b13",
srcs = glob([
"jre/**"
]),
java_home = "jre",
licenses = [],
visibility = [
"//visibility:public"
]
)
config_setting(
name = "jre_version_setting",
values = {"java_runtime_version": "1.8"},
visibility = ["//visibility:private"],
)
toolchain(
name = "jre_toolchain_definition",
target_settings = [":jre_version_setting"],
toolchain_type = "#bazel_tools//tools/jdk:runtime_toolchain_type",
toolchain = ":jre8u181-b13",
visibility = ["//visibility:public"],
)
When I am trying to build and run the target generator bazel throws the error:
//toolchain:jre_toolchain_definition does not have mandatory providers: 'TemplateVariableInfo'
This is the point where I am a little bit lost. As stated in this post the rule should provide toolchain specific make variables. Therefore I was looking around in the bazel github repository and found the rule java_runtime_alias which seems to provide some useful variables that I could use in my sh_binary target. But in this rule automatic toolchain resolution happens. I would like to rewrite the rule such that I can hand over my custom toolchain target as an argument but I don't know how. Should I define an attribute?
I have the following rule definition:
helm_action = rule(
attrs = {
…
"cluster_aliases": attr.string_dict(
doc = "key value pair matching for creating a cluster alias where the name used to evoke a cluster alias is different than the actual cluster's name",
default = DEFAULT_CLUSTER_ALIASES,
),
…
},
…
)
I'd like for DEFAULT_CLUSTER_ALIASES value to be based on the host os but
DEFAULT_CLUSTER_ALIASES = {
"local": select({
"#platforms//os:osx": "docker-desktop",
"#platforms//os:linux": "minikube",
})
}
errors with:
Error in string_dict: expected value of type 'string' for dict value element, but got select({"#platforms//os:osx": "docker-desktop", "#platforms//os:linux": "minikube"}) (select)
How do I go about defining DEFAULT_CLUSTER_ALIASES based on the host os?
Judging from https://github.com/bazelbuild/bazel/issues/2045, selecting based on host os is not possible.
When you create a rule or macro, it is evaluated during the loading phase, before command-line flags are evaluated. Bazel needs to know the default value in your build rule helm_action during the loading phase but can't because it hasn't parsed the command line and analysed the build graph.
The command line is parsed and select statements are evaluated during the analysis phase. As a broad rule, if your select statement isn't in a BUILD.bazel then it's not going to work. So the easiest way to achieve what you are after is to create a macro that uses your rule injecting the default. e.g.
# helm_action.bzl
# Add an '_' prefix to your rule to make the rule private.
_helm_action = rule(
attrs = {
…
"cluster_aliases": attr.string_dict(
doc = "key value pair matching for creating a cluster alias where the name used to evoke a cluster alias is different than the actual cluster's name",
# Remove default attribute.
),
…
},
…
)
# Wrap your rule in a publicly exported macro.
def helm_action(**kwargs):
_helm_action(
name = kwargs["name"],
# Instantiate your rule with a select.
cluster_aliases = DEFAULT_CLUSTER_ALIASES,
**kwargs,
)
It's important to note the difference between a macro and a rule. A macro is a way of generating a set of targets using other build rules, and actually expands out roughly equivalent to it's contents when used in a BUILD file. You can check this by querying a target with the --output build flag. e.g.
load(":helm_action.bzl", "helm_action")
helm_action(
name = "foo",
# ...
)
You can query the output using the command;
bazel query //:foo --output build
This will demonstrate that the select statement is being copied into the BUILD file.
A good example of this approach is in the rules_docker repository.
EDIT: The question was clarified, so I've got an updated answer below but will keep the above answer in case it is useful to others.
A simple way of achieving what you are after is to use Bazels toolchain api. This is a very flexible API and is what most language rulesets use in Bazel. e.g.
Create a build file with your toolchains;
# //helm:BUILD.bazel
load(":helm_toolchains.bzl", "helm_toolchain")
toolchain_type(name = "toolchain_type")
helm_toolchain(
name = "osx",
cluster_aliases = {
"local": "docker-desktop",
},
)
toolchain(
name = "osx_toolchain",
toolchain = ":osx",
toolchain_type = ":toolchain_type",
exec_compatible_with = ["#platforms//os:macos"],
# Optionally use to restrict target platforms too.
# target_compatible_with = []
)
helm_toolchain(
name = "linux",
cluster_aliases = {
"local": "minikube",
},
)
toolchain(
name = "linux_toolchain",
toolchain = ":linux",
toolchain_type = ":toolchain_type",
exec_compatible_with = ["#platforms//os:linux"],
)
Register your toolchains so that Bazel knows what to look for;
# //:WORKSPACE
# the rest of your workspace...
register_toolchains("//helm:all")
# You may need to register your execution platforms too...
# register_execution_platforms("//your_platforms/...")
Implement the toolchain backend;
# //helm:helm_toolchains.bzl
HelmToolchainInfo = provider(fields = ["cluster_aliases"])
def _helm_toolchain_impl(ctx):
toolchain_info = platform_common.ToolchainInfo(
helm_toolchain_info = HelmToolchainInfo(
cluster_aliases = ctx.attr.cluster_aliases,
),
)
return [toolchain_info]
helm_toolchain = rule(
implementation = _helm_toolchain_impl,
attrs = {
"cluster_aliases": attr.string_dict(),
},
)
Update helm_action to use toolchains. e.g.
def _helm_action_impl(ctx):
cluster_aliases = ctx.toolchains["#your_repo//helm:toolchain_type"].helm_toolchain_info.cluster_aliases
#...
helm_action = rule(
_helm_action_impl,
attrs = {
#…
},
toolchains = ["#your_repo//helm:toolchain_type"]
)
I've a c target that always must be compiled for darwin_x86_64, no matter the --cpu set when calling bazel build. It's the only target that always must be compiled for a specific cpu in a big project.
cc_binary(
name = "hello-world",
srcs = ["hello-world.cc"],
)
In the bazel documentation it seems to be possible to do this using transitions. Maybe something like:
def _force_x86_64_impl(settings, attr):
return {"//command_line_option:cpu": "darwin_x86_64"}
force_x86_64 = transition(
implementation = _force_x86_64_impl,
inputs = [],
outputs = ["//command_line_option:cpu"]
)
But how do I tie these two together? I'm probably missing something obvious, but I can't seem to find the relevant documentation over at bazel.build.
You've got the transition defined, now you need to attach it to a rule.
r = rule(
implementation = _r_impl,
attrs = {
"_allowlist_function_transition": attr.label(
default = "#bazel_tools//tools/allowlists/function_transition_allowlist",
),
"srcs": attr.label(cfg = force_x86_64),
},
)
def _r_impl(ctx):
return [DefaultInfo(files = ctx.attr.srcs[0][DefaultInfo].files)]
This defines a rule that attaches the appropriate transition to the srcs you pass it, and then simply forwards the list of files from DefaultInfo. Depending on what you're doing, this might be sufficient, or you might also want to forward the runfiles contained in DefaultInfo and/or other providers.
Use target_compatible_with
Example (only compiled on Windows):
cc_library(
name = "win_driver_lib",
srcs = ["win_driver_lib.cc"],
target_compatible_with = [
"#platforms//cpu:x86_64",
"#platforms//os:windows",
],
)
I have a project that involves multiple BUILD files in a single WORKSPACE, within a fairly complex build system. My goal in short: for some specific target, I want all of its recursive dependencies to be built with an extra set of attributes (copts/defines) compared to when those dependency targets are built in any other way. I have not yet found a way to do this cleanly.
For example, target G is normally built with copts = []. If target P depends on target G, and I run bazel build :P, I want both targets to be built with copts = ["-DMY_DEFINE"], along with all dependencies of target G, etc.
The cc_binary.defines argument propagates in the opposite direction: all targets that depend on some target A will receive all of target A's defines.
Limitations:
prefer to avoid custom command line flags, I don't control how people call bazel {build,test}
duplicating the entire tree of dependency targets is not practical
It doesn't appear possible to set the value of a config_setting from within a BUILD file or a target, so it seems a select-based solution couldn't work.
Previous work:
https://groups.google.com/g/bazel-discuss/c/rZps4nqYqt8/m/YS_pZD6oAQAJ - 2017, recommends "parallel trees" or custom macros (of which we already have many, it would be challenging to wrap them in another)
Propagate copts to all dependencies in Bazel - I believe these all depend on custom command line flags as well
Creating a user-defined build setting doesn't require command-line flags. If you set flag = False, then it actually can't be set on the command line. You can use a user-defined transition to set it instead.
I think something like this will do what you're looking for (save it in extra_copts.bzl):
def _extra_copts_impl(ctx):
context = cc_common.create_compilation_context(
defines = depset(ctx.build_setting_value)
)
return [CcInfo(compilation_context = context)]
extra_copts = rule(
implementation = _extra_copts_impl,
build_setting = config.string_list(flag = False),
)
def _use_extra_copts_implementation(ctx):
return [ctx.attr._copts[CcInfo]]
use_extra_copts = rule(
implementation = _use_extra_copts_implementation,
attrs = "_copts": attr.label(default = "//:extra_copts")},
)
def _add_copts_impl(settings, attr):
return {"//:extra_copts": ["MY_DEFINE"]}
_add_copts = transition(
implementation = _add_copts_impl,
inputs = [],
outputs = ["//:extra_copts"],
)
def _with_extra_copts_implementation(ctx):
infos = [d[CcInfo] for d in ctx.attr.deps]
return [cc_common.merge_cc_infos(cc_infos = infos)]
with_extra_copts = rule(
implementation = _with_extra_copts_implementation,
attrs = {
"deps": attr.label_list(cfg = _add_copts),
"_allowlist_function_transition": attr.label(
default = "#bazel_tools//tools/allowlists/function_transition_allowlist"
)
},
)
and then in the BUILD file:
load("//:extra_copts.bzl", "extra_copts", "use_extra_copts", "with_extra_copts")
extra_copts(name = "extra_copts", build_setting_default = [])
use_extra_copts(name = "use_extra_copts")
cc_library(
name = "G",
deps = [":use_extra_copts"],
)
with_extra_copts(
name = "P_deps",
deps = [":G"],
)
cc_library(
name = "P",
deps = [":P_deps"],
)
extra_copts is the build setting. It returns a CcInfo directly, which means it's straightforward to do any other C++ library swapping with the same approach. Its default is effectively an "empty" CcInfo which won't do anything to libraries that depend on it.
with_extra_copts wraps a set of dependencies, configured to use a different CcInfo. This is the rule that actually changes the value, to create the second version of G with different flags.
_add_copts is the transition which with_extra_copts uses to change the value of the extra_copts build setting. It could examine attr to do something more sophisticated than adding a hard-coded list.
use_extra_copts pulls the CcInfo out of extra_copts so a cc_library can use them.
To avoid rewriting the builtin C++ rules, this uses wrapper rules to pull the copts out and do the transition. You might want to create macros to bundle the wrapper rules along with the corresponding cc_library. Alternatively, you could use rules_cc's my_c_archive as a starting point to create custom rules that reuse the core implementation of the builtin C++ rules while integrating the transition and use of the build setting into a single rule.
I have a test tool (roughly, a diffing tool) that takes two inputs, and returns both an output (the difference between the two inputs), and a return code (0 if the two inputs are matching, 1 otherwise). It's built in Kotlin, and available at //java/fr/enoent/phosphorus in my repo.
I want to write a rule that tests that a file generated by something is identical to the reference file already present in the repository. I tried something with ctx.actions.run, the problem being that my rule, having test = True set, needs to return an executable built by that rule (so not a tool provided to the rule). I then tried to wrap it in a shell script following the example, like this:
def _phosphorus_test_impl(ctx):
output = ctx.actions.declare_file("{name}.phs".format(name = ctx.label.name))
script = phosphorus_compare(
ctx,
reference = ctx.file.reference,
comparison = ctx.file.comparison,
out = output,
)
ctx.actions.write(
output = ctx.outputs.executable,
content = script,
)
runfiles = ctx.runfiles(files = [ctx.executable._phosphorus_tool, ctx.file.reference, ctx.file.comparison])
return [DefaultInfo(runfiles = runfiles)]
phosphorus_test = rule(
_phosphorus_test_impl,
attrs = {
"comparison": attr.label(
allow_single_file = [".phs"],
doc = "File to compare to the reference",
mandatory = True,
),
"reference": attr.label(
allow_single_file = [".phs"],
doc = "Reference file",
mandatory = True,
),
"_phosphorus_tool": attr.label(
default = "//java/fr/enoent/phosphorus",
executable = True,
cfg = "host",
),
},
doc = "Compares two files, and fails if they are different.",
test = True,
)
(phosphorus_compare is just a macro generating the actual command.)
However, this approach has two issues:
The output can't be declared this way. It's not linked to any action (and Bazel is complaining about it). Maybe I don't really need to declare an output for a test? Does Bazel make anything in the test folder available when the test fails?
The runfiles necessary to run the tool don't seem to be available when the test runs:
java/fr/enoent/phosphorus/phosphorus: line 359: /home/kernald/.cache/bazel/_bazel_kernald/58c025fbb926eac6827117ef80f7d2fa/sandbox/linux-sandbox/1979/execroot/fr_enoent/bazel-out/k8-fastbuild/bin/tools/phosphorus/tests/should_pass.runfiles/remotejdk11_linux/bin/java: No such file or directory
Overall I feel like using a shell script is just adding an unnecessary indirection, and losing some context (e.g. tools' runfiles). Ideally, I would just use ctx.actions.run and rely on its return code, but it doesn't seem to be an option as a test apparently needs to generate an executable. What would be the correct approach to write such a rule?
Turns out, generating a script is the correct approach, it's (as far as I understood) impossible to return some kind of pointer to a ctx.actions.run. A test rule needs to have an executable output.
Regarding the output file that the tool is generating: there's no need to declare it, at all. I just need to make sure that it's generated in $TEST_UNDECLARED_OUTPUTS_DIR. Every single file in this directory will be added to an archive called output.zip by Bazel. This is (partly) documented here.
Concerning the runfiles, well, I had the tool's binary, but not its own runfiles. Here is the fixed rule:
def _phosphorus_test_impl(ctx):
script = phosphorus_compare(
ctx,
reference = ctx.file.reference,
comparison = ctx.file.comparison,
out = "%s.phs" % ctx.label.name,
)
ctx.actions.write(
output = ctx.outputs.executable,
content = script,
)
return [
DefaultInfo(
runfiles = ctx.runfiles(
files = [
ctx.executable._phosphorus_tool,
ctx.file.reference,
ctx.file.comparison,
],
).merge(ctx.attr._phosphorus_tool[DefaultInfo].default_runfiles),
executable = ctx.outputs.executable,
),
]
def phosphorus_test(size = "small", **kwargs):
_phosphorus_test(size = size, **kwargs)
_phosphorus_test = rule(
_phosphorus_test_impl,
attrs = {
"comparison": attr.label(
allow_single_file = [".phs"],
doc = "File to compare to the reference",
mandatory = True,
),
"reference": attr.label(
allow_single_file = [".phs"],
doc = "Reference file",
mandatory = True,
),
"_phosphorus_tool": attr.label(
default = "//java/fr/enoent/phosphorus",
executable = True,
cfg = "target",
),
},
doc = "Compares two files, and fails if they are different.",
test = True,
)
The key part being .merge(ctx.attr._phosphorus_tool[DefaultInfo].default_runfiles) in the returned DefaultInfo.
I also made a small mistake about the configuration, as this test is intended to run on the target configuration, not host, it's been fixed accordingly.