How to invoke CROSSTOOL tools from Bazel macros / rules? - bazel

I'm building ARM Cortex-M firmware from Bazel with a custom CROSSTOOL. I'm successfully building elf files and manually objcopying them to binary files with the usual:
path/to/my/objcopy -o binary hello.elf hello.bin
I want to make a Bazel macro or rule called cc_firmware that:
Adds the -Wl,-Map=hello.map flags to generate a mapfile
Changes the output elf name from hello to hello.elf
Invokes path/to/my/objcopy to convert the elf to a bin.
I don't know how to get the name of a CROSSTOOL tool (objcopy) to invoke it, and it feels wrong to have the rule know the path to the tool executable.
Is there a way to use the objcopy that I've already told Bazel about in my CROSSTOOL file?

You can actually access this from a custom rule. Basically you need to tell Bazel that you want access to the cpp configuration information (fragments = ["cpp"]) and then access its path via ctx.fragments.cpp.objcopy_executable, e.g.,:
def _impl(ctx):
print("path: {}".format(ctx.fragments.cpp.objcopy_executable))
# TODO: actually do something with the path...
cc_firmware = rule(
implementation = _impl,
fragments = ["cpp"],
attrs = {
"src" : attr.label(allow_single_file = True),
"map" : attr.label(allow_single_file = True),
},
outputs = {"elf" : "%{name}.elf"}
)
Then create the output you want with something like (untested):
def _impl(ctx):
src = ctx.attr.src.files.to_list()[0]
m = ctx.attr.map.files.to_list()[0]
ctx.action(
command = "{objcopy} -Wl,-Map={map} -o binary {elf_out} {cc_bin}".format(
objcopy=ctx.fragments.cpp.objcopy_executable,
map=m,
elf_out=ctx.outputs.elf.path,
cc_bin=src,
),
outputs = [ctx.outputs.elf],
inputs = [src, m],
)

Related

Instantiating a Bazel macro twice with same generated output file

Suppose I have a Bazel macro that is using a generator rule to generate an output file given an input file:
def my_generator(
name,
input_file,
output_file,
**kwargs):
args = []
args.extend(["--arg1", "$(location %s)" % output_file])
args.extend(["arg2", "$(locations %s)" % input_file])
cmd_params = " ".join(args)
native.genrule(
name = name,
srcs = [input_file],
outs = [output_file],
cmd = "python $(location //path/to:target_generator) %s" % cmd_params,
tools = ["/path/to/tool:mytool"],
)
Then I was previously using this macro as:
my_generator(
name = "gen1",
input_file = ":targetToGeneratetextFile",
output_file = "outputfile.txt",
visibility = ["//myproject/oath/to/current/package/test:__subpackages__"],
)
where a target is passed as input_file. This was working.
Then I wanted to reuse it with a different input but to generate the same output, where the input is now a file within the project but in another folder.
my_generator(
name = "gen2",
input_file = "//path/to/the/file/realFile.txt",
output_file = "outputfile.txt",
visibility = ["//myproject/oath/to/current/package/test:__subpackages__"],
)
I am getting two errors in this way:
For how it is, Bazel cannot find the realFile.txt: it tries to read it as a target:
no such package '//path/to/the/file/realFile.txt': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package
If I copy the file in the current package folder, it is able to read it.
Bazel is complaining that gen1 and gen2 are writing/overwriting the same output file outputfile.txt:
Error in genrule: generated file 'outputfile.txt' in rule 'gen2' conflicts with existing generated file from rule 'gen1', defined at ...
How can I solve these issues?
I think that the problem is that these two calls are both executed, whereas I would like them to be executed depending on some target, i.e., target A needs only run gen1 and target B gen2 exclusively. I do not if that is possible but for example moving each of these call inside the target they belong to might be a solution that avoids this issue.
EDIT
I was thinking as solution to do something like:
my_generator(
name = "gen2",
input_file = select({
":opt1": [":targetToGeneratetextFile"],
":opt2": ["realTextFile.txt"],
"//conditions:default": [":targetToGeneratetextFile"],
}),
output_file = "outputfile.txt",
visibility = ["//myproject/oath/to/current/package/test:__subpackages__"],
)
with proper config_setting and then call it from the target with the proper flag but I am getting the error:
expected value of type 'string' for element 0 of attribute 'srcs' in 'genrule' rule, but got select({":opt1": [":targetToGeneratetextFile"], ":opt2": ["realTextFile.txt"],"//conditions:default": [":targetToGeneratetextFile"],
})
The label //path/to/the/file/realFile.txt is shorthand for //path/to/the/file/realFile.txt:realFile.txt, aka <repository root>/path/to/the/file/realFile.txt/realFile.txt. Depending on where the deepest-nested folder with a BUILD file is (which determines the package), you're looking for something like //path/to/the/file:realFile.txt or //path/to:the/file/realFile.txt instead.
You can't have two rules which write the same file, because then Bazel can't tell which way to build it if you bazel build the file. Some alternatives:
Put them in separate packages (aka separate folders with BUILD files)
Name them differently, like gen1_outputfile.txt and gen2_outputfile.txt, or gen1/outputfile.txt and gen2/outputfile.txt. You could automate this in the macro like srcs = [name + '/outputfile.txt'].
Use a single rule to generate it with an appropriate select, like your edit.
With the select, you're trying to create something like this:
genrule(
srcs = select({..., "//conditions:default": [":targetToGeneratetextFile"]}),
...
)
but as written you have this instead:
genrule(
srcs = [select({..., "//conditions:default": [":targetToGeneratetextFile"]})],
...
)
Effectively, between the list in the select's value and the macro body, you're creating a nested list. I would change your macro argument to input_files and then do srcs = input_files in the body, so the caller of the macro can bundle things into lists as desired.

How can I specify the expected file extension generated by bazel cc_binary rule?

By default, the cc_binary rule of bazel produces an output file without any extension on Linux.
My compiler generates a .s19 file extension as output(I have extended the toolchain).
Is there a way to specify the output file's extension?
I get a "Linking 'App-name' failed: not all outputs were created or valid" although the expected output file 'App-name.s19' is generated.
My second question is:
In addition to an 'App-name.s19' file my compiler also generates a 'App-name.map' file. Is there a way to tell bazel to verify both 'App-name.s19' and 'App-name.map' files. i.e. verify multiple outputs generated by cc_binary.
Question 1)
When configuring your toolchain you can load: artifact_name_pattern from #bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl and set artifact_name_patterns attribute of cc_common.create_cc_toolchain_config_info():
artifact_name_patterns = [
artifact_name_pattern(
category_name = "executable",
prefix = "",
extension = ".s19",
),]
Question 2)
It seems that cc_binary doesn't support yet map files https://github.com/bazelbuild/bazel/issues/6718
A possible workaround would be to use a genrule in your BUILD file to compile your code and generate the mapfile,
genrule(
name = "map",
srcs = ["hello-world.cc"],
outs = ["hello-world.exe","output.map"],
output_to_bindir = 1,
cmd= "/usr/bin/x86_64-w64-mingw32-gcc -o $(location hello-world.exe) $(location hello-world.cc) -Wl,-Map=\"$(location output.map)\" -lstdc++",
)
or alternatively https://groups.google.com/g/bazel-discuss/c/A00d7Ui1f8s/m/vybgGEPIBwAJ

Getting runtime path to a package in Bazel toolchain configuration files

What is the best way to refer to an external package's path in any arbitrary files processed by Bazel?
I'm trying to understand how Bazel preprocesses BUILD and .bzl files. I see instances where strings contain calls to package() and I am wondering how it works (and could not find any relevant documentation). Here is an example of this:
I have a toolchain which BUILD file contains the following expression :
cc_toolchain_config(
name = "cc-toolchain-config",
abi_libc_version = "glibc_" + host_gcc8_bundle()["pkg_version"]["glibc"],
abi_version = "gcc-" + host_gcc8_bundle()["version"],
compiler = "gcc-" + host_gcc8_bundle()["version"],
cpu = "x86_64",
cxx_builtin_include_directories = [
"%package(#host_gcc8_toolchain//include/c++/8)%",
"%package(#host_gcc8_toolchain//lib64/gcc/x86_64-unknown-linux-gnu/8/include-fixed)%",
"%package(#host_gcc8_kernel_headers//include)%",
"%package(#host_gcc8_glibc//include)%",
],
host_system_name = "x86_64-unknown-linux-gnu",
target_libc = "glibc_" + host_gcc8_bundle()["pkg_version"]["glibc"],
target_system_name = "x86_64-unknown-linux-gnu",
toolchain_identifier = "host_linux_gcc8",
)
From my understanding, the cxx_builtin_include_directories defines a list of strings to serve as the --sysroot option passed to GCC as detailed in https://docs.bazel.build/versions/0.23.0/skylark/lib/cc_common.html These strings are in the format %sysroot%.
Since package(#host_gcc8_toolchain//include/c++/8) for example, does not mean anything to GCC, bazel has to somehow expand this function to produce the actual path to the files included in the package before passing them to the compiler driver.
But how can it determine that this needs to be expanded and that it is not a regular string ? So how does Bazel preprocess the BUILD file ? Is it because of the % ... % pattern ? Where is this documented ?
is "%package(#external_package//target)%" a pattern that can be used elsewhere ? In any BUILD file ? Where do I find Bazel documentation showing how this works ?
These directives are expanded by cc_common.create_cc_toolchain_config_info within the cc_toolchain_config rule implementation not any sort of preprocessing on the BUILD file (I.e., "%package(#host_gcc8_glibc//include)%" is literally passed into the cc_toolchain_config rule.) I'm not aware that these special expansions are completely documented anywhere but the source.

How to generate cc_library from an output directory from a genrule?

I have a binary that takes as input a single file and produces an unknown number of header and source C++ files into a single directory. I would like to be able to write a target like:
x_library(
name = "my_x_library",
src = "source.x",
)
where x_library is a macro that ultimately produces the cc_library from the output files. However, I can't bundle all the output files inside the rule implementation or inside the macro. I tried this answer but it doesn't seem to work anymore.
What's the common solution to this problem? Is it possible at all?
Small example of a macro using a genrule (not a huge fan) to get one C file and one header and provide them as a cc_library:
def x_library(name, src):
srcfile = "{}.c".format(name)
hdrfile = "{}.h".format(name)
native.genrule(
name = "files_{}".format(name),
srcs = [src],
outs = [srcfile, hdrfile],
cmd = "./generator.sh $< $(OUTS)",
tools = ["generator.sh"],
)
native.cc_library(
name = name,
srcs = [srcfile],
hdrs = [hdrfile],
)
Used it like this then:
load(":myfile.bzl", "x_library")
x_library(
name = "my_x_library",
src = "source.x",
)
cc_binary(
name = "tgt",
srcs = ["mysrc.c"],
deps = ["my_x_library"],
)
You should be able to extend that with any number of files (and for C++ content; IIRC the suffices are use for automagic decision how to call the tools) as long as your generator input -> generated content is known and stable (generally a good thing for a build). Otherwise you can no longer use genrule as you need your custom rule (probably a good thing anyways) to use TreeArtifact as described in the linked answer. Or two, one with .cc suffix and one with .hh so that you can pass them to cc_library.

Q: Pass a computed linkopts to a cc_binary rule?

Suppose I have this macro which creates a native.cc_binary target:
def build_it(name, **kwargs):
native.cc_binary(
name = name + ".out",
linkopts = [
"-Lsomedir",
"-lsomelib"
],
**kwargs)
And I also have this rule which takes some sources, runs a tool on them, and generates a value, writing that value to an output file:
def _write_value_impl:
args = [f.path for f in ctx.files.srcs] + [ctx.outputs.out.path]
ctx.actions.run(
inputs = ctx.files.srcs,
outputs = [ctx.outputs.out],
arguments = args,
executable = ctx.executable._tool
)
write_value = rule(
implementation=_write_value_impl,
attrs = {
"srcs": attr.label_list(allow_files = True),
"out": attr.output(mandatory = True),
"_tool": attr.label(
executable = True,
allow_files = True,
default = Label("//tools:generate_value")
}
)
Okay, what I'd like to do is modify the macro so that it adds the value generated by the write_value rule to the linkopts. Something like this:
def build_it(name, value, **kwargs):
native.cc_binary(
name = name + ".out",
linkopts = [
"-Lsomedir",
"-lsomelib",
"-Wl,--defsym=SOME_SYMBOL={}".format(value)
],
**kwargs)
How do I make this work? The problem is that the target of build_it is generated at analysis time, but the value it needs is generated at evaluation time. Also, the value got put into a file. How do I get the value out of the file and give it to the macro?
I suspect that instead of a macro, I need a rule, but how do I get the rule to call native.cc_binary?
You can write a repository_rule() to create files and generate values prior to the loading phase, and then files in the #external_repo//... will be accessible by the rules during analysis. https://docs.bazel.build/versions/master/skylark/repository_rules.html
This can't be done in Bazel, precisely because of what you mentioned. All the inputs to rules need to be determined in the analysis phase rather than the execution phase. Bazel wants to build the complete action graph before executing any actions, and this would require the write_value rule to run before build_it could be analyzed.
A workaround might be to generate the BUILD file yourself outside of Bazel beforehand, and then use the generated BUILD file during your build.
Another workaround is to hard-code the linkopts to specify them to what you expect them to be. Then in write_value check if they are what you expected and if not throw an exit code. That way Bazel will at least warn you when they aren't going to match, but it will take some effort to update both places so they're aligned again.
For your specific problem there is a concept of linker scripts, and even implicit linker scripts. Perhaps you could generate one and supply that to cc_binary in the srcs attribute. You may need to name it as a .o file (even though it's not an object file). The GCC linker documentation says:
If you specify a linker input file which the linker can not recognize
as an object file or an archive file, it will try to read the file as
a linker script.

Resources