I'm using pex_pytext and I want to pass some arguments. Some of these args are paths to files: my source code, so that I can get coverage info. There's an args parameter that gets passed to the PEX-pytest executable:
pex_pytest(
name = "test",
srcs = glob(["**/*_test.py"]),
args = [
"--cov=path/to/code",
"--cov-config=path/to/.coveragerc",
],
deps = [
":main",
"//pylibs/test/mock",
],
)
So here, I'm expecting that I'll get my main code (:main) and the mock library rolled into the test-pex. Then I want to run it with the --cov and --cov-config arguments.
The problem is that args takes an array of strings, and I don't know how to pass a file-path. I was thinking that if I could ask Bazel to translate a label into a file-path, I could supply the appropriate labels.
How can I get a path relative to my BUILD file and pass it to a rule defined in that BUILD file as a string?
args (for tests) should have "make" variable substitution, so $(location) may work:
"--cov=$(location :path/to/code)"
Related
Suppose I have a Bazel macro that is using a generator rule to generate an output file given an input file:
def my_generator(
name,
input_file,
output_file,
**kwargs):
args = []
args.extend(["--arg1", "$(location %s)" % output_file])
args.extend(["arg2", "$(locations %s)" % input_file])
cmd_params = " ".join(args)
native.genrule(
name = name,
srcs = [input_file],
outs = [output_file],
cmd = "python $(location //path/to:target_generator) %s" % cmd_params,
tools = ["/path/to/tool:mytool"],
)
Then I was previously using this macro as:
my_generator(
name = "gen1",
input_file = ":targetToGeneratetextFile",
output_file = "outputfile.txt",
visibility = ["//myproject/oath/to/current/package/test:__subpackages__"],
)
where a target is passed as input_file. This was working.
Then I wanted to reuse it with a different input but to generate the same output, where the input is now a file within the project but in another folder.
my_generator(
name = "gen2",
input_file = "//path/to/the/file/realFile.txt",
output_file = "outputfile.txt",
visibility = ["//myproject/oath/to/current/package/test:__subpackages__"],
)
I am getting two errors in this way:
For how it is, Bazel cannot find the realFile.txt: it tries to read it as a target:
no such package '//path/to/the/file/realFile.txt': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package
If I copy the file in the current package folder, it is able to read it.
Bazel is complaining that gen1 and gen2 are writing/overwriting the same output file outputfile.txt:
Error in genrule: generated file 'outputfile.txt' in rule 'gen2' conflicts with existing generated file from rule 'gen1', defined at ...
How can I solve these issues?
I think that the problem is that these two calls are both executed, whereas I would like them to be executed depending on some target, i.e., target A needs only run gen1 and target B gen2 exclusively. I do not if that is possible but for example moving each of these call inside the target they belong to might be a solution that avoids this issue.
EDIT
I was thinking as solution to do something like:
my_generator(
name = "gen2",
input_file = select({
":opt1": [":targetToGeneratetextFile"],
":opt2": ["realTextFile.txt"],
"//conditions:default": [":targetToGeneratetextFile"],
}),
output_file = "outputfile.txt",
visibility = ["//myproject/oath/to/current/package/test:__subpackages__"],
)
with proper config_setting and then call it from the target with the proper flag but I am getting the error:
expected value of type 'string' for element 0 of attribute 'srcs' in 'genrule' rule, but got select({":opt1": [":targetToGeneratetextFile"], ":opt2": ["realTextFile.txt"],"//conditions:default": [":targetToGeneratetextFile"],
})
The label //path/to/the/file/realFile.txt is shorthand for //path/to/the/file/realFile.txt:realFile.txt, aka <repository root>/path/to/the/file/realFile.txt/realFile.txt. Depending on where the deepest-nested folder with a BUILD file is (which determines the package), you're looking for something like //path/to/the/file:realFile.txt or //path/to:the/file/realFile.txt instead.
You can't have two rules which write the same file, because then Bazel can't tell which way to build it if you bazel build the file. Some alternatives:
Put them in separate packages (aka separate folders with BUILD files)
Name them differently, like gen1_outputfile.txt and gen2_outputfile.txt, or gen1/outputfile.txt and gen2/outputfile.txt. You could automate this in the macro like srcs = [name + '/outputfile.txt'].
Use a single rule to generate it with an appropriate select, like your edit.
With the select, you're trying to create something like this:
genrule(
srcs = select({..., "//conditions:default": [":targetToGeneratetextFile"]}),
...
)
but as written you have this instead:
genrule(
srcs = [select({..., "//conditions:default": [":targetToGeneratetextFile"]})],
...
)
Effectively, between the list in the select's value and the macro body, you're creating a nested list. I would change your macro argument to input_files and then do srcs = input_files in the body, so the caller of the macro can bundle things into lists as desired.
I have a yaml file in a Bazel monorepo that has constants I'd like to use in several languages. This is kind of like how protobuffs are created and used.
How can I parse this yaml file at build time and depend on the outputs?
For instance:
item1: "hello"
item2: "world"
nested:
nested1: "I'm nested"
nested2: "I'm also nested"
I then need to parse this yaml file so it can be used in many different languages (e.g., Rust, TypeScript, Python, etc.). For instance, here's the desired output for TypeScript:
export default {
item1: "hello",
item2: "world",
nested: {
nested1: "I'm nested",
nested2: "I'm also nested",
}
}
Notice, I don't want TypeScript code that reads the yaml file and converts it into an object. That conversion should be done in the build process.
For the actual conversion, I'm thinking of writing that in Python, but it doesn't need to be. This would then mean the python also needs to run at build time.
P.S. I care mostly about the functionality, so I'm flexible with the exactly how it's done. I'm even fine using another file format aside from yaml.
Thanks to help from #oakad, I was able to figure it out. Essentially, you can use genrule to create outputs.
So, assuming you have some target like python setup to generate the output named parse_config, you can just do this:
genrule(
name = "generated_output",
srcs = [],
outs = ["output.txt"],
cmd = "./$(execpath :parse_config) > $#" % name,
exec_tools = [":parse_config"],
visibility = ["//visibility:public"],
)
The generated file is output.txt. And you can access it via //lib/config:generated_output.
Note, essentially the cmd is piping the stdout into the file contents. In Python that means anything printed will appear in the generated file.
By default, the cc_binary rule of bazel produces an output file without any extension on Linux.
My compiler generates a .s19 file extension as output(I have extended the toolchain).
Is there a way to specify the output file's extension?
I get a "Linking 'App-name' failed: not all outputs were created or valid" although the expected output file 'App-name.s19' is generated.
My second question is:
In addition to an 'App-name.s19' file my compiler also generates a 'App-name.map' file. Is there a way to tell bazel to verify both 'App-name.s19' and 'App-name.map' files. i.e. verify multiple outputs generated by cc_binary.
Question 1)
When configuring your toolchain you can load: artifact_name_pattern from #bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl and set artifact_name_patterns attribute of cc_common.create_cc_toolchain_config_info():
artifact_name_patterns = [
artifact_name_pattern(
category_name = "executable",
prefix = "",
extension = ".s19",
),]
Question 2)
It seems that cc_binary doesn't support yet map files https://github.com/bazelbuild/bazel/issues/6718
A possible workaround would be to use a genrule in your BUILD file to compile your code and generate the mapfile,
genrule(
name = "map",
srcs = ["hello-world.cc"],
outs = ["hello-world.exe","output.map"],
output_to_bindir = 1,
cmd= "/usr/bin/x86_64-w64-mingw32-gcc -o $(location hello-world.exe) $(location hello-world.cc) -Wl,-Map=\"$(location output.map)\" -lstdc++",
)
or alternatively https://groups.google.com/g/bazel-discuss/c/A00d7Ui1f8s/m/vybgGEPIBwAJ
Suppose I have this macro which creates a native.cc_binary target:
def build_it(name, **kwargs):
native.cc_binary(
name = name + ".out",
linkopts = [
"-Lsomedir",
"-lsomelib"
],
**kwargs)
And I also have this rule which takes some sources, runs a tool on them, and generates a value, writing that value to an output file:
def _write_value_impl:
args = [f.path for f in ctx.files.srcs] + [ctx.outputs.out.path]
ctx.actions.run(
inputs = ctx.files.srcs,
outputs = [ctx.outputs.out],
arguments = args,
executable = ctx.executable._tool
)
write_value = rule(
implementation=_write_value_impl,
attrs = {
"srcs": attr.label_list(allow_files = True),
"out": attr.output(mandatory = True),
"_tool": attr.label(
executable = True,
allow_files = True,
default = Label("//tools:generate_value")
}
)
Okay, what I'd like to do is modify the macro so that it adds the value generated by the write_value rule to the linkopts. Something like this:
def build_it(name, value, **kwargs):
native.cc_binary(
name = name + ".out",
linkopts = [
"-Lsomedir",
"-lsomelib",
"-Wl,--defsym=SOME_SYMBOL={}".format(value)
],
**kwargs)
How do I make this work? The problem is that the target of build_it is generated at analysis time, but the value it needs is generated at evaluation time. Also, the value got put into a file. How do I get the value out of the file and give it to the macro?
I suspect that instead of a macro, I need a rule, but how do I get the rule to call native.cc_binary?
You can write a repository_rule() to create files and generate values prior to the loading phase, and then files in the #external_repo//... will be accessible by the rules during analysis. https://docs.bazel.build/versions/master/skylark/repository_rules.html
This can't be done in Bazel, precisely because of what you mentioned. All the inputs to rules need to be determined in the analysis phase rather than the execution phase. Bazel wants to build the complete action graph before executing any actions, and this would require the write_value rule to run before build_it could be analyzed.
A workaround might be to generate the BUILD file yourself outside of Bazel beforehand, and then use the generated BUILD file during your build.
Another workaround is to hard-code the linkopts to specify them to what you expect them to be. Then in write_value check if they are what you expected and if not throw an exit code. That way Bazel will at least warn you when they aren't going to match, but it will take some effort to update both places so they're aligned again.
For your specific problem there is a concept of linker scripts, and even implicit linker scripts. Perhaps you could generate one and supply that to cc_binary in the srcs attribute. You may need to name it as a .o file (even though it's not an object file). The GCC linker documentation says:
If you specify a linker input file which the linker can not recognize
as an object file or an archive file, it will try to read the file as
a linker script.
I'm building ARM Cortex-M firmware from Bazel with a custom CROSSTOOL. I'm successfully building elf files and manually objcopying them to binary files with the usual:
path/to/my/objcopy -o binary hello.elf hello.bin
I want to make a Bazel macro or rule called cc_firmware that:
Adds the -Wl,-Map=hello.map flags to generate a mapfile
Changes the output elf name from hello to hello.elf
Invokes path/to/my/objcopy to convert the elf to a bin.
I don't know how to get the name of a CROSSTOOL tool (objcopy) to invoke it, and it feels wrong to have the rule know the path to the tool executable.
Is there a way to use the objcopy that I've already told Bazel about in my CROSSTOOL file?
You can actually access this from a custom rule. Basically you need to tell Bazel that you want access to the cpp configuration information (fragments = ["cpp"]) and then access its path via ctx.fragments.cpp.objcopy_executable, e.g.,:
def _impl(ctx):
print("path: {}".format(ctx.fragments.cpp.objcopy_executable))
# TODO: actually do something with the path...
cc_firmware = rule(
implementation = _impl,
fragments = ["cpp"],
attrs = {
"src" : attr.label(allow_single_file = True),
"map" : attr.label(allow_single_file = True),
},
outputs = {"elf" : "%{name}.elf"}
)
Then create the output you want with something like (untested):
def _impl(ctx):
src = ctx.attr.src.files.to_list()[0]
m = ctx.attr.map.files.to_list()[0]
ctx.action(
command = "{objcopy} -Wl,-Map={map} -o binary {elf_out} {cc_bin}".format(
objcopy=ctx.fragments.cpp.objcopy_executable,
map=m,
elf_out=ctx.outputs.elf.path,
cc_bin=src,
),
outputs = [ctx.outputs.elf],
inputs = [src, m],
)