How to use filegroup with target|file syntax in please build? - please.build

I'm using the please build tool and I'm having trouble using the target|file syntax with a filegroup. Specifically, I want to use specific files from the filegroup output in a genrule, but the syntax doesn't seem to work. Here's an example of what I'm trying to do:
# subdir/BUILD.plz
this = package_name()
a = text_file(
name = 'a_file',
content = "aaaaaaaaa",
)
b = text_file(
name = 'b_file',
content = "bbbbbbbbb",
)
c = text_file(
name = 'c_file',
content = "ccccccccc",
)
filegroup(
name = this,
srcs = {
'A': [a],
'B': [b],
'C': [c],
},
visibility = ['PUBLIC']
)
# BUILD.plz
sub = '#//subdir'
a_and_c_concat = genrule(
name = 'a_and_c_concat',
srcs = {
'A': [f'{sub}|A'],
'C': [f'{sub}|C'],
},
outs = ["out"],
cmd = """
set -eux
cat "${SRCS_A}" "${SRCS_C}" > "${OUTS}"
""",
visibility = ['PUBLIC'],
)
It does the expected behavior if I change the filegroup to be a genrule and 're-export' srcs to outs like so:
# subdir/BUILD.plz
...
genrule(
name = this,
srcs = {
'A': [a],
'B': [b],
'C': [c],
},
outs = {
'A': ['a'],
'B': ['b'],
'C': ['c'],
},
cmd = """
set -eux
cp "${SRCS_A}" "${OUTS_A}"
cp "${SRCS_B}" "${OUTS_B}"
cp "${SRCS_C}" "${OUTS_C}"
""",
visibility = ['PUBLIC']
)
Is there a way to use the target|file syntax with filegroup in Please build, or is there another way to achieve the same effect another way?
Any help or advice would be greatly appreciated. Thank you!

This isn't implemented at the moment, but I think it's a reasonable thing for Please to do.
Filegroups are just a thin wrapper around build_rule(), so they accept all the same arguments, however they don't expose named sources as named outputs. I've created an issue here to track this as a feature request:
https://github.com/thought-machine/please/issues/2701

Related

Accessing runfiles during build in genrule

I have a cc_binary target that uses runfiles. I would like to zip the executable and all of the runfiles into a single archive using a genrule.
Something like:
genrule(
name = "zip_binary"
srcs = [
":binary",
],
outs = [
"binary.zip",
],
cmd = "zip -r $(OUTS) $(locations :binary)",
)
However, this only includes the binary and not the binary.runfiles dir.
How can I get bazel to include the runfiles in the srcs?
Genrules don't have access to enough information to do that. With a full custom rule it's pretty easy though. Something like this:
def _zipper_impl(ctx):
inputs = ctx.runfiles(files = ctx.files.srcs)
inputs = inputs.merge_all([a[DefaultInfo].default_runfiles for a in ctx.attr.srcs])
ctx.actions.run_shell(
outputs = [ctx.output.out],
inputs = inputs.files,
command = " ".join(["zip", "-r", ctx.output.out.path] +
[i.path for i in inputs.files.to_list()]),
)
return [DefaultInfo(files = depset(ctx.output.out))]
zipper = rule(
impl = _zipper_impl,
attrs = {
"out": attr.output(mandatory = True),
"srcs": attr.label_list(allow_files = True),
},
)

How do I get my custom header template rule to pass it's output downstream cc_binary/cc_library dependency?

I'm trying to build a rule for bazel which emulates the CMake *.in template system.
This has two challenges, the first is generate the template output. The second is make the output available to both genrules, filegroups and cc_* rules. The third is to get that dependency to transitively be passed to further downstream rules.
I have it generating the output file version.hpp in genfiles (or bazel-bin), and I can get the initial library rule to include it, but I can't seem to figure out how to make my cc_binary rule, which depends on the cc_library and transitively the header_template rule to find the header file.
I have the following .bzl rule:
def _header_template_impl(ctx):
# this generates the output from the template
ctx.actions.expand_template(
template = ctx.file.template,
output = ctx.outputs.out,
substitutions = ctx.attr.vars,
)
return [
# create a provider which says that this
# out file should be made available as a header
CcInfo(compilation_context=cc_common.create_compilation_context(
headers=depset([ctx.outputs.out])
)),
# Also create a provider referencing this header ???
DefaultInfo(files=depset(
[ctx.outputs.out]
))
]
header_template = rule(
implementation = _header_template_impl,
attrs = {
"vars": attr.string_dict(
mandatory = True
),
"extension": attr.string(default=".hpp"),
"template": attr.label(
mandatory = True,
allow_single_file = True,
),
},
outputs = {
"out": "%{name}%{extension}",
},
output_to_genfiles = True,
)
elsewhere I have a cc_library rule:
load("//:tools/header_template.bzl", "header_template")
# version control
BONSAI_MAJOR_VERSION = '2'
BONSAI_MINOR_VERSION = '0'
BONSAI_PATCH_VERSION = '9'
BONSAI_VERSION = \
BONSAI_MAJOR_VERSION + '.' + \
BONSAI_MINOR_VERSION + '.' + \
BONSAI_PATCH_VERSION
header_template(
name = "bonsai_version",
extension = ".hpp",
template = "version.hpp.in",
vars = {
"#BONSAI_MAJOR_VERSION#": BONSAI_MAJOR_VERSION,
"#BONSAI_MINOR_VERSION#": BONSAI_MINOR_VERSION,
"#BONSAI_PATCH_VERSION#": BONSAI_PATCH_VERSION,
"#BONSAI_VERSION#": BONSAI_VERSION,
},
)
# ...
private = glob([
"src/**/*.hpp",
"src/**/*.cpp",
"proto/**/*.hpp",
])
public = glob([
"include/*.hpp",
":bonsai_version",
])
cc_library(
# target name matches directory name so you can call:
# bazel build .
name = "bonsai",
srcs = private,
hdrs = public,
# public headers
includes = [
"include",
],
# ...
deps = [
":bonsai_version",
# ...
],
# ...
)
When I build, my source files need to be able to:
#include "bonsai_version.hpp"
I think the answer involves CcInfo but I'm grasping in the dark as to how it should be constructed.
I've already tried add "-I$(GENDIR)/" + package_name() to the copts, to no avail. The generated header still isn't available.
My expectation is that I should be able to return some kind of Info object that would allow me to add the dependency in srcs. Maybe it should be a DefaultInfo.
I've dug through the bazel rules examples and the source, but I'm missing something fundamental, and I can't find documentation that discuss this particular.
I'd like to be able to do the following:
header_template(
name = "some_header",
extension = ".hpp",
template = "some_header.hpp.in",
vars = {
"#SOMEVAR#": "value",
"{ANOTHERVAR}": "another_value",
},
)
cc_library(
name = "foo",
srcs = ["foo.src", ":some_header"],
...
)
cc_binary(
name = "bar",
srcs = ["bar.cpp"],
deps = [":foo"],
)
and include the generated header like so:
#include "some_header.hpp"
void bar(){
}
The answer looks like it is:
def _header_template_impl(ctx):
# this generates the output from the template
ctx.actions.expand_template(
template = ctx.file.template,
output = ctx.outputs.out,
substitutions = ctx.attr.vars,
)
return [
# create a provider which says that this
# out file should be made available as a header
CcInfo(compilation_context=cc_common.create_compilation_context(
# pass out the include path for finding this header
includes=depset([ctx.outputs.out.dirname]),
# and the actual header here.
headers=depset([ctx.outputs.out])
))
]
elsewhere:
header_template(
name = "some_header",
extension = ".hpp",
template = "some_header.hpp.in",
vars = {
"#SOMEVAR#": "value",
"{ANOTHERVAR}": "another_value",
},
)
cc_library(
name = "foo",
srcs = ["foo.cpp"],
deps = [":some_header"],
...
)
cc_binary(
name = "bar",
srcs = ["bar.cpp"],
deps = [":foo"],
)
If your header has a generic name (eg config.h) and you want it to be private (ie srcs instead of hdrs), you might need a different approach. I've seen this problem for gflags, which "leaked" config.h and affected libraries that depended on it (issue).
Of course, in both cases, the easiest solution is to generate and commit header files for the platforms you target.
Alternatively, you can set copts for the cc_library rule that uses the generated private header:
cc_library(
name = "foo",
srcs = ["foo.cpp", "some_header.hpp"],
copts = ["-I$(GENDIR)/my/package/name"],
...
)
If you want this to work when your repository is included as an external repository, you have a bit more work cut out for you due to bazel issue #4463.
PS. You might want to see if cc_fix_config from https://github.com/antonovvk/bazel_rules works for you. It's just a wrapper around perl but I found it useful.

Best way to create template file in bazel macro

I'm trying to write a macro to abstract over a bunch of rules that I need for a couple of different targets. One thing that I need to do is to create a small file that will be treated as a source file in a later rule. If this was a rule I would just use expand_template. The best I can currently come up with involves a native.genrule and making sure I escape everything correctly and passing it to echo.
I would hope there is an easier way.
Code in question:
racket_contents = """
#lang racket/base
(require
"bootstrap-compiler.rkt"
racket/runtime-path)
(define-runtime-path library-compiler-list-file "%s")
(run-bootstrap-compiler library-compiler-list-file #"%s_main")
""" % (source_file_list, short_name)
native.genrule(
name = "racketize_" + name,
outs = [racket_src_name],
cmd = "echo >>$# '%s'" % racket_contents,
)
You can have a generic rule for expanding templates.
genfile.bzl
def _genfile_impl(ctx):
ctx.actions.expand_template(
template = ctx.file.template,
output = ctx.file.output,
substitutions = substitutions,
)
return [
DefaultInfo(files = depset([ctx.file.output])),
]
genfile = rule(
implementation = _genfile_impl,
attrs = {
"template": attr.label(
mandatory = True,
allow_single_file = True,
),
"output": attr.output(
mandatory = True,
),
"substitutions": attr.string_dict(),
},
)
racket.tpl
#lang racket/base
(require
"bootstrap-compiler.rkt"
racket/runtime-path)
(define-runtime-path library-compiler-list-file "TMPL_SOURCE_FILES")
(run-bootstrap-compiler library-compiler-list-file #"TMPL_SHORTNAME_main")
BUILD
load("//:genfile.bzl", "genfile")
genfile(
name = "racket",
output = "racket.out",
template = "racket.tpl",
substitutions = {
"TMPL_SOURCE_FILES": ",",join(["n1","n2"]),
"TMPL_SHORTNAME": "shortname",
},
)

Idiomatic retrieval of the Bazel execution path

I'm working on my first custom Bazel rules. The rules allow the running of bats command line tests.
I've included the rule definition below verbatim. I'm pretty happy with it so far but there's one part which feels really ugly and non-standard. If the rule user adds a binary dependency to the rule then I make sure that the binary appears on the PATH so that it can be tested. At the moment I do this by making a list of the binary paths and then appending them with $PWD which is expanded inside the script to the complete execution path. This feels hacky and error prone.
Is there a more idiomatic way to do this? I don't believe I can access the execution path in the rule due to it not being created until the execution phase.
Thanks for your help!
BATS_REPOSITORY_BUILD_FILE = """
package(default_visibility = [ "//visibility:public" ])
sh_binary(
name = "bats",
srcs = ["libexec/bats"],
data = [
"libexec/bats-exec-suite",
"libexec/bats-exec-test",
"libexec/bats-format-tap-stream",
"libexec/bats-preprocess",
],
)
"""
def bats_repositories(version="v0.4.0"):
native.new_git_repository(
name = "bats",
remote = "https://github.com/sstephenson/bats",
tag = version,
build_file_content = BATS_REPOSITORY_BUILD_FILE
)
BASH_TEMPLATE = """
#!/usr/bin/env bash
set -e
export TMPDIR="$TEST_TMPDIR"
export PATH="{bats_bins_path}":$PATH
"{bats}" "{test_paths}"
"""
def _dirname(path):
prefix, _, _ = path.rpartition("/")
return prefix.rstrip("/")
def _bats_test_impl(ctx):
runfiles = ctx.runfiles(
files = ctx.files.srcs,
collect_data = True,
)
tests = [f.short_path for f in ctx.files.srcs]
path = ["$PWD/" + _dirname(b.short_path) for b in ctx.files.deps]
sep = ctx.configuration.host_path_separator
ctx.file_action(
output = ctx.outputs.executable,
executable = True,
content = BASH_TEMPLATE.format(
bats = ctx.executable._bats.short_path,
test_paths = " ".join(tests),
bats_bins_path = sep.join(path),
),
)
runfiles = runfiles.merge(ctx.attr._bats.default_runfiles)
return DefaultInfo(
runfiles = runfiles,
)
bats_test = rule(
attrs = {
"srcs": attr.label_list(
allow_files = True,
),
"deps": attr.label_list(),
"_bats": attr.label(
default = Label("#bats//:bats"),
executable = True,
cfg = "host",
),
},
test = True,
implementation = _bats_test_impl,
)
This should be easy to support from Bazel 0.8.0 which will be released in ~2 weeks.
In your skylark implementation you should do ctx.expand_location(binary) where binary should be something like $(execpath :some-label) so you might want to just format the label you got from the user with the $(execpath) and bazel will make sure to give you the execution location of that label.
Some relevant resources:
$location expansion in Bazel
https://github.com/bazelbuild/bazel/issues/2475
https://github.com/bazelbuild/bazel/commit/cff0dc94f6a8e16492adf54c88d0b26abe903d4c

Dispatching C++ generated files into srcs and hdrs

In the Bazel official documentation there is an example explaining how to create a Java library built from regular java files and files generated by a :gen_java_srcs rule. I rewrite this code here for ease of reading:
java_library(
name = "mylib",
srcs = glob(["*.java"]) + [":gen_java_srcs"],
deps = "...",
)
genrule(
name = "gen_java_srcs",
outs = [
"Foo.java",
"Bar.java",
],
...
)
Now in a C++ perspective, I am in a scenario where the genrule generates two kind of files: .hpp and .cpp:
genrule(
name = "gen_cpp_srcs",
outs = [
"myFile_1.hpp","myFile_2.hpp",...,"myFile_N.hpp",
"myFile.cpp","myFile_2.cpp",...,"myFile_N.cpp",
],
...
)
where N is some tens.
My problem/question is: how to write the cc_library rule, with an automatic dispatching of the hpp and cpp files into hdrs and srcs field?
I want something like:
cc_library(
name = "mylib",
srcs = glob(["*.cpp"]) + (howto: .cpp files of [":gen_cpp_srcs"]),
hdrs = glob(["*.hpp"]) + (howto: .hpp files of [":gen_cpp_srcs"]),
...
)
Some magic like:
output_filter(":gen_cpp_srcs","*.cpp")
would be perfect, but I do not know enough of Bazel to make it real.
Globs only get expanded when they're passed into rules, so you'll need to write a simple rule. I would package it like this (in a file named filter.bzl):
# The actual rule which does the filtering.
def _do_filter_impl(ctx):
return struct(
files = set([f for f in ctx.files.srcs if f.path.endswith(ctx.attr.suffix)]),
)
_do_filter = rule(
implementation = _do_filter_impl,
attrs = {
"srcs": attr.label_list(
mandatory = True,
allow_files = True,
),
"suffix": attr.string(
mandatory = True,
),
},
)
# A convenient macro to wrap the custom rule and cc_library.
def filtered_cc_library(name, srcs, hdrs, **kwargs):
_do_filter(
name = "%s_hdrs" % name,
visibility = ["//visibility:private"],
srcs = hdrs,
suffix = ".hpp",
)
_do_filter(
name = "%s_srcs" % name,
visibility = ["//visibility:private"],
srcs = srcs,
suffix = ".cpp",
)
native.cc_library(
name = name,
srcs = [ ":%s_srcs" % name ],
hdrs = [ ":%s_hdrs" % name ],
**kwargs
)
This is what my demo BUILD file looks like (I changed the globs so they both include *.cpp and *.hpp files; using the label of a genrule will work the same way):
load("//:filter.bzl", "filtered_cc_library")
filtered_cc_library(
name = "mylib",
srcs = glob(["*.*pp"]),
hdrs = glob(["*.*pp"]),
)
This is easy to extend to more sophisticated filtering by changing _do_filter_impl. In particular, changing suffix to an attr.string_list so you can accept multiple C/C++ source/header extensions seems like a good idea.
Depending on the genrule by name (:gen_cpp_srcs) will give you all of the outputs of the genrule, as you have noted. Instead, you can depend on the individual outputs of the genrule (e.g. hdrs = [:myFile.hpp] and srcs = [:myFile.cpp]).
See also the answer to Bazel & automatically generated cpp / hpp files.
Looks like you know the total number of files that should be generated. Can you put those in their own variables and then reuse them in both targets. Something like this in your BUILD file:
output_cpp_files = [
"myFile_1.cpp",
"myFile_2.cpp",
"myFile_3.cpp"
]
output_hpp_files = [
"myFile_1.hpp",
"myFile_2.hpp",
"myFile_3.hpp"
]
genrule(
name = "gen_cpp_srcs",
outs = output_cpp_files + output_hpp_files,
cmd = """
touch $(OUTS)
"""
)
cc_library(
name = "mylib",
srcs = output_cpp_files,
hdrs = output_hpp_files
)

Resources