I'm trying to write a macro to abstract over a bunch of rules that I need for a couple of different targets. One thing that I need to do is to create a small file that will be treated as a source file in a later rule. If this was a rule I would just use expand_template. The best I can currently come up with involves a native.genrule and making sure I escape everything correctly and passing it to echo.
I would hope there is an easier way.
Code in question:
racket_contents = """
#lang racket/base
(require
"bootstrap-compiler.rkt"
racket/runtime-path)
(define-runtime-path library-compiler-list-file "%s")
(run-bootstrap-compiler library-compiler-list-file #"%s_main")
""" % (source_file_list, short_name)
native.genrule(
name = "racketize_" + name,
outs = [racket_src_name],
cmd = "echo >>$# '%s'" % racket_contents,
)
You can have a generic rule for expanding templates.
genfile.bzl
def _genfile_impl(ctx):
ctx.actions.expand_template(
template = ctx.file.template,
output = ctx.file.output,
substitutions = substitutions,
)
return [
DefaultInfo(files = depset([ctx.file.output])),
]
genfile = rule(
implementation = _genfile_impl,
attrs = {
"template": attr.label(
mandatory = True,
allow_single_file = True,
),
"output": attr.output(
mandatory = True,
),
"substitutions": attr.string_dict(),
},
)
racket.tpl
#lang racket/base
(require
"bootstrap-compiler.rkt"
racket/runtime-path)
(define-runtime-path library-compiler-list-file "TMPL_SOURCE_FILES")
(run-bootstrap-compiler library-compiler-list-file #"TMPL_SHORTNAME_main")
BUILD
load("//:genfile.bzl", "genfile")
genfile(
name = "racket",
output = "racket.out",
template = "racket.tpl",
substitutions = {
"TMPL_SOURCE_FILES": ",",join(["n1","n2"]),
"TMPL_SHORTNAME": "shortname",
},
)
Related
I'm using the please build tool and I'm having trouble using the target|file syntax with a filegroup. Specifically, I want to use specific files from the filegroup output in a genrule, but the syntax doesn't seem to work. Here's an example of what I'm trying to do:
# subdir/BUILD.plz
this = package_name()
a = text_file(
name = 'a_file',
content = "aaaaaaaaa",
)
b = text_file(
name = 'b_file',
content = "bbbbbbbbb",
)
c = text_file(
name = 'c_file',
content = "ccccccccc",
)
filegroup(
name = this,
srcs = {
'A': [a],
'B': [b],
'C': [c],
},
visibility = ['PUBLIC']
)
# BUILD.plz
sub = '#//subdir'
a_and_c_concat = genrule(
name = 'a_and_c_concat',
srcs = {
'A': [f'{sub}|A'],
'C': [f'{sub}|C'],
},
outs = ["out"],
cmd = """
set -eux
cat "${SRCS_A}" "${SRCS_C}" > "${OUTS}"
""",
visibility = ['PUBLIC'],
)
It does the expected behavior if I change the filegroup to be a genrule and 're-export' srcs to outs like so:
# subdir/BUILD.plz
...
genrule(
name = this,
srcs = {
'A': [a],
'B': [b],
'C': [c],
},
outs = {
'A': ['a'],
'B': ['b'],
'C': ['c'],
},
cmd = """
set -eux
cp "${SRCS_A}" "${OUTS_A}"
cp "${SRCS_B}" "${OUTS_B}"
cp "${SRCS_C}" "${OUTS_C}"
""",
visibility = ['PUBLIC']
)
Is there a way to use the target|file syntax with filegroup in Please build, or is there another way to achieve the same effect another way?
Any help or advice would be greatly appreciated. Thank you!
This isn't implemented at the moment, but I think it's a reasonable thing for Please to do.
Filegroups are just a thin wrapper around build_rule(), so they accept all the same arguments, however they don't expose named sources as named outputs. I've created an issue here to track this as a feature request:
https://github.com/thought-machine/please/issues/2701
I created a first rule file to geneate a scirpt using the ctx.actions.expand_template and ran it. and I wanted that pass the result of bazel run with first rule to the next 2nd rule file as a resource. But, I could't get the result which is generated by bazel run 1st rule in the 2nd rule.
It does not mean the script created by the first rule, but the file created when the script is executed.
Below example is what I tested.
This is bazel rule file
def _1st_rule_impl(ctx):
...
out = ctx.outputs.executable
template = ctx.file.my_template
ctx.actions.expand_template(
output = out,
template = template,
substitutions = {
"{ARG1}: ctx.attr.my_file_name,
"{ARG2}: ctx.attr.my_file_content,
}
return [
DefaultInfo(
files = depset([out]),
runfiles = ctx.runfiles(files = [out])
)
]
1st_rule = rule(
implementation = _1st_rule_impl,
attrs = {
"my_template":attr.label(
allow_single_file = True,
default = Label("#my_test//my_rules:my_script.sh.template"),
),
"my_file_name": attr.string(
default = "myfile.txt",
),
"my_file_content": attr.string(
default = "hello world",
),
},
executable = True,
)
def _2nd_rule_impl(ctx):
...
for dep in ctx.attr.deps:
//
// HOW CAN I GET THE RESULT OF `bazel run` THE '1st_rule'?
// I WANT TO GET THE `myfile.txt` WHICH IS GENERATED BY '1st_rule'
//
return [
DefaultInfo(
files = depset([out]),
runfiles = ctx.runfiles(files = [out]),
)
]
2nd_rule = rule(
implementation = _2nd_rule_impl,
attrs = {
"dep":attr.label_list(
mandatory= True,
),
...
},
executable = True,
)
This is BUILD file
1st_rule(
name = "my_1st_rule",
my_file_name = "myfile.txt",
my_file_content = "hello world",
)
2nd_rule(
name = "my_2nd_rule",
dep = [
":my_1st_rule",
],
)
This is template shell script
...
function create_file() {
echo "{ARG2}" > "{ARG1}"
}
...
I tested with the example described above.
There are several ways to access the outputs of dependencies. Perhaps, the simplest is ctx.files. For example, print(ctx.files.dep) in _2nd_rule_impl should show the output of my_1st_rule when my_2nd_rule is analyzed.
I am trying to write a custom rule, where I first generate a file from a template, then pass this file to a script to generate some c++ headers that are the output of my rule.
def _msg_library_impl(ctx):
# For each target in deps, print its label and files.
for source in enumerate(ctx.attr.srcs):
print("File = " + str(source))
out_header = ctx.actions.declare_file("some_header.hpp")
out_arguments = ctx.actions.declare_file("arguments.json")
ctx.actions.expand_template(
template = ctx.file._arguments_file,
output = out_arguments,
substitutions = {
"{output_dir}": out_header.dirname,
"{idl_tuples}": out_header.path,
},
)
args = ctx.actions.args()
args.add("--arguments-file")
args.add(out_arguments)
ctx.actions.run(
outputs = [out_header],
progress_message = "Generating headers '{}'".format(out_header.short_path),
executable = ctx.executable._generator,
arguments = [args],
)
return [
CcInfo(compilation_context=cc_common.create_compilation_context(
includes=depset([out_header.dirname]),
headers=depset([out_header])))
]
msg_library = rule(
implementation = _msg_library_impl,
output_to_genfiles = True,
attrs = {
"srcs": attr.label_list(allow_files = True),
"outs": attr.output_list(),
"_arguments_file": attr.label(
allow_single_file = [".json"],
default = Label("//examples/generation_rule:arguments_template.json"),
),
"_generator": attr.label(
default = Label("//examples/generation_rule:generator"),
executable = True,
cfg = "exec"
),
},
)
Here, generator is a python library that, given an input file provided to srcs and an arguments file generates headers.
The issue that I am facing is that it seems that the expand_template doesn't actually run before run is called, so the generated file is nowhere to be found. What am I doing wrong here? Did I misunderstand how things work?
You need to indicate the file is an input to the action, in addition to passing its path in the arguments. Change the ctx.actions.run to:
ctx.actions.run(
outputs = [out_header],
inputs = [out_arguments],
progress_message = "Generating headers '{}'".format(out_header.short_path),
executable = ctx.executable._generator,
arguments = [args],
)
I'm working on my first custom Bazel rules. The rules allow the running of bats command line tests.
I've included the rule definition below verbatim. I'm pretty happy with it so far but there's one part which feels really ugly and non-standard. If the rule user adds a binary dependency to the rule then I make sure that the binary appears on the PATH so that it can be tested. At the moment I do this by making a list of the binary paths and then appending them with $PWD which is expanded inside the script to the complete execution path. This feels hacky and error prone.
Is there a more idiomatic way to do this? I don't believe I can access the execution path in the rule due to it not being created until the execution phase.
Thanks for your help!
BATS_REPOSITORY_BUILD_FILE = """
package(default_visibility = [ "//visibility:public" ])
sh_binary(
name = "bats",
srcs = ["libexec/bats"],
data = [
"libexec/bats-exec-suite",
"libexec/bats-exec-test",
"libexec/bats-format-tap-stream",
"libexec/bats-preprocess",
],
)
"""
def bats_repositories(version="v0.4.0"):
native.new_git_repository(
name = "bats",
remote = "https://github.com/sstephenson/bats",
tag = version,
build_file_content = BATS_REPOSITORY_BUILD_FILE
)
BASH_TEMPLATE = """
#!/usr/bin/env bash
set -e
export TMPDIR="$TEST_TMPDIR"
export PATH="{bats_bins_path}":$PATH
"{bats}" "{test_paths}"
"""
def _dirname(path):
prefix, _, _ = path.rpartition("/")
return prefix.rstrip("/")
def _bats_test_impl(ctx):
runfiles = ctx.runfiles(
files = ctx.files.srcs,
collect_data = True,
)
tests = [f.short_path for f in ctx.files.srcs]
path = ["$PWD/" + _dirname(b.short_path) for b in ctx.files.deps]
sep = ctx.configuration.host_path_separator
ctx.file_action(
output = ctx.outputs.executable,
executable = True,
content = BASH_TEMPLATE.format(
bats = ctx.executable._bats.short_path,
test_paths = " ".join(tests),
bats_bins_path = sep.join(path),
),
)
runfiles = runfiles.merge(ctx.attr._bats.default_runfiles)
return DefaultInfo(
runfiles = runfiles,
)
bats_test = rule(
attrs = {
"srcs": attr.label_list(
allow_files = True,
),
"deps": attr.label_list(),
"_bats": attr.label(
default = Label("#bats//:bats"),
executable = True,
cfg = "host",
),
},
test = True,
implementation = _bats_test_impl,
)
This should be easy to support from Bazel 0.8.0 which will be released in ~2 weeks.
In your skylark implementation you should do ctx.expand_location(binary) where binary should be something like $(execpath :some-label) so you might want to just format the label you got from the user with the $(execpath) and bazel will make sure to give you the execution location of that label.
Some relevant resources:
$location expansion in Bazel
https://github.com/bazelbuild/bazel/issues/2475
https://github.com/bazelbuild/bazel/commit/cff0dc94f6a8e16492adf54c88d0b26abe903d4c
We are generating a number of Go source files as part of our build. Previously we used a genrule (example here) which resulted in the generated files being stored in bazel-genfiles/.
We recently switched to using a custom rule as demonstrated in rules_go (https://github.com/bazelbuild/rules_go/tree/master/examples/bindata). This change means that the output source files are stored in bazel-bin/ instead of bazel-genfiles/.
This change of output location has broken Go integration in some of the IDEs used by our developers. Notably, gocode, the autocompletion engine used by vim-go and VSCode, when running in bzl (Bazel) lookup mode seems to expect to find generated sources in bazel-genfiles/, not bazel-bin/, and therefore fails.
How do I modify my rule to save the output to bazel-genfiles/ instead of bazel-bin/? My rule is equivalent to the example in rules_go:
def _bindata_impl(ctx):
out = ctx.new_file(ctx.label.name + ".go")
ctx.action(
inputs = ctx.files.srcs,
outputs = [out],
executable = ctx.file._bindata,
arguments = [
"-o", out.path,
"-pkg", ctx.attr.package,
"-prefix", ctx.label.package,
] + [src.path for src in ctx.files.srcs],
)
return [
DefaultInfo(
files = depset([out])
)
]
bindata = rule(
_bindata_impl,
attrs = {
"srcs": attr.label_list(allow_files = True, cfg = "data"),
"package": attr.string(mandatory=True),
"_bindata": attr.label(allow_files=True, single_file=True, default=Label("#com_github_jteeuwen_go_bindata//go-bindata:go-bindata")),
},
)
I would expect an argument to either ctx.new_file or ctx.action but cannot find anything relevant in the Skylark reference or tutorial.
Many thanks!
Try setting output_to_genfiles=True in the rule() definition. It is mentioned in the rule docs.
So:
bindata = rule(
_bindata_impl,
attrs = {
"srcs": attr.label_list(allow_files = True, cfg = "data"),
"package": attr.string(mandatory=True),
"_bindata": attr.label(allow_files=True, single_file=True, default=Label("#com_github_jteeuwen_go_bindata//go-bindata:go-bindata")),
},
output_to_genfiles = True,
)