A bazel rule "template" to generate many rules - bazel

Question

I have some number of these input files that are called src0, src1, src2, ... srcN, and they need to be compiled into dst1, dst2, dst3, ... dstN.
I have a simple my_convert rule, and macros to generate full filename paths:
def src_filename(id):
return ":some/path/to/src{}".format(id)
def dest_filename(id):
return ":some/path/to/dest{}".format(id)
my_convert(
src=src_filename(0),
dest=dest_filename(0)
)
my_convert(
src=src_filename(1),
dest=dest_filename(1)
)
...
Now, I could copy & paste my_convert N times, but N is sometimes in the hundreds, and the number of files depends on some configuration... so I'd really like to have a dynamic rule of sorts where I can pass in 'N' from command-line and my_convert gets called for all ids in 0..N range.
What's the best way of doing this within Bazel? Is there some way to describe rules in a for loop? (I'll fall back to writing a script to generate a BUILD file with all the rules, but I'm hoping that I don't have to do that)

Answer 1

Does it need to be from some configuration? Or can it be something like "all the files in a directory"? Because then you could do something like this:
my_convert(
name = "converstions",
srcs = glob(["some/path/to/src*"]),
)
def _my_convert_impl(ctx):
outputs = []
for src in ctx.files.srcs:
output = ctx.actions.declare_file(src.basename + ".out")
outputs.append(output)
ctx.actions.run_shell(
inputs = [src],
outputs = [output],
command = "wc -l '%s' > '%s'" % (src.path, output.path),
mnemonic = "Convert",
progress_message = "Convert %{input}",
)
return [DefaultInfo(files = depset(outputs))]
my_convert = rule(
implementation = _my_convert_impl,
attrs = {
"srcs": attr.label_list(allow_files = True),
},
)
Then bazel build //package:conversions.
If it can't be based on a directory, another approach is to generate targets for each of the files, and then explicitly build the subset you want. Something like this:
generate_my_conversions(glob(["some/path/to/src*"]))
def generate_my_conversions(files):
for file in files:
my_convert(
name = "convert_" + file.replace("/", "_"), # this might need adjusting
src = file,
)
Then build the targets you want (maybe generate the list):
bazel build //package:convert_some_path_to_src0 //package:convert_some_path_to_src1 //package:convert_some_path_to_src2 //package:convert_some_path_to_srcN
There's a similar way to do this using output files: https://docs.bazel.build/versions/main/skylark/rules.html#declaring-outputs
which avoids instantiating a rule per file (the distinction/advantage may or may not be important until there are many more files).
If it really needs to be a flag for this, it's possible though a bit awkward, because it's a bit uncommon to have a flag to configure a specific individual target (flags are usually for things that apply to the entire build). Something like this:
$ tree
.
├── package
│   ├── BUILD
│   ├── defs.bzl
│   └── some
│   └── path
│   └── to
│   ├── src0
│   ├── src1
│   ├── src10
│   ├── src11
│   ├── src12
│   ├── src2
│   ├── src3
│   ├── src4
│   ├── src5
│   ├── src6
│   ├── src7
│   ├── src8
│   └── src9
└── WORKSPACE
package/defs.bzl:
def _my_convert_impl(ctx):
file_count = ctx.attr.file_count[FileCountToConvertProvider].count
if file_count < 0:
fail("Must set flag --%s" % ctx.attr.file_count.label)
if file_count > len(ctx.files.srcs):
fail("More files requested than sources")
# srcs could be in any order, so extract the source number from the file name
srcs = {}
for src in ctx.files.srcs:
file_number_char_count = 0
# reversed() doesn't work with string in Starlark...
for i in range(len(src.basename) - 1, 0, -1):
if src.basename[i].isdigit():
file_number_char_count += 1
else:
break
file_number = int(src.basename[-file_number_char_count:])
srcs[file_number] = src
outputs = []
for i in range(file_count):
src = srcs[i]
if src.path.startswith(ctx.label.package + "/"):
package_relative_path = src.path[len(ctx.label.package) + 1:]
else:
package_relative_path = src.path
output = ctx.actions.declare_file(package_relative_path.replace("src", "dst"))
outputs.append(output)
ctx.actions.run_shell(
inputs = [src],
outputs = [output],
command = "wc -l '%s' > '%s'" % (src.path, output.path),
mnemonic = "Convert",
progress_message = "Convert %{input}",
)
return [DefaultInfo(files = depset(outputs))]
_my_convert = rule(
implementation = _my_convert_impl,
attrs = {
"srcs": attr.label_list(allow_files = True, mandatory = True),
"file_count": attr.label(),
},
)
FileCountToConvertProvider = provider(fields = ["count"])
def _file_count_to_convert_flag_impl(ctx):
return FileCountToConvertProvider(count = ctx.build_setting_value)
_file_count_to_convert_flag = rule(
implementation = _file_count_to_convert_flag_impl,
build_setting = config.int(flag = True),
)
# Use a macro to create the my_convert target + associated flag
def my_convert(name, srcs):
flag_name = name + "_file_count"
_file_count_to_convert_flag(
name = flag_name,
build_setting_default = -1,
)
_my_convert(
name = name,
srcs = srcs,
file_count = flag_name,
)
package/BUILD:
load(":defs.bzl", "my_convert")
my_convert(
name = "conversions",
srcs = glob(["some/path/to/src*"]),
)
Note that the glob takes in every file that might be needed, because there's no way in Starlark to connect the value of the flag to generate a list of files at loading time (i.e., at the time the build file is evaluated).
$ bazel build //package:conversions --//package:conversions_file_count=5
INFO: Analyzed target //package:conversions (4 packages loaded, 20 targets configured).
INFO: Found 1 target...
Target //package:conversions up-to-date:
bazel-bin/package/some/path/to/dst0
bazel-bin/package/some/path/to/dst1
bazel-bin/package/some/path/to/dst2
bazel-bin/package/some/path/to/dst3
bazel-bin/package/some/path/to/dst4
INFO: Elapsed time: 0.302s, Critical Path: 0.03s
INFO: 6 processes: 1 internal, 5 linux-sandbox.
INFO: Build completed successfully, 6 total actions
$ bazel build //package:conversions --//package:conversions_file_count=11
INFO: Build option --//package:conversions_file_count has changed, discarding analysis cache.
INFO: Analyzed target //package:conversions (0 packages loaded, 20 targets configured).
INFO: Found 1 target...
Target //package:conversions up-to-date:
bazel-bin/package/some/path/to/dst0
bazel-bin/package/some/path/to/dst1
bazel-bin/package/some/path/to/dst2
bazel-bin/package/some/path/to/dst3
bazel-bin/package/some/path/to/dst4
bazel-bin/package/some/path/to/dst5
bazel-bin/package/some/path/to/dst6
bazel-bin/package/some/path/to/dst7
bazel-bin/package/some/path/to/dst8
bazel-bin/package/some/path/to/dst9
bazel-bin/package/some/path/to/dst10
INFO: Elapsed time: 0.114s, Critical Path: 0.02s
INFO: 7 processes: 1 internal, 6 linux-sandbox.
INFO: Build completed successfully, 7 total actions