Bazel: genrule that outputs a directory - bazel

Question

I'm just getting started working with Bazel. So, I apologize in advance that I haven't been able to figure this out.
I'm trying to run a command that outputs a bunch of files to a directory and make this directory available for subsequent targets. I have two different attempts:
Use genrule
Write my own rule
I was naively hoping to just do this with a genrule. But, it doesn't seem you can say "I don't know exactly what this command is going to output" and put a directory in outs. Now I'm trying to write a rule that can use ctx.actions.declare_directory but I haven't gotten it quite right. I can't seem to get the tools over from my workspace and into my rule.
My genrule attempt looks something like this:
genrule(
name = "doit",
srcs = [
"doitConfigA",
"doitConfigB",
],
cmd = 'HOME=. ./$(location path/to/doit) install',
# Neither of the below outs work - seems like bazel wants to know
# exactly this list of files. I don't know the files that
# will be output ahead of time.
# This one looks at the `out_dir` that I already have and
# expects the files to be the same which they might not be
outs = glob(["out_dir/**/*.*"]),
# this fails with:
# "declared output 'out_dir' was not
# created by genrule. This is probably because the genrule actually
# didn't create this output, or because the output was a directory
# and the genrule was run remotely (note that only the contents of
# declared file outputs are copied from genrules run remotely)"
outs = ['out_dir'],
tools = ['path/to/doit'],
)
My custom rule attempt looks something like this:
def _impl(ctx):
dir = ctx.actions.declare_directory("out_dir")
ctx.actions.run_shell(
outputs=[dir],
progress_message="Running doit install ...",
command="HOME=. ./path/to/doit install",
tools=[ctx.attr.tools],
)
doit = rule(
implementation=_impl,
attrs={
"tools": attr.label_list(allow_files=True),
},
outputs={"out": "out_dir"},
)
Then, to run my doit rule, my BUILD file looks like this:
doit(
name = 'doit',
tools = ['path/to/doit'],
)
In my genrule, the command runs but it doesn't like my trying to use a directory in outs, it seems. In my custom rule, I can't seem to tell Bazel that I want to use ./path/to/doit as a tool from my workspace, eg expected type 'File' for 'tools' element but got type 'list' instead ...
Seems like I must be missing something basic because surely this is a common situation to run a command and output a bunch of unknown stuff to a directory?

Answer 1

The output of a genrule must be a fixed list of files. As a work-around, you can create a zip from the output directory.
I used this approach to manipulate the output of yarn install where the usual method was not viable:
genrule(
name = "node_modules",
srcs = [
"package.json",
"yarn.lock",
],
cmd = " && ".join([
"yarn install --pure-lockfile",
"zip -r $# node_modules",
]),
outs = [
"node_modules.zip",
],
)
Then a rule that consumes the zip:
# Rule that generates a list of the folders in node_modules
genrule(
name = "node_modules_ls",
srcs = [
":node_modules",
],
cmd = " && ".join([
"unzip $(location :node_modules) -d . ",
"ls > $#",
]),
outs = [
"out.txt",
],
)

Answer 2

A while ago I created this example showing how to use directories with skylark action: How to build static library from the Generated source files using Bazel Build. Maybe it still works :)
Genrule won't work, this is too advanced use case.

Answer 3

https://github.com/aspect-build/bazel-lib/blob/main/docs/run_binary.md has a similar API to genrule, and it supports directory outputs.