If I have
cc_binary(
name = "stooges",
srcs = [ "larry.cc", "curly.cc", "moe.cc" ],
)
is there a bazel query which will return "larry.cc", "curly.cc", "moe.cc" ?
At the moment the only thing I can think of is
$ bazel query --output=build //:stooges | perl -nwle 'print $1 if /srcs\s*=\s*\[([^]]*)\]/'
Get all labels listed in the srcs attribute:
bazel query 'labels(srcs,//your_package:your_target)'
In your case bazel query 'labels(srcs,//:stooges)'.
Should return:
//:larry.cc
//:curly.cc
//:moe.cc
If you want to have all hdrs and srcs labels:
bazel query 'labels(srcs,//your_package:your_target) union labels(hdrs,//your_package:your_target)'
You can also make use of Bazel Aspects to query for source files. More details here.
Related
New to bazel so please bear with me :) I have a genrule which basically downloads and unpacks a a package:
genrule(
name = "extract_pkg",
srcs = ["#deb_pkg//file:pkg.deb"],
outs = ["pkg_dir"],
cmd = "dpkg-deb --extract $< $(#D)/pkg_dir",
)
Naturally pkg_dir here is a directory. There is another rule which uses this rule as input to create executable, but the main point is that I now need to add a rule (or something) which will allow me to use some headers from that package. This rule is used as an input to a cc_library which is then used in other parts of the repository to get access to the headers. Tried like this:
genrule(
name = "pkg_headers",
srcs = [":extract_pkg"],
outs = [
"pkg_dir/usr/include/pkg/h1.h",
"pkg_dir/usr/include/pkg/h2.h"
]
)
But it seems Bazel doesn't like the fact that both rules use the same directory as output, even though the second one doesn't do anything (?):
output file 'pkg_dir' of rule 'extract_pkg' conflicts with output file 'pkg_dir/usr/include/pkg/h1.h' of rule 'pkg_headers'
It works fine if I use different "root" directory for both rules, but I think there must be some better way to do this.
EDIT
I tried to use declare_directory as follows (compiled from different sources):
unpack_deb.bzl:
def _unpack_deb_impl(ctx):
input_deb_file = ctx.file.deb
output_dir = ctx.actions.declare_directory(ctx.attr.name + ".cc")
print(input_deb_file.path)
print(output_dir.path)
ctx.actions.run_shell(
inputs = [ input_deb_file ],
outputs = [ output_dir ],
arguments = [ input_deb_file.path, output_dir.path ],
progress_message = "Unpacking %s to %s" % (input_deb_file.path, output_dir.path),
command = "dpkg-deb --extract \"$1\" \"$2\"",
)
return [DefaultInfo(files = depset([output_dir]))]
unpack_deb = rule(
implementation = _unpack_deb_impl,
attrs = {
"deb": attr.label(
mandatory = True,
allow_single_file = True,
doc = "The .deb file to be unpacked",
),
},
doc = """
Unpacks a .deb file and returns a directory.
""",
)
BUILD.bazel:
load(":unpack_deb.bzl", "unpack_deb")
unpack_deb(
name = "pkg_dir",
deb = "#deb_pkg//file:pkg.deb"
)
cc_library(
name = "headers",
linkstatic = True,
srcs = [ "pkg_dir" ],
hdrs = ["pkg_dir.cc/usr/include/pkg/h1.h",
"pkg_dir.cc/usr/include/pkg/h2.h"],
strip_include_prefix = "pkg_dir.cc/usr/include",
)
The trick with adding .cc so the input can be accepted by cc_library was stolen from this answer. However the command fails on
ERROR: missing input file 'blah/blah/pkg_dir.cc/usr/include/pkg/h1.h'
From the library.
When I run with debug, I can see the command being "executed" (strange thing is that I don't always see this printout):
SUBCOMMAND: # //blah/pkg:pkg_dir [action 'Unpacking tmp/deb_pkg/file/pkg.deb to blah/pkg/pkg_dir.cc', configuration: xxxx]
(cd /home/user/.../execroot/src && \
exec env - \
/bin/bash -c 'dpkg-deb --extract "$1" "$2"' '' tmp/deb_pkg/file/pkg.deb bazel-out/.../pkg/pkg_dir.cc)
After execution, bazel-out/.../pkg/pkg_dir.cc exists but is empty. If I run the command manually it extracts files correctly. What might be the reason? Also, is it correct that there's an empty string directly after bash command line string?
Bazel's genrule doesn't work very well with directory outputs. See https://docs.bazel.build/versions/master/be/general.html#general-advice
Bazel mostly works with individual files, although there's some support for working with directories in Starlark rules with https://docs.bazel.build/versions/master/skylark/lib/actions.html#declare_directory
Your best bet is probably to extract all the files you're interested in in the genrule, then create filegroups for the different groups of files:
genrule(
name = "extract_pkg",
srcs = ["#deb_pkg//file:pkg.deb"],
outs = [
"pkg_dir/usr/include/pkg/h1.h",
"pkg_dir/usr/include/pkg/h2.h",
"pkg_dir/other_files/file1",
"pkg_dir/other_files/file2",
],
cmd = "dpkg-deb --extract $< $(#D)/pkg_dir",
)
filegroup(
name = "pkg_headers",
srcs = [
":pkg_dir/usr/include/pkg/h1.h",
":pkg_dir/usr/include/pkg/h2.h",
],
)
filegroup(
name = "pkg_other_files",
srcs = [
":pkg_dir/other_files/file1",
":pkg_dir/other_files/file2",
],
)
If you've seen glob, you might be tempted to use glob(["pkg_dir/usr/include/pkg/*.h"]) or similar for the srcs of the filegroup, but note that glob works only with "source files", which means files already on disk, not with the outputs of other rules.
There are rules for creating debs, but I'm not aware of rules for importing them. It's possible to write such rules using Starlark:
https://docs.bazel.build/versions/master/skylark/repository_rules.html
With repository rules, it's possible to avoid having to explicitly write out all the files you want to extract, among other things. Might be more work than you want to do though.
I want to build "foo.c" as a library and then execute "readelf" on the generated .so but not the ".a", how can I write it in bazel?
The following BUILD.bazel file doesn't work:
cc_library(
name = "foo",
srcs = ["foo.c"],
)
genrule(
name = "readelf_foo",
srcs = ["libfoo.so"],
outs = ["readelf_foo.txt"],
cmd = "readelf -a $(SRCS) > $#",
)
The error is "missing input file '//:libfoo.so'".
Changing the genrule's srcs attribute to ":foo" passes both the ".a" and ".so" file to readelf, which is not what I need.
Is there any way to specify which output of ":foo" to pass to the genrule?
cc_library produces several outputs, which are separated by output groups. If you want to get only .so outputs, you can use filegroup with dynamic_library output group.
So, this should work:
cc_library(
name = "foo",
srcs = ["foo.c"],
)
filegroup(
name='libfoo',
srcs=[':foo'],
output_group = 'dynamic_library'
)
genrule(
name = "readelf_foo",
srcs = [":libfoo"],
outs = ["readelf_foo.txt"],
cmd = "readelf -a $(SRCS) > $#",
)
We're using Bazel at work for wrapping and organizing some build systems that we use for our projects. Since, each project has its own build system (like bitbake for example), we can't use Bazel for the actual building (e.g creating cc_binary and cc_library rules that contain all the source code). But we do use it for wrapping and for a uniformed API.
I want to make a shell script (A) dependent of another shell script (B), and I want to do so, such that B will run before A. Since we have some configurations inside the BUILD files, I can't simply run from command line B then A. I need Bazel to do so, and inject the configuration to both scripts. Something like :
In the BUILD file of one component:
sh_binary(
name = "bazel_build_multivisor",
srcs = ["bazel-build-multivisor.sh"],
data = ["wrkspceinfo"],
deps = ["//core-build/components/bazel-pull:bazel_pull_multivisor"])
In the BUILD file of a second component which is in charge of pulling from git:
sh_binary(
name = "bazel_pull_multivisor",
srcs = ["pull_repo_repository.sh"],
data = ["wrkspceinfo", "pull_repo_repository.sh"],
args = [ARGS, "NAME=multivisor_repo;MODULE_NAME=*;GIT_BRANCH=*;MANIFEST=*.xml;PULL_PARAMS=8,8,8"],)
Meaning, I want a sh_binary rule to depend on a different sh_binary rule.
Is this possible? Is there a better way doing so?
Thanks.
Consider using a genrule.
You can specify both sh_binary targets in the genrule.tools attribute, and then in the genrule.cmd attribute, specify the shell command that calls them in order.
For example,
genrule(
name = "pull_and_build",
srcs = "some_config_file.txt",
outs = ["built_project.tar.gz"]
tools = [":pull_sh", ":build_sh"],
cmd = """
$(location :pull_sh) $< | \
xargs -1 $(location :build_sh) | \
xargs -1 tar -czvf $#"""
)
Another possible way is to wrap the dependency sh_binary in a sh_library instead, and aggregate them at the top level sh_binary.
I'm looking for a good recipe to run "checks" or "verify" steps in Bazel, like go vet, gofmt, pylint, cppcheck. These steps don't create any output file. The only thing that matters is the return code (like a test).
Right now I'm using the following recipe:
sh_test(
name = "verify-pylint",
srcs = ["verify-pylint.sh"],
data = ["//:all-srcs"],
)
And verify-pylint.sh looks like this:
find . -name '*.py' | xargs pylint
This has two problems:
The verify logic is split between the shell script and the BUILD file. Ideally I would like to have both in the same place (in the BUILD file)
Anytime one of the source file changes (in //:all-srcs), bazel test verify-pylint re-runs pylint on every single file (and that can be expensive/slow).
What is the idiomatic way in bazel to run these steps?
There are more than one solutions.
The cleanest way is to do the verification at build time: you create a genrule for each file (or batch of files) you want to verify, and if verification succeeds, the genrule outputs something, if it fails, then the rule outputs nothing, which automatically fails the build as well.
Since success of verification depends on the file's contents, and the same input should yield the same output, the genrules should produce an output file that's dependent on the contents of the input(s). The most convenient thing is to write the digest of the file(s) to the output if verification succeeded, and no output if verification fails.
To make the verifier reusable, you could create a Skylark macro and use it in all your packages.
To put this all together, you'd write something like the following.
Contents of //tools:py_verify_test.bzl:
def py_verify_test(name, srcs, visibility = None):
rules = {"%s-file%d" % (name, hash(s)): s for s in srcs}
for rulename, src in rules.items():
native.genrule(
name = rulename,
srcs = [s],
outs = ["%s.md5" % rulename],
cmd = "$(location //tools:py_verifier) $< && md5sum $< > $#",
tools = ["//tools:py_verifier"],
visibility = ["//visibility:private"],
)
native.sh_test(
name = name,
srcs = ["//tools:build_test.sh"],
data = rules.keys(),
visibility = visibility,
)
Contents of //tools:build_test.sh:
#!/bin/true
# If the test rule's dependencies could be built,
# then all files were successfully verified at
# build time, so this test can merely return true.
Contents of //tools:BUILD:
# I just use sh_binary as an example, this could
# be a more complicated rule of course.
sh_binary(
name = "py_verifier",
srcs = ["py_verifier.sh"],
visibility = ["//visibility:public"],
)
Contents of any package that wants to verify files:
load("//tools:py_verify_test.bzl", "py_verify_test")
py_verify_test(
name = "verify",
srcs = glob(["**/*.py"]),
)
A simple solution.
In your BUILD file:
load(":gofmt.bzl", "gofmt_test")
gofmt_test(
name = "format_test",
srcs = glob(["*.go"]),
)
In gofmt.bzl:
def gofmt_test(name, srcs):
cmd = """
export TMPDIR=.
out=$$(gofmt -d $(SRCS))
if [ -n "$$out" ]; then
echo "gmfmt failed:"
echo "$$out"
exit 1
fi
touch $#
"""
native.genrule(
name = name,
cmd = cmd,
srcs = srcs,
outs = [name + ".out"],
tools = ["gofmt.sh"],
)
Some remarks:
If your wrapper script grows, you should put it in a separate .sh file.
In the genrule command, we need $$ instead $ due to escaping (see documentation)
gofmt_test is actually not a test and will run with bazel build :all. If you really need a test, see Laszlo's example and call sh_test.
I call touch to create a file because genrule requires an output to succeed.
export TMPDIR=. is needed because by default the sandbox prevents writing in other directories.
To cache results for each file (and avoid rechecking a file that hasn't changed), you'll need to create multiple actions. See Laszlo's for loop.
To simplify the code, we could provide a generic rule. Maybe this is something we should put in a standard library.
I need to copy some files to binary directory while preserving their names. What I've got so far:
filegroup(
name = "resources",
srcs = glob(["resources/*.*"]),
)
genrule(
name = "copy_resources",
srcs = ["//some/package:resources"],
outs = [ ],
cmd = "cp $(SRCS) $(#D)",
local = 1,
output_to_bindir = 1,
)
Now I have to specify file names in outs but I can't seem to figure out how to resolve the labels to obtain the actual file names.
To make a filegroup available to a binary (executed using bazel run) or to a test (when executed using bazel test) then one usually lists the filegroup as part of the data of the binary, like so:
cc_binary(
name = "hello-world",
srcs = ["hello-world.cc"],
data = [
"//your_project/other/deeply/nested/resources:other_test_files",
],
)
# known to work at least as of bazel version 0.22.0
Usually the above is sufficient.
However, the executable must then recurse through the directory structure "other/deeply/nested/resources/" in order to find the files from the indicated filegroup.
In other words, when populating the runfiles of an executable, bazel preserves the directory nesting that spans from the WORKSPACE root to all the packages enclosing the given filegroup.
Sometimes, this preserved directory nesting is undesirable.
THE CHALLENGE:
In my case, I had several filegroups located at various points in my project directory tree, and I wanted all the individual files of those groups to end up side-by-side in the runfiles collection of the test binary that would consume them.
My attempts to do this with a genrule were unsuccessful.
In order to copy individual files from multiple filegroups, preserving the basename of each file but flattening the output directory, it was necessary to create a custom rule in a bzl bazel extension.
Thankfully, the custom rule is fairly straightforward.
It uses cp in a shell command much like the unfinished genrule listed in the original question.
The extension file:
# contents of a file you create named: copy_filegroups.bzl
# known to work in bazel version 0.22.0
def _copy_filegroup_impl(ctx):
all_input_files = [
f for t in ctx.attr.targeted_filegroups for f in t.files
]
all_outputs = []
for f in all_input_files:
out = ctx.actions.declare_file(f.basename)
all_outputs += [out]
ctx.actions.run_shell(
outputs=[out],
inputs=depset([f]),
arguments=[f.path, out.path],
# This is what we're all about here. Just a simple 'cp' command.
# Copy the input to CWD/f.basename, where CWD is the package where
# the copy_filegroups_to_this_package rule is invoked.
# (To be clear, the files aren't copied right to where your BUILD
# file sits in source control. They are copied to the 'shadow tree'
# parallel location under `bazel info bazel-bin`)
command="cp $1 $2")
# Small sanity check
if len(all_input_files) != len(all_outputs):
fail("Output count should be 1-to-1 with input count.")
return [
DefaultInfo(
files=depset(all_outputs),
runfiles=ctx.runfiles(files=all_outputs))
]
copy_filegroups_to_this_package = rule(
implementation=_copy_filegroup_impl,
attrs={
"targeted_filegroups": attr.label_list(),
},
)
Using it:
# inside the BUILD file of your exe
load(
"//your_project:copy_filegroups.bzl",
"copy_filegroups_to_this_package",
)
copy_filegroups_to_this_package(
name = "other_files_unnested",
# you can list more than one filegroup:
targeted_filegroups = ["//your_project/other/deeply/nested/library:other_test_files"],
)
cc_binary(
name = "hello-world",
srcs = ["hello-world.cc"],
data = [
":other_files_unnested",
],
)
You can clone a complete working example here.