I'm trying to package a bundle for uploading to Google Cloud. I have an output of pkg_web from an angular build that I did, which, if I pass into this custom rule I'm generating, is a File object that is a directory of the files. The custom rule I am generating takes the app.yaml, etc, and the bundle, and uploads.
However, the bundle becomes a directory, and I need the files of that directory expanded for uploading in the root of command.
For example:
- bundle/index.html <-- bundle directory
- bundle/main.js
- app.yaml
and I need:
- index.html
- main.js
- app.yaml
My rule:
deploy(
name = "deploy",
srcs = [":bundle"] <-- pkg_web rule,
yaml = ":app.yaml"
)
Rule implementation:
def _deploy_pkg(ctx):
inputs = []
inputs.append(ctx.file.yaml)
inputs.extend(ctx.files.srcs)
script_template = """
#!/bin/bash
gcloud app deploy {yaml_path}
"""
script = ctx.actions.declare_file("%s-deploy" % ctx.label.name)
ctx.actions.write(script, script_content, is_executable = True)
runfiles = ctx.runfiles(files = inputs, transitive_files = depset(ctx.files.srcs))
return [DefaultInfo(executable = script, runfiles = runfiles)]
Thank you for your ideas!
Seems a bit excessive, but I ended using a custom shell command to accomplish this:
def _deploy_pkg(ctx):
inputs = []
out = ctx.actions.declare_directory("out")
yaml_out = ctx.actions.declare_file(ctx.file.yaml.basename)
inputs.append(out)
ctx.actions.run_shell(
outputs = [yaml_out],
inputs = [ctx.file.yaml],
arguments = [ctx.file.yaml.path, yaml_out.path],
progress_message = "Copying yaml to output directory.",
command = "cp $1 $2",
)
for f in ctx.files.srcs:
if f.is_directory:
ctx.actions.run_shell(
outputs = [out],
inputs = [f],
arguments = [f.path, out.path],
progress_message = "Copying %s to output directory.".format(f.basename),
command = "cp -a -R $1/* $2",
)
else:
out_file = ctx.actions.declare_file(f.basename)
inputs.append(out_file)
ctx.actions.run_shell(
outputs = [out_file],
inputs = [f],
arguments = [f.path, out_file.path],
progress_message = "Copying %s to output directory.".format(f.basename),
# This is what we're all about here. Just a simple 'cp' command.
# Copy the input to CWD/f.basename, where CWD is the package where
# the copy_filegroups_to_this_package rule is invoked.
# (To be clear, the files aren't copied right to where your BUILD
# file sits in source control. They are copied to the 'shadow tree'
# parallel location under `bazel info bazel-bin`)
command = "cp -a $1 $2",
)
....
``
Related
I have a Python script named blob_to_cpp.py (located at scirpts/blob_to_cpp.py relative to the WORKSPACE.bazel file). The Python script takes an input file (e.g. weights/rt_alb.tza) and generates from that a C++ header (.h) and source file (.cpp) that I want to add to a cc_binary.
The source code of my minimal reproducible example can be found here.
The Python script can be called via:
bazel run //:blob_to_cpp -- -o weights/rt_alb.cpp -H weights/rt_alb.h weights/rt_alb.tza
I try to use a genrule to invoke the python script (bazelized via py_binary as //:blob_to_cpp)
bazel/odin_generate_cpp_from_blob.bzl:
"""
SPDX-FileCopyrightText: 2023 Julian Amann <dev#vertexwahn.de>
SPDX-License-Identifier: Apache-2.0
"""
def generate_cpp_from_blob_cc_library(name, **kwargs):
native.genrule(
name = "%s_weights_gen" % name,
srcs = ["weights/" + name],
outs = [
"weights/" + name[0:-4] + ".cpp",
"weights/" + name[0:-4] + ".h",
],
cmd = "./$(location //:blob_to_cpp) weights/%s -o weights/%s.cpp -H weights/%s.h" % (name, name[0:-4], name[0:-4]),
tools = ["//:blob_to_cpp"],
)
native.cc_library(
name = name,
srcs = ["weights/" + name[0:-4] + ".cpp"],
hdrs = ["weights/" + name[0:-4] + ".h"],
**kwargs
)
When the generate_cpp_from_blob_cc_library Bazel macro is invoked I recive the following error messages (bazel build //:Demo):
ERROR: /Users/vertexwahn/dev/Piper/BazelDemos/intermediate/Cpp/BlobToCpp/BUILD.bazel:14:34: declared output 'weights/rt_alb.cpp' was not created by genrule. This is probably because the genrule actually didn't create this output, or because the output was a directory and the genrule was run remotely (note that only the contents of declared file outputs are copied from genrules run remotely)
ERROR: /Users/vertexwahn/dev/Piper/BazelDemos/intermediate/Cpp/BlobToCpp/BUILD.bazel:14:34: declared output 'weights/rt_alb.h' was not created by genrule. This is probably because the genrule actually didn't create this output, or because the output was a directory and the genrule was run remotely (note that only the contents of declared file outputs are copied from genrules run remotely)
ERROR: /Users/vertexwahn/dev/Piper/BazelDemos/intermediate/Cpp/BlobToCpp/BUILD.bazel:14:34: Executing genrule //:rt_alb.tza_weights_gen failed: not all outputs were created or valid
Target //:Demo failed to build
My goal is to generate the files weights/rt_alb.cpp and weights/rt_alb.h. I need them in the weights folder since my cc_binary is expecting that the header file is within the weights folder (#include "weights/rt_alb.h").
My BUILD.bazel file looks like this:
load("//bazel:odin_generte_cpp_from_blob.bzl", "generate_cpp_from_blob_cc_library")
py_binary(
name = "blob_to_cpp",
srcs = ["scripts/blob_to_cpp.py"],
data = ["weights/rt_alb.tza"]
)
generate_cpp_from_blob_cc_library(
name = "rt_alb.tza"
)
cc_binary(
name = "Demo",
srcs = ["main.cpp"],
deps = [":rt_alb.tza"],
)
Any hints to get this working are welcome!
The problem
declared output 'weights/rt_alb.cpp' was not created by genrule
usually means the command in the genrule is putting the files someplace other than where bazel expects them. You can use $(location target) for inputs and outputs, as well as for tools:
# Copyright 2023 Google LLC.
# SPDX-License-Identifier: Apache-2.0
def generate_cpp_from_blob_cc_library(name, **kwargs):
src = "weights/" + name
cpp_out = "weights/" + name[0:-4] + ".cpp"
header_out = "weights/" + name[0:-4] + ".h"
native.genrule(
name = "%s_weights_gen" % name,
srcs = [src],
outs = [
cpp_out,
header_out,
],
cmd = ("./$(location //:blob_to_cpp) $(location {src}) " +
"-o $(location {cpp_out}) " +
"-H $(location {header_out})").format(
src = src,
cpp_out = cpp_out,
header_out = header_out),
tools = ["//:blob_to_cpp"],
)
native.cc_library(
name = name,
srcs = [cpp_out],
hdrs = [header_out],
**kwargs
)
Suppose I am writing a custom Bazel rule for foo-compiler.
The user provides a list of source-files to the rule:
foo_library(
name = "hello",
srcs = [ "A.foo", "B.foo" ],
)
To build this without Bazel, the steps would be:
Create a config file config.json that lists the sources:
{
"srcs": [ "./A.foo", "./B.foo" ]
}
Place the config alongside the sources:
$ ls .
A.foo
B.foo
config.json
Call foo-compiler in that directory:
$ foo-compiler .
Now, in my Bazel rule implementation I can declare a file like this:
config_file = ctx.actions.declare_file("config.json")
ctx.actions.write(
output = config_file,
content = json_for_srcs(ctx.files.srcs),
)
The file is created and it has the right content.
However, Bazel does not place config.json alongside the srcs.
Is there a way to tell Bazel where to place the file?
Or perhaps I need to copy each source-file alongside the config?
You can do this with ctx.actions.symlink e.g.
srcs = []
# Declare a symlink for each src files in the same directory as the declared
# config file.Then write that symlink.
for f in ctx.files.srcs:
src = ctx.actions.declare_file(f.basename)
srcs.append(src)
ctx.actions.symlink(
output = src,
target_file = f,
)
config_file = ctx.actions.declare_file("config.json")
ctx.actions.write(
output = config_file,
content = json_for_srcs(ctx.files.srcs),
)
# Run compiler
ctx.actions.run(
inputs = srcs + [config_file],
outputs = # TODO: Up to you,
tools = [ctx.file.__compiler], #TODO: Update this to match your rule.
command = ctx.file.__compiler.path,
args = ["."],
#...
)
Note that when you return your provider that you should only return the result of your compilation not the srcs. Otherwise, you'll likely run into problems with duplicate outputs.
I want to create the following structure in bazel.
dir1
|_ file1
|_ file2
|_ dir2
|_file3
Creating a specific structure doesn't seem trivial.
I'm hoping there's a simple and reusable rule.
Something like:
makedir(
name = "dir1",
path = "dir1",
)
makedir(
name = "dir2",
path = "dir1/dir2",
deps = [":dir1"],
)
What I've tried:
I could create a macro with a python script, but want something cleaner.
I tried creating a genrule with mkdir -p path/to/directoy which didn't work
The use case is that I want to create a squashfs using bazel.
It's important to note that Bazel provides some packaging functions.
To create a squashfs, the command requires a directory structure populated with artifacts.
In my case, I want to create a directory structure and run mksquashfs to produce a squashfs file.
To accomplish this, I ended up modifying the basic example from bazel's docs on packaging.
load("#bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar")
genrule(
name = "file1",
outs = ["file1.txt"],
cmd = "echo exampleText > $#",
)
pkg_tar(
name = "dir1",
strip_prefix = ".",
package_dir = "/usr/bin",
srcs = [":file1"],
mode = "0755",
)
pkg_tar(
name = "dir2",
strip_prefix = ".",
package_dir = "/usr/share",
srcs = ["//main:file2.txt", "//main:file3.txt"],
mode = "0644",
)
pkg_tar(
name = "pkg",
extension = "tar.gz",
deps = [
":dir1",
":dir2",
],
)
If there's an easier way to create a tar or directory structure without the need for intermediate tars, I'll make that top answer.
You could create such a Bazel macro, that uses genrule:
def mkdir(name, out_dir, marker_file = "marker"):
"""Create an empty directory that you can use as an input in another rule
This will technically create an empty marker file in that directory to avoid Bazel warnings.
You should depend on this marker file.
"""
path = "%s/%s" % (out_dir, marker_file)
native.genrule(
name = name,
outs = [path],
cmd = """mkdir -p $$(dirname $(location :%s)) && touch $(location :%s)""" % (path, path),
)
Then you can use the outputs generated by this macro in a pkg_tar definition:
mkdir(
name = "generate_a_dir",
out_dir = "my_dir",
)
pkg_tar(
name = "package",
srcs = [
# ...
":generate_a_dir",
],
# ...
)
You can always create a genrule target or a shell_binary target that will execute bash command or a shell script (respectively) that creates these directories.
with genrule you can use bazel's $(location) that will make sure that the dir structure you create will be under an output path that is inside bazel's sandbox environment.
The genrule example shows how to use it exactly.
Here you can find more details on predefined output paths.
I'm trying to run qemu on the output of a cc_binary rule. For that I have created a custom rule, which is pretty similiar to this example, but instead of the cat command on the txt-file, I want to invoke qemu on the output elf-file (":test_portos.elf") of the cc_binary rule. My files are the following:
run_tests.bzl
def _impl(ctx):
# The path of ctx.file.target.path is:
'bazel-out/cortex-a9-fastbuild/bin/test/test_portos.elf'
target = ctx.file.target.path
command = "qemu-system-arm -M xilinx-zynq-a9 -cpu cortex-a9 -nographic
-monitor null -serial null -semihosting
-kernel %s" % (target)
ctx.actions.write(
output=ctx.outputs.executable,
content=command,
is_executable=True)
return [DefaultInfo(
runfiles=ctx.runfiles(files=[ctx.file.target])
)]
execute = rule(
implementation=_impl,
executable=True,
attrs={
"command": attr.string(),
"target" : attr.label(cfg="data", allow_files=True,
single_file=True, mandatory=True)
},
)
BUILD
load("//make:run_tests.bzl", "execute")
execute(
name = "portos",
target = ":test_portos.elf"
)
cc_binary(
name = "test_portos.elf",
srcs = glob(["*.cc"]),
deps = ["//src:portos",
"#unity//:unity"],
copts = ["-Isrc",
"-Iexternal/unity/src",
"-Iexternal/unity/extras/fixture/src"]
)
The problem is, that in the command (of the custom rule) the location of the ":test_portos.elf" is used and not the location of the runfile. I have also tried, like shown in the example, to use $(location :test_portos.elf) together with ctx.expand_location but the result was the same.
How can I get the location of the "test_portos.elf" runfile and insert it into the command of my custom rule?
Seems that the runfiles are safed according to the short_path of the File, so this was all I needed to change in my run_tests.bzl file:
target = ctx.file.target.short_path
I'm looking for a good recipe to run "checks" or "verify" steps in Bazel, like go vet, gofmt, pylint, cppcheck. These steps don't create any output file. The only thing that matters is the return code (like a test).
Right now I'm using the following recipe:
sh_test(
name = "verify-pylint",
srcs = ["verify-pylint.sh"],
data = ["//:all-srcs"],
)
And verify-pylint.sh looks like this:
find . -name '*.py' | xargs pylint
This has two problems:
The verify logic is split between the shell script and the BUILD file. Ideally I would like to have both in the same place (in the BUILD file)
Anytime one of the source file changes (in //:all-srcs), bazel test verify-pylint re-runs pylint on every single file (and that can be expensive/slow).
What is the idiomatic way in bazel to run these steps?
There are more than one solutions.
The cleanest way is to do the verification at build time: you create a genrule for each file (or batch of files) you want to verify, and if verification succeeds, the genrule outputs something, if it fails, then the rule outputs nothing, which automatically fails the build as well.
Since success of verification depends on the file's contents, and the same input should yield the same output, the genrules should produce an output file that's dependent on the contents of the input(s). The most convenient thing is to write the digest of the file(s) to the output if verification succeeded, and no output if verification fails.
To make the verifier reusable, you could create a Skylark macro and use it in all your packages.
To put this all together, you'd write something like the following.
Contents of //tools:py_verify_test.bzl:
def py_verify_test(name, srcs, visibility = None):
rules = {"%s-file%d" % (name, hash(s)): s for s in srcs}
for rulename, src in rules.items():
native.genrule(
name = rulename,
srcs = [s],
outs = ["%s.md5" % rulename],
cmd = "$(location //tools:py_verifier) $< && md5sum $< > $#",
tools = ["//tools:py_verifier"],
visibility = ["//visibility:private"],
)
native.sh_test(
name = name,
srcs = ["//tools:build_test.sh"],
data = rules.keys(),
visibility = visibility,
)
Contents of //tools:build_test.sh:
#!/bin/true
# If the test rule's dependencies could be built,
# then all files were successfully verified at
# build time, so this test can merely return true.
Contents of //tools:BUILD:
# I just use sh_binary as an example, this could
# be a more complicated rule of course.
sh_binary(
name = "py_verifier",
srcs = ["py_verifier.sh"],
visibility = ["//visibility:public"],
)
Contents of any package that wants to verify files:
load("//tools:py_verify_test.bzl", "py_verify_test")
py_verify_test(
name = "verify",
srcs = glob(["**/*.py"]),
)
A simple solution.
In your BUILD file:
load(":gofmt.bzl", "gofmt_test")
gofmt_test(
name = "format_test",
srcs = glob(["*.go"]),
)
In gofmt.bzl:
def gofmt_test(name, srcs):
cmd = """
export TMPDIR=.
out=$$(gofmt -d $(SRCS))
if [ -n "$$out" ]; then
echo "gmfmt failed:"
echo "$$out"
exit 1
fi
touch $#
"""
native.genrule(
name = name,
cmd = cmd,
srcs = srcs,
outs = [name + ".out"],
tools = ["gofmt.sh"],
)
Some remarks:
If your wrapper script grows, you should put it in a separate .sh file.
In the genrule command, we need $$ instead $ due to escaping (see documentation)
gofmt_test is actually not a test and will run with bazel build :all. If you really need a test, see Laszlo's example and call sh_test.
I call touch to create a file because genrule requires an output to succeed.
export TMPDIR=. is needed because by default the sandbox prevents writing in other directories.
To cache results for each file (and avoid rechecking a file that hasn't changed), you'll need to create multiple actions. See Laszlo's for loop.
To simplify the code, we could provide a generic rule. Maybe this is something we should put in a standard library.