I am trying to run objcopy --redefine-syms=filename command in Bazel genrule.
My idea is: first create the filename using echo command in the first genrule, and use the filename in objcopy command in the second genrule. But I got this error message: in cmd attribute of genrule rule #lib_mod//:mod-symbol: label '#lib_mod//:symsfile' in $(location) expression is not a declared prerequisite of this rule. How to solve this issue?
The partial bazel file is here:
filegroup(
name = "symsfile",
srcs = [":sym_map"],
)
genrule(
name = "sym_map",
outs = ["syms.map"],
cmd = """touch $#;
echo "js_string js_string_mod" >> $#;
""",
)
genrule(
name = "mod-symbol",
srcs = [LIB_PATH + "lib.a"],
outs = [LIB_PATH + "lib_mod.a"],
cmd = "objcopy --redefine-syms=$(location symsfile) $< $#",
)
The symbol renaming file is also an input to the last genrule; it must be added to srcs:
genrule(
name = "mod-symbol",
srcs = [
LIB_PATH + "lib.a",
":sym_map",
],
outs = [LIB_PATH + "lib_mod.a"],
cmd = "objcopy --redefine-syms=$(location :sym_map) $< $#",
)
(The intermediate symsfile filegroup is not necessary.)
Related
I have a Python script named blob_to_cpp.py (located at scirpts/blob_to_cpp.py relative to the WORKSPACE.bazel file). The Python script takes an input file (e.g. weights/rt_alb.tza) and generates from that a C++ header (.h) and source file (.cpp) that I want to add to a cc_binary.
The source code of my minimal reproducible example can be found here.
The Python script can be called via:
bazel run //:blob_to_cpp -- -o weights/rt_alb.cpp -H weights/rt_alb.h weights/rt_alb.tza
I try to use a genrule to invoke the python script (bazelized via py_binary as //:blob_to_cpp)
bazel/odin_generate_cpp_from_blob.bzl:
"""
SPDX-FileCopyrightText: 2023 Julian Amann <dev#vertexwahn.de>
SPDX-License-Identifier: Apache-2.0
"""
def generate_cpp_from_blob_cc_library(name, **kwargs):
native.genrule(
name = "%s_weights_gen" % name,
srcs = ["weights/" + name],
outs = [
"weights/" + name[0:-4] + ".cpp",
"weights/" + name[0:-4] + ".h",
],
cmd = "./$(location //:blob_to_cpp) weights/%s -o weights/%s.cpp -H weights/%s.h" % (name, name[0:-4], name[0:-4]),
tools = ["//:blob_to_cpp"],
)
native.cc_library(
name = name,
srcs = ["weights/" + name[0:-4] + ".cpp"],
hdrs = ["weights/" + name[0:-4] + ".h"],
**kwargs
)
When the generate_cpp_from_blob_cc_library Bazel macro is invoked I recive the following error messages (bazel build //:Demo):
ERROR: /Users/vertexwahn/dev/Piper/BazelDemos/intermediate/Cpp/BlobToCpp/BUILD.bazel:14:34: declared output 'weights/rt_alb.cpp' was not created by genrule. This is probably because the genrule actually didn't create this output, or because the output was a directory and the genrule was run remotely (note that only the contents of declared file outputs are copied from genrules run remotely)
ERROR: /Users/vertexwahn/dev/Piper/BazelDemos/intermediate/Cpp/BlobToCpp/BUILD.bazel:14:34: declared output 'weights/rt_alb.h' was not created by genrule. This is probably because the genrule actually didn't create this output, or because the output was a directory and the genrule was run remotely (note that only the contents of declared file outputs are copied from genrules run remotely)
ERROR: /Users/vertexwahn/dev/Piper/BazelDemos/intermediate/Cpp/BlobToCpp/BUILD.bazel:14:34: Executing genrule //:rt_alb.tza_weights_gen failed: not all outputs were created or valid
Target //:Demo failed to build
My goal is to generate the files weights/rt_alb.cpp and weights/rt_alb.h. I need them in the weights folder since my cc_binary is expecting that the header file is within the weights folder (#include "weights/rt_alb.h").
My BUILD.bazel file looks like this:
load("//bazel:odin_generte_cpp_from_blob.bzl", "generate_cpp_from_blob_cc_library")
py_binary(
name = "blob_to_cpp",
srcs = ["scripts/blob_to_cpp.py"],
data = ["weights/rt_alb.tza"]
)
generate_cpp_from_blob_cc_library(
name = "rt_alb.tza"
)
cc_binary(
name = "Demo",
srcs = ["main.cpp"],
deps = [":rt_alb.tza"],
)
Any hints to get this working are welcome!
The problem
declared output 'weights/rt_alb.cpp' was not created by genrule
usually means the command in the genrule is putting the files someplace other than where bazel expects them. You can use $(location target) for inputs and outputs, as well as for tools:
# Copyright 2023 Google LLC.
# SPDX-License-Identifier: Apache-2.0
def generate_cpp_from_blob_cc_library(name, **kwargs):
src = "weights/" + name
cpp_out = "weights/" + name[0:-4] + ".cpp"
header_out = "weights/" + name[0:-4] + ".h"
native.genrule(
name = "%s_weights_gen" % name,
srcs = [src],
outs = [
cpp_out,
header_out,
],
cmd = ("./$(location //:blob_to_cpp) $(location {src}) " +
"-o $(location {cpp_out}) " +
"-H $(location {header_out})").format(
src = src,
cpp_out = cpp_out,
header_out = header_out),
tools = ["//:blob_to_cpp"],
)
native.cc_library(
name = name,
srcs = [cpp_out],
hdrs = [header_out],
**kwargs
)
New to bazel so please bear with me :) I have a genrule which basically downloads and unpacks a a package:
genrule(
name = "extract_pkg",
srcs = ["#deb_pkg//file:pkg.deb"],
outs = ["pkg_dir"],
cmd = "dpkg-deb --extract $< $(#D)/pkg_dir",
)
Naturally pkg_dir here is a directory. There is another rule which uses this rule as input to create executable, but the main point is that I now need to add a rule (or something) which will allow me to use some headers from that package. This rule is used as an input to a cc_library which is then used in other parts of the repository to get access to the headers. Tried like this:
genrule(
name = "pkg_headers",
srcs = [":extract_pkg"],
outs = [
"pkg_dir/usr/include/pkg/h1.h",
"pkg_dir/usr/include/pkg/h2.h"
]
)
But it seems Bazel doesn't like the fact that both rules use the same directory as output, even though the second one doesn't do anything (?):
output file 'pkg_dir' of rule 'extract_pkg' conflicts with output file 'pkg_dir/usr/include/pkg/h1.h' of rule 'pkg_headers'
It works fine if I use different "root" directory for both rules, but I think there must be some better way to do this.
EDIT
I tried to use declare_directory as follows (compiled from different sources):
unpack_deb.bzl:
def _unpack_deb_impl(ctx):
input_deb_file = ctx.file.deb
output_dir = ctx.actions.declare_directory(ctx.attr.name + ".cc")
print(input_deb_file.path)
print(output_dir.path)
ctx.actions.run_shell(
inputs = [ input_deb_file ],
outputs = [ output_dir ],
arguments = [ input_deb_file.path, output_dir.path ],
progress_message = "Unpacking %s to %s" % (input_deb_file.path, output_dir.path),
command = "dpkg-deb --extract \"$1\" \"$2\"",
)
return [DefaultInfo(files = depset([output_dir]))]
unpack_deb = rule(
implementation = _unpack_deb_impl,
attrs = {
"deb": attr.label(
mandatory = True,
allow_single_file = True,
doc = "The .deb file to be unpacked",
),
},
doc = """
Unpacks a .deb file and returns a directory.
""",
)
BUILD.bazel:
load(":unpack_deb.bzl", "unpack_deb")
unpack_deb(
name = "pkg_dir",
deb = "#deb_pkg//file:pkg.deb"
)
cc_library(
name = "headers",
linkstatic = True,
srcs = [ "pkg_dir" ],
hdrs = ["pkg_dir.cc/usr/include/pkg/h1.h",
"pkg_dir.cc/usr/include/pkg/h2.h"],
strip_include_prefix = "pkg_dir.cc/usr/include",
)
The trick with adding .cc so the input can be accepted by cc_library was stolen from this answer. However the command fails on
ERROR: missing input file 'blah/blah/pkg_dir.cc/usr/include/pkg/h1.h'
From the library.
When I run with debug, I can see the command being "executed" (strange thing is that I don't always see this printout):
SUBCOMMAND: # //blah/pkg:pkg_dir [action 'Unpacking tmp/deb_pkg/file/pkg.deb to blah/pkg/pkg_dir.cc', configuration: xxxx]
(cd /home/user/.../execroot/src && \
exec env - \
/bin/bash -c 'dpkg-deb --extract "$1" "$2"' '' tmp/deb_pkg/file/pkg.deb bazel-out/.../pkg/pkg_dir.cc)
After execution, bazel-out/.../pkg/pkg_dir.cc exists but is empty. If I run the command manually it extracts files correctly. What might be the reason? Also, is it correct that there's an empty string directly after bash command line string?
Bazel's genrule doesn't work very well with directory outputs. See https://docs.bazel.build/versions/master/be/general.html#general-advice
Bazel mostly works with individual files, although there's some support for working with directories in Starlark rules with https://docs.bazel.build/versions/master/skylark/lib/actions.html#declare_directory
Your best bet is probably to extract all the files you're interested in in the genrule, then create filegroups for the different groups of files:
genrule(
name = "extract_pkg",
srcs = ["#deb_pkg//file:pkg.deb"],
outs = [
"pkg_dir/usr/include/pkg/h1.h",
"pkg_dir/usr/include/pkg/h2.h",
"pkg_dir/other_files/file1",
"pkg_dir/other_files/file2",
],
cmd = "dpkg-deb --extract $< $(#D)/pkg_dir",
)
filegroup(
name = "pkg_headers",
srcs = [
":pkg_dir/usr/include/pkg/h1.h",
":pkg_dir/usr/include/pkg/h2.h",
],
)
filegroup(
name = "pkg_other_files",
srcs = [
":pkg_dir/other_files/file1",
":pkg_dir/other_files/file2",
],
)
If you've seen glob, you might be tempted to use glob(["pkg_dir/usr/include/pkg/*.h"]) or similar for the srcs of the filegroup, but note that glob works only with "source files", which means files already on disk, not with the outputs of other rules.
There are rules for creating debs, but I'm not aware of rules for importing them. It's possible to write such rules using Starlark:
https://docs.bazel.build/versions/master/skylark/repository_rules.html
With repository rules, it's possible to avoid having to explicitly write out all the files you want to extract, among other things. Might be more work than you want to do though.
I want to build "foo.c" as a library and then execute "readelf" on the generated .so but not the ".a", how can I write it in bazel?
The following BUILD.bazel file doesn't work:
cc_library(
name = "foo",
srcs = ["foo.c"],
)
genrule(
name = "readelf_foo",
srcs = ["libfoo.so"],
outs = ["readelf_foo.txt"],
cmd = "readelf -a $(SRCS) > $#",
)
The error is "missing input file '//:libfoo.so'".
Changing the genrule's srcs attribute to ":foo" passes both the ".a" and ".so" file to readelf, which is not what I need.
Is there any way to specify which output of ":foo" to pass to the genrule?
cc_library produces several outputs, which are separated by output groups. If you want to get only .so outputs, you can use filegroup with dynamic_library output group.
So, this should work:
cc_library(
name = "foo",
srcs = ["foo.c"],
)
filegroup(
name='libfoo',
srcs=[':foo'],
output_group = 'dynamic_library'
)
genrule(
name = "readelf_foo",
srcs = [":libfoo"],
outs = ["readelf_foo.txt"],
cmd = "readelf -a $(SRCS) > $#",
)
I want to create the following structure in bazel.
dir1
|_ file1
|_ file2
|_ dir2
|_file3
Creating a specific structure doesn't seem trivial.
I'm hoping there's a simple and reusable rule.
Something like:
makedir(
name = "dir1",
path = "dir1",
)
makedir(
name = "dir2",
path = "dir1/dir2",
deps = [":dir1"],
)
What I've tried:
I could create a macro with a python script, but want something cleaner.
I tried creating a genrule with mkdir -p path/to/directoy which didn't work
The use case is that I want to create a squashfs using bazel.
It's important to note that Bazel provides some packaging functions.
To create a squashfs, the command requires a directory structure populated with artifacts.
In my case, I want to create a directory structure and run mksquashfs to produce a squashfs file.
To accomplish this, I ended up modifying the basic example from bazel's docs on packaging.
load("#bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar")
genrule(
name = "file1",
outs = ["file1.txt"],
cmd = "echo exampleText > $#",
)
pkg_tar(
name = "dir1",
strip_prefix = ".",
package_dir = "/usr/bin",
srcs = [":file1"],
mode = "0755",
)
pkg_tar(
name = "dir2",
strip_prefix = ".",
package_dir = "/usr/share",
srcs = ["//main:file2.txt", "//main:file3.txt"],
mode = "0644",
)
pkg_tar(
name = "pkg",
extension = "tar.gz",
deps = [
":dir1",
":dir2",
],
)
If there's an easier way to create a tar or directory structure without the need for intermediate tars, I'll make that top answer.
You could create such a Bazel macro, that uses genrule:
def mkdir(name, out_dir, marker_file = "marker"):
"""Create an empty directory that you can use as an input in another rule
This will technically create an empty marker file in that directory to avoid Bazel warnings.
You should depend on this marker file.
"""
path = "%s/%s" % (out_dir, marker_file)
native.genrule(
name = name,
outs = [path],
cmd = """mkdir -p $$(dirname $(location :%s)) && touch $(location :%s)""" % (path, path),
)
Then you can use the outputs generated by this macro in a pkg_tar definition:
mkdir(
name = "generate_a_dir",
out_dir = "my_dir",
)
pkg_tar(
name = "package",
srcs = [
# ...
":generate_a_dir",
],
# ...
)
You can always create a genrule target or a shell_binary target that will execute bash command or a shell script (respectively) that creates these directories.
with genrule you can use bazel's $(location) that will make sure that the dir structure you create will be under an output path that is inside bazel's sandbox environment.
The genrule example shows how to use it exactly.
Here you can find more details on predefined output paths.
I have a library that depends on graphics files that are generated by a shell script.
I would like the library, when it is compiled, to use the shell script to generate the graphics files, which should be copied as if it were a 'data' statement, but whenever I try to make the library depend on the genrule, I get
in deps attribute of cc_library rule //graphics_assets
genrule rule '//graphics_assets:assets_gen_rule' is misplaced here
(expected cc_inc_library, cc_library, objc_library or
cc_proto_library)
# This is the correct format.
# Here we want to run all the shader .glsl files through the program
# file_utils:archive_tool (which we also build elsewhere) and copy the
# output .arc file to the data.
# 1. List the source files
filegroup(
name = "shader_source",
srcs = glob([
"shaders/*.glsl",
]),
)
# 2. invoke file_utils::archive_tool on the shaders
genrule(
name = "shaders_gen_rule",
srcs = [":shader_source"],
outs = ["shaders.arc"],
cmd = "echo $(locations shader_source) > temp.txt ; \
$(location //common/file_utils:archive_tool) \
--create_from_list=temp.txt \
--archive $(OUTS) ; \
$(location //common/file_utils:archive_tool) \
--ls --archive $(OUTS) ",
tools = ["//common/file_utils:archive_tool"],
)
# 3. when a a binary depends on this tool the arc file will be copied.
# This is the thing I had trouble with
cc_library(
name = "shaders",
srcs = [], # Something
data = [":shaders_gen_rule"],
linkstatic = 1,
)