bazel repository_rule not getting executed - bazel

I have this WORKSPACE file
workspace(name = "libedgetpu")
load(":workspace.bzl", "libedgetpu_dependencies")
libedgetpu_dependencies()
load("#org_tensorflow//tensorflow:workspace.bzl", "tf_workspace")
tf_workspace(tf_repo_name = "org_tensorflow")
#load("#coral_crosstool//:configure.bzl", "cc_crosstool")
#cc_crosstool(name = "crosstool")
local_repository(name = 'coral_crosstool1', path = 'coral_crosstool1')
load("#coral_crosstool1//:configure.bzl", "cc_crosstool1")
cc_crosstool1(name = "crosstool")
I am trying to load my local repository coral_crosstool1 which it loads (becaue I see it printing the ####1, but does not run the repository rule. How can I get the rule to execute?
def _impl(repository_ctx):
dir_labels = repository_ctx.attr.additional_system_include_directories
additional_include_dirs = ", ".join([
'"%s"' % repository_ctx.path(dir_label.relative("BUILD")).dirname
for dir_label in dir_labels
])
print("#############################2")
gcc_version = repository_ctx.execute(["/bin/bash", "-c", "gcc -dumpversion | cut -f1 -d."]).stdout
bcm2708_toolchain_root = repository_ctx.os.environ.get("BCM2708_TOOLCHAIN_ROOT", "/tools/arm-bcm2708")
repository_ctx.symlink(Label("//:BUILD.tpl"), "BUILD")
repository_ctx.template(
"cc_toolchain_config.bzl",
Label("//:cc_toolchain_config.bzl.tpl"),
{
"%{gcc_version}%": gcc_version,
"%{c_version}%": repository_ctx.attr.c_version,
"%{cpp_version}%": repository_ctx.attr.cpp_version,
"%{bcm2708_toolchain_root}%": bcm2708_toolchain_root,
"%{additional_system_include_directories}%": additional_include_dirs,
},
)
cc_crosstool1 = repository_rule(
environ = [
"BCM2708_TOOLCHAIN_ROOT",
],
attrs = {
# Consult https://gcc.gnu.org/onlinedocs/gcc/C-Dialect-Options.html for
# valid c_version and cpp_version values (-std option).
"c_version": attr.string(default = "c99"),
"cpp_version": attr.string(default = "c++11"),
"additional_system_include_directories": attr.label_list(allow_files = True),
},
implementation = _impl,
local = True,
)
print("#############################1")
My directory tree looks like this:
edgetpu
+- WORKSPACE //shown above
+- BUILD
+- coral_crosstool1
+- BUILD //this is empty
+- WORKSPACE // also empty
+- configure.bzl //shown above
+- cc_toolchain_config.bzl.tpl
+- BUILD.tpl
I need the rule to execute so it can generate the cc_toolchain_config.bzl from cc_toolchain_config.bzl.tpl
I have at this for many days. Any help will be appreciated!
This is the output
bazel build --sandbox_debug --subcommands --experimental_repo_remote_exec --compilation_mode=opt --define darwinn_portable=1 --action_env PYTHON_BIN_PATH=/usr/bin/python3 --cpu=armv6 --embed_label='TENSORFLOW_COMMIT=f394a768719a55b5c351ed1ecab2ec6f16f99dd4' --stamp -s --crosstool_top=#coral_crosstool1//:toolchains --cpu=armv6 --compiler=gcc --linkopt=-L/usr/lib/arm-linux-gnueabihf/ //tflite/public:libedgetpu_direct_all.so
DEBUG: /home/dev/.cache/bazel/_bazel_dev/770af56a999066d501f5c33777691c07/external/coral_crosstool1/configure.bzl:53:6: #############################1
ERROR: /home/dev/oosman/.leila/lib/edgetpu/tflite/public/BUILD:95:21: every rule of type cc_binary implicitly depends upon the target '#coral_crosstool1//:toolchains', but this target could not be found because of: no such target '#coral_crosstool1//:toolchains': target 'toolchains' not declared in package '' defined by /home/dev/.cache/bazel/_bazel_dev/770af56a999066d501f5c33777691c07/external/coral_crosstool1/BUILD
ERROR: Analysis of target '//tflite/public:libedgetpu_direct_all.so' failed; build aborted: Analysis failed

I'm pretty sure your problem is in the regular rules/targets generated by the repository rule, not the repository rule itself. That missing label is coming from the --crosstool_top=#coral_crosstool1//:toolchains in your command line. coral_crosstool1/BUILD is empty, so it definitely doesn't have that target. Maybe you want --crosstool_top=#crosstool//:toolchains instead? Or to load and call a function from #crosstool//:cc_toolchain_config.bzl? It's hard to say without understanding what your repository rule is intended to do.
In general, I debug repository rules by looking at the resulting external repository folder. It should be at bazel-out/../../../external/coral_crosstool1, or you can find it with find $(bazel info output_base) -name coral_crosstool1 (although that will pick up your source folder too, so be careful which one you look at). That folder is the generated repository. Sometimes it's helpful to copy the whole thing somewhere else, replace your repository rule with local_repository in your WORKSPACE, and edit things to be the way you want them without worrying about how to write a repository rule to generate them. Then, you can work on the repository rule until it generates those same files.

Related

Finding all Nix build time dependencies, including bootstrapping ones

I'm trying to set up a Nix cache containing all the store paths needed to build a simple derivation. The goal is for this to work on an empty store, with no cache misses so I don't have to hit cache.nixos.org at all. I'm having trouble because Nix seems to download a bunch of extra bootstrapping stuff, apparently to help with fetching.
For example, consider this derivation:
# empty_store_test_simple.nix
with { inherit (import <nixpkgs> {}) fetchFromGitHub; };
(fetchFromGitHub {
owner = "codedownio";
repo = "templates";
rev = "ba68b83d25d2b74f5475521ac00de3bbb884c983";
sha256 = "sha256-LNTi1ZBEsThmGWK53U9Na1j5DKHljcS42/PRXj97p6s=";
})
If I build this on an empty store with --dry-run, I see the following:
λ nix build --impure --store ~/experimental-store --substituters https://cache.nixos.org/ --dry-run -f ./empty_store_test_simple.nix
this derivation will be built:
/nix/store/jr55kq3kk6va95rvjcdyn5jmh059007p-source.drv
these 48 paths will be fetched (24.52 MiB download, 121.41 MiB unpacked):
/nix/store/02bfycjg1607gpcnsg8l13lc45qa8qj3-libssh2-1.10.0
/nix/store/0fi0432kdh46x9kbngnmz2y7z0q68cdz-xz-5.2.5-bin
/nix/store/0rizskpri8d8qawx6qjqcnvlxcvzr1bm-keyutils-1.6.3-lib
/nix/store/1l4r0r4ab3v3a3ppir4jwiah3icalk9d-zlib-1.2.11
/nix/store/1xyz8jwyg9rya2f7gs549c7n2ah378v6-stdenv-linux
/nix/store/3h4a92kysiw3s3rvbsa6a2nys3lf8f8v-libkrb5-1.18
/nix/store/3ibnw61rlgj2lj5hycy2dn3ybpq7wapm-libev-4.33
/nix/store/5qbrz5fimkbywws73vaim8allyh8kjy5-nghttp2-1.43.0-bin
/nix/store/67x6kxbanrqafx3hg7pb3bc83i3d1v3f-gzip-1.11
/nix/store/6irxz4fbf1d1ac7wvdjf8cqb3sgmnvg8-zlib-1.2.11-dev
/nix/store/71pachqc22wlvf3xjhwjh2rqbl6l3ngg-diffutils-3.8
/nix/store/9mp06ni69a44dmrjhn28mn15brdry52w-gnused-4.8
/nix/store/9ppi191zsi7zvynkm8vy2bi22lci9iwg-bzip2-1.0.6.0.2
/nix/store/c9f15p1kwm0mw5p13wsnvd1ixrhbhb12-gcc-10.3.0-lib
/nix/store/d1n274a607fmqdgr7888nq19hdsj7av0-openssl-1.1.1l-bin
/nix/store/d8p27w4d21xs6svkaf3ij60lsw243rn2-openssl-1.1.1l-dev
/nix/store/fdbwa5jrijn0yzwl8l4xdxa0l5daf5j6-curl-7.79.1
/nix/store/fvprxgcxf4px865gdjd81fbwnxcjrg41-coreutils-9.0
/nix/store/gf6j3k1flnhayvpnwnhikkg0s5dxrn1i-openssl-1.1.1l
/nix/store/gmnh4jfjhx83aggwgwzcnrwmpmqr8fwf-gnutar-1.34
/nix/store/gmzhclix3kzhir5jmmwakwhpg6j5zwf1-acl-2.3.1
/nix/store/h0b8ajwz9lvw3a3vqrf41cxrhlx9dz7p-nghttp2-1.43.0-lib
/nix/store/h7srws2r1nalsih91lrm0hfhhar14jzm-libkrb5-1.18-dev
/nix/store/h97sr1q1rpv1ry83031q51jbkba7q0m4-bzip2-1.0.6.0.2-bin
/nix/store/ihscadskdrvwc9dvbirff51lr70cphjj-curl-7.79.1-bin
/nix/store/ikvp5db9hygc14da45lvxi1c9b4ylna9-pcre-8.44
/nix/store/ilszk5f0zcv8lifkixg47ja1f2lsgxkd-nghttp2-1.43.0-dev
/nix/store/k0qa3rjifblr2vrgx4g54a59zxlfhg90-xz-5.2.5
/nix/store/kd14wd2wfmb56zpv5y71yq2lqs11l06k-attr-2.5.1
/nix/store/ksqy6mszsld4z3w8ybxa2vkjf5cqxw3f-c-ares-1.17.2
/nix/store/l0wlqpbsvh1pgvhcdhw7qkka3d31si7k-bash-5.1-p8
/nix/store/lhambyc1v2c7qzzr5sq7p449xs1j6pg8-gnugrep-3.7
/nix/store/lypy3bif096j0qc1divwa87gdvv3r575-curl-7.79.1-dev
/nix/store/p12km8psjlmvbmi52wb9r6gfykqxcdnd-libssh2-1.10.0-dev
/nix/store/pkpynsyxm8c38z4m8ngv52c7v8vhkr2h-unzip-6.0
/nix/store/prq96vz3ywk955nnxlr7s892wf5qvbr0-mirrors-list
/nix/store/psqacrv7k5fxz6mdiawc28sxcdchb4c9-ed-1.17
/nix/store/qbdsd82q5fyr0v31cvfxda0n0h7jh03g-libunistring-0.9.10
/nix/store/qzr7r4w5gm5m20afn2wz4vlv7ah4sr89-gnumake-4.3
/nix/store/r5niwjr8r8qags2bzv9z583r9vajxag3-patchelf-0.13
/nix/store/rnx655nq2qs53yb5arv2gapa91r1wsbn-findutils-4.8.0
/nix/store/scz4zbxirykss3hh5iahgl39wk9wpaps-libidn2-2.3.2
/nix/store/sqn31ly001033hsz0dpxwcsay5qdbk2w-gawk-5.1.1
/nix/store/vslsa0l17xjcrdgm2knwj0z5hlvf73m7-perl-5.34.0
/nix/store/x6pz7c0ffcd6kxzc8m1rflvqmdbjiihh-nghttp2-1.43.0
/nix/store/yj11v0gdjqli4nzax4x48xjnh9y36b2q-curl-7.79.1-man
/nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56
/nix/store/zjm4xv4nr872mdhvv3j22bzb08rgf1hk-patch-2.7.6
However, I can't find all these paths by using the means I would expect:
λ nix repl empty_store_test_simple.nix
nix-repl> :b with import <nixpkgs> {}; closureInfo { rootPaths = [inputDerivation]; }
This derivation produced the following outputs:
out -> /nix/store/1iqm7sr2rr6i5njnfxlqqyzi567mb4cz-closure-info
λ cat /nix/store/1iqm7sr2rr6i5njnfxlqqyzi567mb4cz-closure-info/store-paths
/nix/store/icvdinlsgl6y2kxk9wzkj82a53jgpdlm-source
I would expect to see ~48 paths here, but I only see 1! How can I get all the build-time dependencies indicated by the dry run? I've seen this kind of issue in the past when IFD is present, could there be some going on in Nixpkgs?
If you build your derivation and store it in your own Nix cache that should work; Nix shouldn't need to get the build-time dependencies of that thing, it can just download the result (i.e. the source code you're fetching) from your cache.
If you want to get the build-time dependencies anyway, try:
nix-store -qR $(nix-instantiate test.nix)
I think this will get you one step closer but it's not a complete solution. You'd probably need to build all the derivations in this list or something.

How to query sibling rules from a Bazel rule

I would like to be able to do the following in a Bazel BUILD file:
alpha(
name = "hello world",
color = "blue"
)
beta(
name = "hello again"
)
Where alpha and beta are custom rules. I want beta to be able to access the color attribute of the alpha rule, without adding a label attribute. In Bazel query, I can do something like this:
bazel query 'kind(beta, siblings(kind(alpha, //...)))'
which gives me the beta which is side by side to alpha. Can I achieve the same somehow from within the implementation function of the beta rule?
def _beta_rule_impl(ctx):
# This does not exist, I wish it did: ctx.siblings(kind='alpha')
I've seen this been done with a label like this
beta(
name = "hello again",
alpha_link = ":hello world" # explicitly linking
)
but I find this a bit verbose, especially since there is a sibling query support.
The way the question is formulated, the answer is no. It is not possible.
Bazel design philosophy is to be explicit about target dependencies. Providers mechanism is meant to provide the access to the dependency graph information during the analysis phase.
It is difficult to tell what is the actual use case is. Using Aspects might be the answer.
In my scenario, I'm trying to get a genrule to call a test rule before proceeding:
genrule(
name = "generate_buf_image",
srcs = [":protos", "cookie"],
outs = ["buf-image.json"],
cmd = "$(location //third_party/buf:cas_buf_image) //example-grpc/proto/v1:proto_backwards_compatibility_check $(SRCS) >$(OUTS)",
tools = [
"//third_party/buf:cas_buf_image",
"#buf",
],
)
If cas_buf_image.sh has ls -l "example-grpc/proto/v1" >&2, it shows:
… cookie -> …/example-grpc/proto/v1/cookie
… example.proto -> …/example-grpc/proto/v1/example.proto
IOW, examining what example-grpc/proto/v1/cookie is linked to and cding to its directory then performing the git commands should work.

Bazel fetch remote file not as a WORKSPACE rule?

In Bazel, how do I fetch a remote file as a build rule not as a WORKSPACE rule?
I want to use a build rule because WORKSPACE rules are not loaded for transitively.
e.g. this fails
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
http_file(
name = "foo",
urls = [ "https://example.com" ],
sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
executable = True,
)
Error in repository_rule: 'repository rule http_file' can only be called during workspace loading
If you really want to do that, you have to implement your own rule, a naïve trivial example relying on curl to fetch could be:
def _impl(ctx):
args = ctx.actions.args()
args.add("-o", ctx.outputs.out)
args.add(ctx.attr.url)
ctx.actions.run(
outputs = [ctx.outputs.out],
executable = "curl",
arguments = [args],
)
get_stuff = rule(
_impl,
attrs = {
"url": attr.string(
mandatory = True,
),
},
outputs = {"out": "%{name}.out"},
)
But (and esp. in such a trivial) for, it comes with problems. Apart from, do you want to step out of sandbox during the build? And do you want to talk to someone across the network during the build (out of the sandbox)? Bypassing repository_cache, and possibly getting remote_cache involved (networked caching of networked fetching). Specifically in this example, if content of the file pointed to by url changes... build has no idea and only fetches it when it either hasn't done so or the url itself has changed. I.e. the implementation would need to be more robust (mimic that of http_file for instance).
But it actually sounds like you're trying to address a different problem (transitive external dependencies, for which there could be another solution). One trick used for that is to define a macro (in your first level dependency to load define the next hop) and after declaring that first step as an external dependency in your parent project, load the that macro and use it from parent project WORKSPACE. This too has a price though, namely the first level dependency has to always be present (fetched or already cached), even if build target asked for does not actually need it (as that load and macro call will always pull it in).

Bazel external dependency rebuilt unnecessarily in rule used as a tool from within another rule

I am working on a set of Bazel rules where one test rule is also executed as a tool from within another executable rule. The test rule depends on an external tool which is built by rules_foreign_cc.
symbiyosys_test = rule(
implementation = _symbiyosys_test_impl,
doc = "Formal verification of (System) Verilog.",
attrs = {
...,
"_yosys_toolchain": attr.label(
doc = "Yosys toolchain.",
default = Label("#rules_symbiyosys//symbiyosys/tools:yosys"),
),
"_yices_toolchain": attr.label(
doc = "Yices toolchain.",
default = Label("#rules_symbiyosys//symbiyosys/tools:yices"),
),
},
test = True,
)
symbiyosys_trace = rule(
implementation = _symbiyosys_trace_impl,
doc = "View VCD trace from Symbiyosys.",
attrs = {
"test": attr.label(
doc = "Symbiyosys test target to produce VCD file.",
mandatory = True,
executable = True,
cfg = "exec",
),
...,
},
executable = True,
)
With a virgin Bazel cache, when an instance of the test rule is run with bazel test //examples:counter_fail the external tool is built. The external tool is also built when an instance of the executable rule (which utilizes the test rule) is run with bazel run //examples:counter_fail_trace. Once the external tool has been built in these two contexts, subsequent tests or runs use the cached outputs.
Building the external tool twice seems unnecessary as both the test and executable rule have the same configuration ("exec"). I have a hunch that this may have to do with bazel test and bazel run invoking different command line options causing the cache to miss on the external dependency.
My question is primarily what is causing this rebuild and how do I get rid of it? And short of answering that, what are some techniques to dig into what is causing this rebuild? I have tried some basic Bazel queries, but haven't had much luck.
EDIT
I still haven't cracked this one. I do suspect a subtle difference between bazel test and bazel run but unfortunately there is limited information about how specifically the two differ in the documentation.

Bazel TestNG XML output

I have a small Java project: one package with dependencies on Google Truth, Google Guava, the JSR305 annotations, and TestNG for unit tests. I've been having some trouble running the tests with Bazel. I can create a java_test rule and run it with bazel test, but Bazel's XML output gives me a single pass/fail for the entire test suite, with no information on individual failures. The XML from TestNG gets cleaned up along with the sandbox.
To get around this, I've created a genrule for TestNG's XML, but the documentation explicitly says "don't use genrules for testing" so I'm wondering if there's a better approach.
My BUILD file looks like this:
java_library(
name='myproject',
srcs=glob(['src/main/java/**/*.java']),
deps=[
'#com_google_code_findbugs_jsr305//jar',
'#com_google_guava_guava//jar',
],
)
java_library(
name='myproject-test-lib',
srcs=glob(['src/test/java/**/*.java']),
deps=[
':myproject',
'#com_google_code_findbugs_jsr305//jar',
'#com_google_guava_guava//jar',
'#com_google_truth_truth//jar',
'#org_testng_testng//jar',
],
)
java_test(
name='myproject-test',
size='small',
runtime_deps=[
':myproject',
':myproject-test-lib',
'#org_testng_testng//jar',
'#com_beust_jcommander//jar', # Used by TestNG CLI
'#org_yaml_snakeyaml//jar', # Used by TestNG to parse YAML
'#junit_junit//jar', # Dependency of Truth
],
data=['testng.yaml'],
use_testrunner=False,
main_class='org.testng.TestNG',
args=['testng.yaml'],
)
genrule(
name='myproject-test-report',
srcs=['testng.yaml'],
tools=[
':myproject',
':myproject-test-lib',
'#com_google_code_findbugs_jsr305//jar',
'#com_google_guava_guava//jar',
'#com_google_truth_truth//jar',
'#org_testng_testng//jar',
'#com_beust_jcommander//jar', # Used by TestNG CLI
'#org_yaml_snakeyaml//jar', # Used by TestNG to parse YAML
'#junit_junit//jar', # Dependency of Truth
],
outs=['testng_report'],
cmd='$(JAVA) -cp $(location :myproject):$(location :myproject-test-lib):$(location #com_google_code_findbugs_jsr305//jar):$(location #com_google_guava_guava//jar):$(location #com_google_truth_truth//jar):$(location #org_testng_testng//jar):$(location #com_beust_jcommander//jar):$(location #org_yaml_snakeyaml//jar):$(location #junit_junit//jar) org.testng.TestNG -d $(OUTS) -usedefaultlisteners false testng.yaml'
)
...I suspect there's also a better way to deal with the classpath. My WORKSPACE file, for completeness:
workspace(name='com_example_myproject')
maven_jar(
name='com_google_code_findbugs_jsr305',
artifact='com.google.code.findbugs:jsr305:3.0.1',
sha1='f7be08ec23c21485b9b5a1cf1654c2ec8c58168d',
)
maven_jar(
name='com_google_guava_guava',
artifact='com.google.guava:guava:21.0',
sha1='3a3d111be1be1b745edfa7d91678a12d7ed38709',
)
maven_jar(
name='com_google_truth_truth',
artifact='com.google.truth:truth:0.32',
sha1='e996fb4b41dad04365112786796c945f909cfdf7',
)
maven_jar(
name='org_testng_testng',
artifact='org.testng:testng:6.11',
sha1='1fdd5e22f50b14f6d846163456e8c9a7657626fb',
)
maven_jar(
name='com_beust_jcommander',
artifact='com.beust:jcommander:1.64',
sha1='456a985ac9b12d34820e4d5de063b2c2fc43ed5a',
)
maven_jar(
name='org_yaml_snakeyaml',
artifact='org.yaml:snakeyaml:1.17',
sha1='7a27ea250c5130b2922b86dea63cbb1cc10a660c',
)
maven_jar(
name='junit_junit',
artifact='junit:junit:4.10',
sha1='e4f1766ce7404a08f45d859fb9c226fc9e41a861',
)
By default bazel test will output just a summary of the test results. To see a more detailed report you can use --test_output all. You could also set --test_summary detailed.
If this won't give you the desired output and you would prefer the testng log, I can think of 2 alternatives:
Disable sandboxing.
Declare testng_report as an input file (using
data attribute of java_test). Bazel needs to know the set of input/output files
and will remove everything not declared beforehand. Since for
java_test there is no way to declare additional output files, try
to declare it as an input if having it at all times in the package
is not an inconvenience. This is a bit hackish and I wouldn't prefer it.
Hope this helps.
I think this is a new option, but I was able to use --sandbox_writable_path to make a directory in CI writable, and then specify my test output to go to that directory.
--sandbox_writable_path=<a string> multiple uses are accumulated
For sandboxed actions, make an existing directory writable in the sandbox (if supported by the sandboxing implementation, ignored otherwise).

Resources