Force Bazel to use local java only - bazel

I'd like to use bazel in a locked-down environment, one without uncontrolled internet access. I'd like to force bazel to use the locally installed java, instead of downloading one. I could not figure out the required combination of flags - this is what I tried:
$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
$ javac -version
javac 1.8.0_171
$ bazel clean --expunge
$ bazel info release
release 0.24.1
$ bazel build //hello
# This builds a 'hello world' cpp file.
# This rule also downloads some rules, would be nice to avoid it as
INFO: Build completed successfully, 6 total actions well
$ bazel test --nofetch //hello:hello_test
ERROR: /home/erenon/.cache/bazel/_bazel_erenon/afacf41c7e3fc3f4ea7510d344b4de38/external/bazel_tools/tools/jdk/BUILD:218:1: no such package '#remote_java_tools//': to fix, run
bazel fetch //...
External repository #remote_java_tools not found and fetching repositories is disabled. and referenced by '#bazel_tools//tools/jdk:JacocoCoverageRunner'
ERROR: Analysis of target '//hello:hello_test' failed; build aborted: no such package '#remote_java_tools//': to fix, run
bazel fetch //...
External repository #remote_java_tools not found and fetching repositories is disabled.
INFO: Elapsed time: 0.160s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (3 packages loaded, 27 targets configured)
FAILED: Build did NOT complete successfully (3 packages loaded, 27 targets configured)
I get the same results with a more intricate command line:
$ bazel test --nofetch --define=ABSOLUTE_JAVABASE=/usr/lib/jvm/java-8-openjdk-amd64 --host_javabase=#local_jdk//:jdk --javabase=#local_jdk//:jdk --host_java_toolchain=#bazel_tools//tools/jdk:toolchain_hostjdk8 --java_toolchain=#bazel_tools//tools/jdk:toolchain_hostjdk8 //hello:hello_test
INFO: Build options --define, --host_java_toolchain, --host_javabase, and 2 more have changed, discarding analysis cache.
ERROR: /home/erenon/.cache/bazel/_bazel_erenon/afacf41c7e3fc3f4ea7510d344b4de38/external/bazel_tools/tools/jdk/BUILD:218:1: no such package '#remote_java_tools//': to fix, run
bazel fetch //...
External repository #remote_java_tools not found and fetching repositories is disabled. and referenced by '#bazel_tools//tools/jdk:JacocoCoverageRunner'
ERROR: Analysis of target '//hello:hello_test' failed; build aborted: no such package '#remote_java_tools//': to fix, run
bazel fetch //...
External repository #remote_java_tools not found and fetching repositories is disabled.
INFO: Elapsed time: 0.215s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 50 targets configured)
FAILED: Build did NOT complete successfully (0 packages loaded, 50 targets configured)
I tried every combination of the java flags, no luck. I also tried to enable every incompatible change (--all_incompatible_changes), which yields to a different result:
INFO: Build options --incompatible_allow_python_version_transitions, --incompatible_auto_configure_host_platform, --incompatible_cc_coverage, and 24 more have changed, discarding analysis cache.
ERROR: While resolving toolchains for target //hello:hello_test: com.google.devtools.build.lib.packages.BuildFileNotFoundException: no such package '#local_config_platform//': to fix, run
bazel fetch //...
External repository #local_config_platform not found and fetching repositories is disabled.
ERROR: Analysis of target '//hello:hello_test' failed; build aborted: com.google.devtools.build.lib.packages.BuildFileNotFoundException: no such package '#local_config_platform//': to fix, run
bazel fetch //...
External repository #local_config_platform not found and fetching repositories is disabled.
INFO: Elapsed time: 0.228s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 0 targets configured)
FAILED: Build did NOT complete successfully (1 packages loaded, 0 targets configured)
It now looks for #local_config_platform.
Is there a bazel target which I can build from source and includes these dependencies?
Is there a different command line, which enables bazel test --nofetch?

It appear that bootstrapping bazel from source can solve the issue. After downloading bazel-<version>-dist.zip from GitHub, in the unzipped folder:
env EXTRA_BAZEL_ARGS="--host_javabase=#local_jdk//:jdk" bash ./compile.sh
The compiled output is placed into output/bazel. This is a self-contained Bazel binary, without an embedded JDK.
https://docs.bazel.build/versions/master/install-compile-source.html

Related

How do you debug a Bazel sandboxing failure?

Repo with full code example.
When I try to run a test with Bazel involving the tree-sitter binary, I get an Operation not permitted error without any further information.
❯ bazel test --test_output=errors --verbose_failures --sandbox_debug //tree-sitter-nbts:check-generated
INFO: Analyzed target //tree-sitter-nbts:check-generated (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //tree-sitter-nbts:check-generated (see /private/var/tmp/_bazel_varun/e55142ab9bc72292b8c54fb9627025eb/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/tree-sitter-nbts/check-generated/test.log)
INFO: From Testing //tree-sitter-nbts:check-generated:
==================== Test output for //tree-sitter-nbts:check-generated:
++ dirname /private/var/tmp/_bazel_varun/e55142ab9bc72292b8c54fb9627025eb/sandbox/darwin-sandbox/17/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/tree-sitter-nbts/check-generated.runfiles/__main__/tree-sitter-nbts/check-generated
+ cd /private/var/tmp/_bazel_varun/e55142ab9bc72292b8c54fb9627025eb/sandbox/darwin-sandbox/17/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/tree-sitter-nbts/check-generated.runfiles/__main__/tree-sitter-nbts
+ git init -b main
Initialized empty Git repository in /private/var/tmp/_bazel_varun/e55142ab9bc72292b8c54fb9627025eb/sandbox/darwin-sandbox/17/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/tree-sitter-nbts/check-generated.runfiles/__main__/tree-sitter-nbts/.git/
+ git add .
+ GIT_AUTHOR_EMAIL=_
+ GIT_AUTHOR_NAME=_
+ GIT_COMMITTER_EMAIL=_
+ GIT_COMMITTER_NAME=_
+ git commit -m 'Initial commit'
[main (root-commit) 1f79ef2] Initial commit
11 files changed, 11 insertions(+)
create mode 120000 Cargo.lock
create mode 120000 Cargo.toml
create mode 120000 bindings/rust/build.rs
create mode 120000 bindings/rust/lib.rs
create mode 120000 check-generated
create mode 120000 check-generated.sh
create mode 120000 grammar.js
create mode 120000 src/grammar.json
create mode 120000 src/node-types.json
create mode 120000 src/parser.c
create mode 120000 src/tree_sitter/parser.h
+ tree-sitter generate
Operation not permitted (os error 1)
================================================================================
Target //tree-sitter-nbts:check-generated up-to-date:
bazel-bin/tree-sitter-nbts/check-generated
INFO: Elapsed time: 0.220s, Critical Path: 0.11s
INFO: 2 processes: 2 darwin-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//tree-sitter-nbts:check-generated FAILED in 0.1s
/private/var/tmp/_bazel_varun/e55142ab9bc72292b8c54fb9627025eb/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/tree-sitter-nbts/check-generated/test.log
INFO: Build completed, 1 test FAILED, 2 total actions
What is a good way of debugging this? So far, I know the following:
Running tree-sitter generate directly in the source tree works fine.
The bazel test command works fine with --spawn_strategy=standalone, which I think confirms that this is a sandboxing issue.
The bazel test command works fine inside the Nix environment (see the linked repo). This one is a little weird, not sure what's going on here.
Configuration information:
macOS 12.4 on an M1 Mac
Bazel v5.2.0 (both in Nix and outside)
tree-sitter v0.20.6 (both in Nix and outside) (I tried using the Nix-installed binary with Bazel and that didn't seem to make any difference)
OpenJDK v11.0.15 in the global environment vs OpenJDK v11.0.11 in Nix.
GitHub issues which seem related:
ln call fails with "operation not permitted" during bazel test - This comment mentions there is no command printed with --verbose_failures --sandbox_debug.
Strange "Operation not permitted" problem for test program accessing files in the sand box on MacOS

MEDIAPIPE failed run hello world from example

UBUNTU 20.04
bazel 3.0.0- (#non-git)
openjdk 11.0.8 2020-07-14
Actually, the error itself:
dmitry#dmitry-pc:~/mediapipe$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
Starting local Bazel server and connecting to it...
INFO: SHA256 (https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip) = f358144776d3dfc5a928ef32a0e4fbe93c99b55772d70cca77a6478e34d96aa7
DEBUG: Rule 'rules_foreign_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "f358144776d3dfc5a928ef32a0e4fbe93c99b55772d70cca77a6478e34d96aa7"
DEBUG: Call stack for the definition of repository 'rules_foreign_cc' which is a http_archive (rule definition at /home/dmitry/.cache/bazel/_bazel_dmitry/870728c32b519bd8ea9ab1059ef39614/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
- <builtin>
- /home/dmitry/mediapipe/WORKSPACE:39:1
INFO: SHA256 (https://github.com/bazelbuild/rules_cc/archive/master.zip) = d1d266f0ee34a413af9e70b27354faddc2f300d873bcc388508901121e08aff4
DEBUG: Rule 'rules_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "d1d266f0ee34a413af9e70b27354faddc2f300d873bcc388508901121e08aff4"
DEBUG: Call stack for the definition of repository 'rules_cc' which is a http_archive (rule definition at /home/dmitry/.cache/bazel/_bazel_dmitry/870728c32b519bd8ea9ab1059ef39614/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
- <builtin>
- /home/dmitry/mediapipe/WORKSPACE:33:1
ERROR: /home/dmitry/.cache/bazel/_bazel_dmitry/870728c32b519bd8ea9ab1059ef39614/external/rules_cc/cc/private/rules_impl/cc_flags_supplier.bzl:28:21: rule() got unexpected keyword argument 'incompatible_use_toolchain_transition'
ERROR: While resolving toolchains for target //mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu: com.google.devtools.build.lib.packages.BuildFileContainsErrorsException: error loading package '#bazel_tools//tools/cpp': in /home/dmitry/.cache/bazel/_bazel_dmitry/870728c32b519bd8ea9ab1059ef39614/external/rules_cc/cc/defs.bzl: Extension file 'cc/private/rules_impl/cc_flags_supplier.bzl' has errors
ERROR: Analysis of target '//mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu' failed; build aborted: com.google.devtools.build.lib.packages.BuildFileContainsErrorsException: error loading package '#bazel_tools//tools/cpp': in /home/dmitry/.cache/bazel/_bazel_dmitry/870728c32b519bd8ea9ab1059ef39614/external/rules_cc/cc/defs.bzl: Extension file 'cc/private/rules_impl/cc_flags_supplier.bzl' has errors
INFO: Elapsed time: 12,845s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded, 0 targe\
ts configured)
I came across the same error and the option incompatible_use_toolchain_transition doesn't seem to be supported in bazel version 3.0.0.
Try updating your bazel following their manual.
On more thing, the latest 3.5 version of bazel seem to have another issue. So consider getting 3.4 version or apply the workaround in the Github issue.

Bazel inside Alpine container issue

I'm trying to test the build of google/or-tools using the Bazel based build system on various GNU/Linux distro by using various common distros
When trying to use bazel inside an Alpine:edge based Dockerfile (i.e. in a RUN cmd), at "docker build" time.
I don't have a consistency build between my Archlinux and on Github action workflow runners (ubuntu 18.04 IIRC).
Dockerfile: https://github.com/google/or-tools/blob/master/bazel/docker/alpine/Dockerfile
I run it using my Makefile target alpine_build in google/or-tools/bazel
ref: https://github.com/google/or-tools/blob/master/bazel/Makefile
From GH ubuntu-latest (18.04 LTS ?) runner, I got this trace
$ make alpine_build
...
Step 11/11 : RUN bazel build --curses=no --copt='-Wno-sign-compare' //...:all
---> Running in 9a2f9b6f24c7
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
DEBUG: Rule 'com_google_protobuf' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "fe1790ca0df67173702f70d5646b82f48f412b99", shallow_since = "1576187991 -0800"
DEBUG: Call stack for the definition of repository 'com_google_protobuf' which is a git_repository (rule definition at /root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/bazel_tools/tools/build_defs/repo/git.bzl:195:18):
- /home/lib/WORKSPACE:22:1
Loading: 2 packages loaded
Analyzing: 301 targets (16 packages loaded, 0 targets configured)
Analyzing: 301 targets (25 packages loaded, 43 targets configured)
Analyzing: 301 targets (26 packages loaded, 43 targets configured)
...
ref: https://github.com/google/or-tools/runs/568849544?check_suite_focus=true
So everything seems fine up to this point (still need to figure out the jdk javac issue but it's an other topic)
On the contrary on my Archlinux, I've got:
$ make alpine_build
...
Step 11/11 : RUN bazel build --curses=no --copt='-Wno-sign-compare' //...:all
---> Running in e13ca9fd3e84
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
INFO: Call stack for the definition of repository 'com_google_protobuf' which is a git_repository (rule definition at /root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/bazel_tools/tools/build_defs/repo/git.bzl:195:18):
- /home/lib/WORKSPACE:22:1
ERROR: An error occurred during the fetch of repository 'com_google_protobuf':
java.io.IOException: unlinkat(/root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/com_google_protobuf/.git/logs/refs/remotes/origin) (Directory not empty)
ERROR: no such package '#com_google_protobuf//': java.io.IOException: unlinkat(/root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/com_google_protobuf/.git/logs/refs/remotes/origin) (Directory not empty)
ERROR: no such package '#com_google_protobuf//': java.io.IOException: unlinkat(/root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/com_google_protobuf/.git/logs/refs/remotes/origin) (Directory not empty)
INFO: Elapsed time: 29.713s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
The command '/bin/sh -c bazel build --curses=no --copt='-Wno-sign-compare' //...:all' returned a non-zero code: 1
make: *** [Makefile:121: alpine_build] Error 1
I've tried to look at the file /root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/com_google_protobuf/.git/logs/refs/remotes/origin
using the previous step (alpine_devel) container:
$ make sh_alpine_devel
/home/lib # bazel build --curses=no --copt='-Wno-sign-compare' //...:all
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
INFO: Call stack for the definition of repository 'com_google_protobuf' which is a git_repository (rule definition at /root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/bazel_tools/tools/build_defs/repo/git.bzl:195:18):
- /home/lib/WORKSPACE:22:1
ERROR: An error occurred during the fetch of repository 'com_google_protobuf':
java.io.IOException: unlinkat(/root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/com_google_protobuf/.git/logs/refs/remotes/origin) (Directory not empty)
ERROR: no such package '#com_google_protobuf//': java.io.IOException: unlinkat(/root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/com_google_protobuf/.git/logs/refs/remotes/origin) (Directory not empty)
ERROR: no such package '#com_google_protobuf//': java.io.IOException: unlinkat(/root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/com_google_protobuf/.git/logs/refs/remotes/origin) (Directory not empty)
INFO: Elapsed time: 29.924s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
cat /root/.cache/bazel/_bazel_root/86fee77ec27da0053940f3f327a6fd59/external/com_google_protobuf/.git/logs/
refs/remotes/origin/revert-6272-MutableSequence-import
0000000000000000000000000000000000000000 c5fedd61a48a174054f98684d5ddbc2d11530367 root <root#Flex2.home> 1586336706 +0000 fetch origin refs/heads/*:refs/remotes/origin/* refs/tags/*:refs/tags/*: storing head
note Flex2 is my local machine...
So my questions:
Is it a known issue (ed don't find anything on java.io.IOException: unlinkat) ?
Does Bazel deal with the kernel (or uname -a etc...) which could explain why I don't have the same behaviour from one host to an other ?
May I have more trace to debug this issue ?
How can I fix it ?
Thanks,
Basically you need to tell to bazel to use the java locally installed by alpine than downloading one.
this can be done using the option --host_javabase=#local_jdk//:jdk
ref: https://github.com/google/or-tools/blob/2cb85b4eead4c38e1c54b48044f92087cf165bce/bazel/docker/alpine/Dockerfile#L26
Minimal working example:
https://github.com/Mizux/bazel-cpp/blob/main/ci/docker/alpine/Dockerfile

I am migrating a project from maven to bazel and getting error

i am following the official bazel documentation for migrating from maven to bazel and used rules_jvm_external to defined external dependencies in workspace file and Build file. But I am getting the follwing error.
C:/users/kunal_visoulia/_bazel_kunal_visoulia/3f5dzdsz/external/maven/BUILD:1212:1: in deps attribute of jvm_import rule #maven//:commons_digester_commons_digester: rule '#maven//:xml_apis_xml_apis' does not exist
ERROR: Analysis of target '//:everything' failed; build aborted: Analysis of target '#maven//:commons_digester_commons_digester' failed; build aborted
INFO: Elapsed time: 724.364s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (4 packages loaded, 371 targets configured)
Fetching #remotejdk11_win; Restarting.
i am using private nexus repo and also the official maven repo. Please help
I've successfully reproduced your error and found the following workaround:
Add xml-apis:xml-apis:1.4.01 and junit:junit-dep:4.9 to your
list of artifacts.
Add the version_conflict_policy = "pinned"
attribute to maven_install.
The root cause is https://github.com/bazelbuild/rules_jvm_external/issues/250. The missing dependencies are artifacts that relocated to new GAV coordinates, which the coursier resolver cannot handle yet. The general workaround is to manually add the legacy GAV coordinates into maven_install.
Specifically, xml-apis:xml-apis:2.0.2 relocated to xml-apis:xml-apis:1.0.b2 (but 1.4.0.1 wasn't relocated), and junit:junit-dep:4.9 relocated to junit:junit.

build abseil on windows using bazel

im trying to build abseil on windows using bazel,
the version im trying to build is :
abseil-cpp-20181200
im using the next bazel command:
bazel build
im getting this output:
INFO: Invocation ID: d85f94c1-e562-4ede-9bcd-9ab7e39020f3
Loading:
Loading: 0 packages loaded
Analyzing: 0 targets (0 packages loaded, 0 targets configured)
INFO: Analysed 0 targets (0 packages loaded, 0 targets configured).
INFO: Found 0 targets...
[0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
INFO: Elapsed time: 0.261s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
I see that some folders were created in abseil-cpp-20181200
bazel-abseil-cpp-20181200
bazel-bin
bazel-genfiles
bazel-out
bazel-testlogs
these folders are empty.
the system im using is windows7 64 bit
if more info is needed please tell me and I will supply it,
help is appreciated
bazel build itself doesn't build anything. You'll have to specify a target to tell Bazel what to build. For example, if you want to build strings, specify the target //absl/strings:strings, or //absl/strings for short:
$ bazel build //absl/strings
See the C++ Quickstart for more information.

Resources