I am trying to run a "hello world" server in Spark building it with Bazel, but I am getting this error:
$ bazel run //:app
INFO: Analysed target //:app (0 packages loaded).
INFO: Found 1 target...
Target //:app up-to-date:
bazel-bin/app.jar
bazel-bin/app
INFO: Elapsed time: 0.201s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at spark.Service.<clinit>(Service.java:56)
at spark.Spark$SingletonHolder.<clinit>(Spark.java:51)
at spark.Spark.getInstance(Spark.java:55)
at spark.Spark.<clinit>(Spark.java:61)
at io.app.server.Main.main(Main.java:7)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 5 more
BUILD:
java_binary(
name = "app",
main_class = "io.app.server.Main",
srcs = ["src/main/java/io/app/server/Main.java"],
deps = [
"#org_slf4j_slf4j_simple//jar",
"#com_sparkjava_spark_core//jar",
]
)
The same error happens if I don't include slf4j, and it should not be a required dependency of spark.
WORKSPACE:
maven_jar(
name = "com_sparkjava_spark_core",
artifact = "com.sparkjava:spark-core:2.7.2"
)
maven_jar(
name = "org_slf4j_slf4j_simple",
artifact = "org.slf4j:slf4j-simple:1.7.21"
)
And finally, src/main/java/io/app/server/Main.java:
package io.app.server;
import static spark.Spark.*;
public class Main {
public static void main(String[] args) {
port(3000);
get("/", (req, res) -> "Hello World");
}
}
Any idea of what I could be doing wrong here?
Found what I was missing. It seems that maven_jar does not automatically fetch the "transitive dependencies" that the library itself has, see this.
Bazel only reads dependencies listed in your WORKSPACE file. If your
project (A) depends on another project (B) which list a dependency on
a third project (C) in its WORKSPACE file, you'll have to add both B
and C to your project's WORKSPACE file. This requirement can balloon
the WORKSPACE file size, but hopefully limits the chances of having
one library include C at version 1.0 and another include C at 2.0.
Large WORKSPACE files can be generated using the tool
generate_workspace. For details, see Generate external dependencies
from Maven projects.
So the solution seems to be to write a pom.xml and use generate_workspace.
EDIT: generate_workspace seems to be deprecated, use bazel_deps instead.
Another solution might be to use maven_install
git_repository(
name = "rules_jvm_external",
commit = "22b463c485f31b240888c89d17e67c460d7e68c0",
remote = "https://github.com/bazelbuild/rules_jvm_external.git",
)
load("#rules_jvm_external//:defs.bzl", "maven_install")
maven_install(
artifacts = [
"org.apache.spark:spark-core_2.12:3.1.2",
"org.apache.spark:spark-sql_2.12:3.1.2",
],
repositories = [
"https://repo.maven.apache.org/maven2/",
]
)
Related
We are seeing duplicate builds of the same target in Bazel and wondering what could cause this.
Here is a sample output:
[52,715 / 55,135] 12 action running
Bazel package: some-pkg - Target: a_target - Generating files at bazel-out/host/bin/some-pkg/a_target_generate [for host]; 264s remote-cache, processwrapper-sandbox
Bazel package: some-pkg - Target: a_target - Generating files at bazel-out/k8-fastbuild/bin/some-pkg/a_target_generate; 264s remote-cache, processwrapper-sandbox
...
We have not been able to identify the issue. It looks like this is only happening on Linux but not on Macs.
The target a_target is a custom_rule target. It should be platform independent.
custom_rule = rule(
attrs = dict(
...
_custom_rule_java_binary = attr.label(
cfg = "host",
default = Label("//tools/bazel/build/rules/custom-rule:custom_rule_bin"),
executable = True,
),
_singlejar = attr.label(
cfg = "host",
default = Label("#bazel_tools//tools/jdk:singlejar"),
executable = True,
allow_files = True,
),
),
implementation = ...,
)
custom_rule_bin is defined as follow:
java_library(
name = "custom_rule",
srcs = glob(["src/main/java/**/*.java"]),
deps = [
...,
],
)
java_binary(
name = "custom_rule_bin",
visibility = ["//visibility:public"],
main_class = "...",
runtime_deps = [
"#org_slf4j_simple",
":custom_rule",
"//some-pkg:some_pkg", # same some-pkg where a_target is built twice
],
)
The difference is that one says "for host" and the other doesn't. Anyone knows what the extra "for host" build is?
I do have a feeling that it's somehow related to the cfg attribute on the custom rule. This is likely coming from some example code. We use the same value on all our rules which generate code. This custom rule is special because it requires code from the application being built by Bazel to run and generate additional code.
Any insights appreciated why host would be wrong and what would be the correct value.
Any ideas/tips how to debug this?
First, one note is that the host configuration is being mostly deprecated, and "exec" is usually preferred. Some info about that is here: https://bazel.build/rules/rules#configurations.
What's happening is that that target is being depended upon in multiple configurations, and so bazel will build that target in each configuration. You can use cquery to figure out what's going on
As a very simple example:
genrule(
name = "gen_bin",
outs = ["bin"],
srcs = [":gen_lib"],
exec_tools = [":gen_tool"],
cmd = "touch $#",
)
genrule(
name = "gen_tool",
outs = ["tool"],
srcs = [":gen_lib"],
cmd = "touch $#",
)
genrule(
name = "gen_lib",
outs = ["lib"],
cmd = "touch $#; sleep 10",
)
Building bin, bazel runs the gen_lib genrule twice (in parallel):
$ bazel build bin
INFO: Analyzed target //:bin (5 packages loaded, 16 targets configured).
INFO: Found 1 target...
[1 / 5] 2 actions running
Executing genrule //:gen_lib; 1s linux-sandbox
Executing genrule //:gen_lib; 1s linux-sandbox
bazel config gives the configurations that are currently in the in-memory build graph:
$ bazel config
Available configurations:
5b39bc31deb1f1d37f1f858e7eec3964394eacce5bede4456dd59d417af4a6e9 (exec)
723da02ae6d0c5577e98242c8f06ca1bd1c6d7b295c97345ac31b844bfe8f79c
8960923b9e7dc13418be101268efd8e57d80283213d18174705345598b699c6b
fd805cc1de357c04c7abac1b40bae600e3d9ee56a8d17af0c28b5031ca09bfb9 (host)
then cquery:
$ bazel cquery "rdeps(//..., gen_lib)"
INFO: Analyzed 3 targets (0 packages loaded, 1 target configured).
INFO: Found 3 targets...
//:gen_lib (5b39bc3)
//:gen_lib (8960923)
//:gen_tool (5b39bc3)
//:gen_bin (8960923)
//:gen_tool (8960923)
INFO: Elapsed time: 0.052s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions
(cquery gives the first 7 digits of the configuration hash)
--output=graph gives a dot graph which is a little more useful:
$ bazel cquery "rdeps(//..., gen_lib)" --output=graph > /tmp/graph
$ xdot /tmp/graph
So gen_bin is in the target configuration (8960923), and it depends on gen_lib, so gen_lib will also be built in the target configuration.
gen_bin also depends on gen_tool via the exec_tools attribute, and exec_tools builds everything in the exec configuration (5b39bc3).
gen_tool also depends on gen_lib, and since gen_tool is in the exec configuration, a version of gen_lib is built in the exec configuration.
(There's also another version of gen_tool in the target configuration in the output, and that's an artifact of using //... in the "universe" argument of rdeps(), since //... will capture every target. Similarly, doing bazel build //... would cause gen_tool to be built twice.)
I want to use rules_pkg
I have the following setup: Windows 10 x64 (Version 2004, Bazel 3.7.0, Visual Studio 16 2019, MSYS2 x86_64)
My minimal setup looks like this:
WORKSPACE.bazel
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
# rules_pkg
http_archive(
name = "rules_pkg",
sha256 = "6b5969a7acd7b60c02f816773b06fcf32fbe8ba0c7919ccdc2df4f8fb923804a",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_pkg/releases/download/0.3.0/rules_pkg-0.3.0.tar.gz",
"https://github.com/bazelbuild/rules_pkg/releases/download/0.3.0/rules_pkg-0.3.0.tar.gz",
],
)
BUILD.bazel
load("#bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar")
cc_binary(
name = "HelloWorld",
srcs = ["main.cpp"],
)
pkg_tar(
name = "deploy_HelloWorld",
srcs = [
":HelloWorld",
],
extension = "tar.gz",
)
main.cpp
#include <iostream>
int main() {
std::cout << "Hello World!" << std::endl;
}
When I try to build i.e. bazel build //... I get:
PS G:\dev\BazelDemos\HelloWorld> bazel build //...
INFO: Analyzed 2 targets (20 packages loaded, 143 targets configured).
INFO: Found 2 targets...
ERROR: G:/dev/bazeldemos/helloworld/BUILD.bazel:8:8: PackageTar deploy_HelloWorld.tar.gz failed (Exit 9009): build_tar.exe failed: error executing command bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/build_tar.exe --flagfile bazel-out/x64_windows-fastbuild/bin/deploy_HelloWorld.args
INFO: Elapsed time: 0.642s, Critical Path: 0.29s
INFO: 8 processes: 7 internal, 1 local.
FAILED: Build did NOT complete successfully
I can build without problems on two other Windows 10 machines with the same/similar setup. Any ideas?
More details to setup:
Path containes C:\msys64\usr\bin. BAZEL_SH is set to C:\msys64\usr\bin\bash.exe.
Python3 was not installed in not in my Path variable. Exit 9009 refers usually to an error triggered by a batch script that fails to call a specific command.
The dependency I'm trying to integrate is TensorFlowLite:
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/swift
I'm already using Carthage and SwiftPackageManager to manage dependencies of an Xcode project.
Since TensorFlowLite supports Bazel, I tried to download, build, and link the dependency using the Bazel configuration.
I downloaded the Bazel iOS demo app from here:
https://github.com/bazelbuild/examples/tree/master/tutorial
And followed the instruction in this page:
https://docs.bazel.build/versions/master/tutorial/ios-app.html
When I try to build the demo iOS app using the following command:
bazel build //ios-app
I get the following error:
ERROR: /Users/nebil/Downloads/examples-master/tutorial/ios-app/BUILD:15:14:
no such package '#TensorFlowLite//tensorflow/lite/experimental/swift': BUILD file not found in directory 'tensorflow/lite/experimental/swift' of external repository #TensorFlowLite.
Add a BUILD file to a directory to mark it as a package. and referenced by '//ios-app:TensorFlowLite'
ERROR: Analysis of target '//ios-app:ios-app' failed; build aborted: Analysis failed
In the main WORKSPACE file I added the following dependency:
git_repository(
name = "TensorFlowLite",
remote = "https://github.com/tensorflow/tensorflow",
tag = "v2.2.0",
)
And the BUILD file of the ios_app package:
load("#build_bazel_rules_apple//apple:ios.bzl", "ios_application")
load("#build_bazel_rules_swift//swift:swift.bzl", "swift_library")
swift_library(
name = 'TensorFlowLite',
deps = [
"#TensorFlowLite//tensorflow/lite/experimental/swift:TensorFlowLite",
],
)
ios_application(
name = "ios-app",
bundle_id = "Google.UrlGet",
families = [
"iphone",
"ipad",
],
infoplists = [":UrlGet/UrlGet-Info.plist"],
launch_storyboard = "UrlGet/UrlGetViewController.xib",
minimum_os_version = "8.0",
visibility = ["//visibility:public"],
deps = [:TensorFlowLite"],
)
I'm using the same project structure of the same project and simply added a new dependency to the main app.
The build is failing because the BUILD file is missing from the third party code, but this is not true:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/swift/BUILD.apple
I'm not an expert with Bazel but I don't see why this shouldn't be possibile. Any clarification would be much appreciated.
Thank you
we are trying to create a Scala project which uses Spark also but we are facing issue Encountered error while reading extension file 'intellij_info_bundled.bzl': no such package '#intellij_aspect//': No WORKSPACE file found in C:/users//_bazel_user/i45wuf6d/external/intellij_aspect. Is it has something missing in Intellij?
Scala file
package src.main.scala
object HelloWorld extends App {
def main(args: Array[String]) {
println("Hello, world!")
}
}
Build file
package(default_visibility = ["//visibility:public"])
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "hello-world",
srcs = glob(["src/main/scala/*.scala"]),
)
scala_test(
name = "Hello_test",
srcs = glob(["src/main/scala/*.scala"]),
size = "small", # Expect this test to run quickly
)
Work Space
workspace(name = "scala_example")
rules_scala_version="7522c866450cf7810eda443e91ff44d2a2286ba1" # update this as needed
http_archive(
name = "io_bazel_rules_scala",
url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip"%rules_scala_version,
type = "zip",
strip_prefix= "rules_scala-%s" % rules_scala_version
)
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()`enter code here`
# register default scala toolchain
load("#io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()
Command and Error from console
Command: C:\ProgramData\chocolatey\bin\bazel.exe build --tool_tag=ijwb:IDEA:community --keep_going --curses=no --color=yes --experimental_ui=no --progress_in_terminal_title=no --aspects=#intellij_aspect//:intellij_info_bundled.bzl%intellij_info_aspect --override_repository=intellij_aspect=C:\Users\ADMIN.IdeaIC2017.3\config\plugins\ijwb\aspect --output_groups=intellij-compile-java,intellij-compile-py -- //...:all
INFO: Loading complete. Analyzing...
ERROR: Encountered error while reading extension file 'intellij_info_bundled.bzl': no such package '#intellij_aspect//': No WORKSPACE file found in C:/users/admin/appdata/local/temp/_bazel_sandhya/criyrv6d/external/intellij_aspect.
INFO: Found 3 targets...
WARNING: failed to create one or more convenience symlinks for prefix 'bazel-':
cannot create symbolic link bazel-out -> C:/users/admin/appdata/local/temp/_bazel_sandhya/criyrv6d/execroot/scala_example/bazel-out: Cannot create junction (name=C:\users\admin\scalaprojects\example1\bazel-out, target=C:\users\admin\appdata\local\temp_bazel_sandhya\criyrv6d\execroot\scala_example\bazel-out): ERROR: src/main/native/windows/file-jni.cc(86): nativeCreateJunction(C:\users\admin\scalaprojects\example1\bazel-out, C:\users\admin\appdata\local\temp_bazel_sandhya\criyrv6d\execroot\scala_example\bazel-out): ERROR: src/main/native/windows/file.cc(128): CreateJunction(\?\C:\users\admin\scalaprojects\example1\bazel-out): Cannot create a file when that file already exists.
cannot create symbolic link bazel-out -> C:/users/admin/appdata/local/temp/_bazel_sandhya/criyrv6d/execroot/scala_example/bazel-out: Cannot create junction (name=C:\users\admin\scalaprojects\example1\bazel-out, target=C:\users\admin\appdata\local\temp_bazel_sandhya\criyrv6d\execroot\scala_example\bazel-out): ERROR: src/main/native/windows/file-jni.cc(86): nativeCreateJunction(C:\users\admin\scalaprojects\example1\bazel-out, C:\users\admin\appdata\local\temp_bazel_sandhya\criyrv6d\execroot\scala_example\bazel-out): ERROR: src/main/native/windows/file.cc(128): CreateJunction(\?\C:\users\admin\scalaprojects\example1\bazel-out): Cannot create a file when that file already exists.
.
INFO: Building...
ERROR: command succeeded, but not all targets were analyzed.
INFO: Elapsed time: 18.108s, Critical Path: 0.05s
Make failed
This is a sample Helloworld program only
In general, like #Ittai, I would suggest you open an issue in the intellij plugin github repo.
Unfortunately, your version of the plugin is no longer supported. I, too, previously ran into an issue with an older version of the plugin and was recommended to upgrade to the latest version. Which resolved the specific issue I was facing.
When reporting the issue make sure to include the following bits of information:
intellij build number
plugin version number
rules_scala version
operating system (it seems your using Windows, while most users use unix based systems)
bazel release number
how you have opened the intellij project (BUILD file, WORKSPACE, .blazeproject)
Additionally, to verify this is in fact an issue with the plugin, I would also suggest you try to reproduce this issue on a Unix based system. It seems you are using Intellij
compile on Windows. This may be Windows specific issue with aspects not being recognized.
When attempting to reproduce, make sure to clone your repository in a separate directory, close the intellij project, and reopen the project
Problem
I'm trying to modify this tutorial for compiling a TensorFlow network for use with C++. It currently requires you to copy your network file into the TensorFlow source code so the dependencies can be found, but I'd rather not do that. Note that TensorFlow is also built with Bazel.
Here is my BUILD file:
cc_binary(
name = "mnistpredict_keras",
srcs = ["mnist_keras.cc", "MNIST.h"],
deps = [
"//tensorflow/core:tensorflow",
],
)
When I try to run $ bazel build :mnistpredict_keras I get the error:
ERROR: /home/saubin/git/tf-keras-speed-test/loadgraph/BUILD:17:1: no such package 'tensorflow/core': BUILD file not found on package path and referenced by '//:mnistpredict_keras'.
ERROR: Analysis of target '//:mnistpredict_keras' failed; build aborted.
INFO: Elapsed time: 0.105s
Obviously, the problem is I'm trying to compile something in my folder ~/git/tf-keras-speed-test/loadgraph but it can't find the dependency //tensorflow/core:tensorflow. How do I properly give the path to the dependency? The documentation for deps appears to be non-existent.
Attempted Solutions
Adding a local_repository to my WORKSPACE:
local_repository(
name = "tensorflow",
path = "/home/saubin/src/tensorflow",
)
This changed nothing.
Trying to pass the full path:
cc_binary(
name = "mnistpredict_keras",
srcs = ["mnist_keras.cc", "MNIST.h"],
deps = [
"/home/saubin/src/tensorflow/tensorflow/core:tensorflow",
],
)
But I get the same error:
ERROR: /home/saubin/git/tf-keras-speed-test/loadgraph/BUILD:17:1: no such package 'tensorflow/core': BUILD file not found on package path and referenced by '//:mnistpredict_keras'.
ERROR: Analysis of target '//:mnistpredict_keras' failed; build aborted.
INFO: Elapsed time: 0.287s
Mark TensorFlow as an external repository dependency:
cc_binary(
name = "mnistpredict_keras",
srcs = ["mnist_keras.cc", "MNIST.h"],
deps = [
"#tensorflow//tensorflow/core:tensorflow",
],
)
But this gives me this error:
WARNING: /home/saubin/.cache/bazel/_bazel_saubin/74f664e7cf53364557da8b57a716c919/external/tensorflow/WORKSPACE:1: Workspace name in /home/saubin/.cache/bazel/_bazel_saubin/74f664e7cf53364557da8b57a716c919/external/tensorflow/WORKSPACE (#org_tensorflow) does not match the name given in the repository's definition (#tensorflow); this will cause a build error in future versions.
ERROR: /home/saubin/git/tensorgraph/loadgraph/BUILD:1:1: error loading package '#tensorflow//tensorflow/core': Encountered error while reading extension file 'sycl/build_defs.bzl': no such package '#local_config_sycl//sycl': error loading package 'external': The repository named 'local_config_sycl' could not be resolved and referenced by '//:mnistpredict'.
ERROR: Analysis of target '//:mnistpredict' failed; build aborted.
INFO: Elapsed time: 0.326s
The documentation for external dependencies is https://bazel.build/versions/master/docs/external.html.
The syntax that you are looking for is #tensorflow//tensorflow/core:tensorflow.
Labels that start with // refer to the current repository. Labels that start with #reponame// refer to the reponame repository.