I've got two repositories, Client and Library.
Inside of Client's WORKSPACE file Client imports Library as a http_archive with the name "foo".
Inside of Client, I want to use Library macros that reference targets inside Library. My problem is that the Library macros don't know that were imported as "foo", so when the macro is expanded the targets are not found.
library/WORKSPACE:
workspace(name = "library")
library/some.bzl:
def my_macro():
native.java_library(name = "my_macro_lib",
deps = ["#library//:my_macro_lib_dependnecy"]
)
library/BUILD.bazel:
java_library(name = "my_macro_lib_dependnecy",
...
)
client/WORKSPACE:
workspace(name = "client")
http_archive(
name = "library",
urls = [...],
strip_prefix = ...,
sha256 = ...,
)
Because both workspaces use the same name for library workspace (name = "library") and because the macro refers to the workspace name in its dependencies (#library//:my_macro_lib_dependnecy) this works.
Note this works but has some quirks which will be resolved in 0.17.0
Related
I have two repositories with Bazel builds. The one uses the Bazel imports for building protobuf types. When I make this repo/workspace a dependency of my other repo/workspace I have to re-add all of the protobuf imports for rules_proto and rules_cc to this new dependent workspace. Is there a way to not have to do this, or is this expected behavior with Bazel builds?
Example:
WORKSPACE A:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "bazel_skylib",
strip_prefix = "bazel-skylib-67bfa0ce4de5d4b512178d5f63abad1696f6c05b",
urls = ["https://github.com/bazelbuild/bazel-skylib/archive/67bfa0ce4de5d4b512178d5f63abad1696f6c05b.tar.gz"],
)
http_archive(
name = "rules_cc",
sha256 = "d1886f0ea5b6cfe7519b87030811f52620db31bcca7ef9964aa17af2d14f23c4",
strip_prefix = "rules_cc-cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be.tar.gz",
"https://github.com/bazelbuild/rules_cc/archive/cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be.tar.gz",
],
)
http_archive(
name = "rules_proto",
sha256 = "fb7f1959d2d2bf4d7a1f4f29d650845a9a2303b7879c6792320ba8244910ab01",
strip_prefix = "rules_proto-3212323502e21b819ac4fbdd455cb227ad0f6394",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_proto/archive/3212323502e21b819ac4fbdd455cb227ad0f6394.tar.gz",
"https://github.com/bazelbuild/rules_proto/archive/3212323502e21b819ac4fbdd455cb227ad0f6394.tar.gz",
],
)
load("#rules_cc//cc:repositories.bzl", "rules_cc_dependencies")
rules_cc_dependencies()
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()
WORKSPACE B (has dependency on WORKSPACE A):
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
name = "workspace-a",
branch = "bazel",
remote = "url/to/my/git/repo",
)
# WHY DO I HAVE TO DO THE FOLLOWING WHEN IT'S DONE IN WORKSPACE A ALREADY?
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "bazel_skylib",
strip_prefix = "bazel-skylib-67bfa0ce4de5d4b512178d5f63abad1696f6c05b",
urls = ["https://github.com/bazelbuild/bazel-skylib/archive/67bfa0ce4de5d4b512178d5f63abad1696f6c05b.tar.gz"],
)
http_archive(
name = "rules_cc",
sha256 = "d1886f0ea5b6cfe7519b87030811f52620db31bcca7ef9964aa17af2d14f23c4",
strip_prefix = "rules_cc-cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be.tar.gz",
"https://github.com/bazelbuild/rules_cc/archive/cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be.tar.gz",
],
)
http_archive(
name = "rules_proto",
sha256 = "fb7f1959d2d2bf4d7a1f4f29d650845a9a2303b7879c6792320ba8244910ab01",
strip_prefix = "rules_proto-3212323502e21b819ac4fbdd455cb227ad0f6394",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_proto/archive/3212323502e21b819ac4fbdd455cb227ad0f6394.tar.gz",
"https://github.com/bazelbuild/rules_proto/archive/3212323502e21b819ac4fbdd455cb227ad0f6394.tar.gz",
],
)
load("#rules_cc//cc:repositories.bzl", "rules_cc_dependencies")
rules_cc_dependencies()
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()
It's mostly because bazel wasn't really originally designed to be a package manager. Bazel comes from a monorepo design where everything (your applications, libraries, third-party dependencies, data, tools, toolchains, compilers, Bazel itself, etc) is checked into a single repository with a single version of everything. So there's no question of where things come from and how to integrate them beyond the domain-specific rulesets (java, c++, python, etc). It's all just there in BUILD files. The WORKSPACE design and external dependencies are for bridging the monorepo approach and non-monorepo approach.
Some mitigations of the inconveniences are the macros like rules_proto_dependencies() and rules_proto_toolchains(), but you still have to load many things at the top-level, and resolve diamond-dependency problems manually.
There's a new system call "bzlmod" that's being developed to replace all of this. It was recently experimentally released in Bazel 5.0:
https://bazel.build/docs/bzlmod
I have several third party libraries that depend on openssl, so I fetch and build openssl via repository mechanic (http_archive()). And I have GRPC that fetch boringssl that has same symbols that an openssl (after linking i get an error due collision).
I want to redefine boringsll using new_local_repository() method. But I don't known how to pass generated path to path argument and how make new_local_repository() call depend on openssl target.
The code that a want to get, looks like this:
new_local_repository(
name = "boringssl",
??? path = "bazel-out/k8-fastbuild/bin/external/openssl/openssl/", <-- generated path with openssl
build_file_content = """
cc_library(
name = "ssl",
deps = ["#openssl"],
srcs = ["lib/libssl.a"],
hdrs = glob(["include/openssl/*.h"]),
strip_include_prefix = "/include/openssl",
visibility = ["//visibility:public"],
)
cc_library(
name = "crypto",
deps = ["#openssl"],
srcs = ["lib/libcrypto.a"],
hdrs = glob(["include/openssl/*.h"]),
strip_include_prefix = "/include/openssl",
visibility = ["//visibility:public"],
)
""",
)
I would write a replacement #boringssl with aliases for each rule. Something like this as third_party/boringssl/BUILD.bazel:
alias(
name = "crypto",
actual = "#openssl//:libcrypto",
visibility = ["//visibility:public"],
)
Then you can ad it to your WORKSPACE with a relative path:
local_repository(
name = "boringssl",
path = "third_party/boringssl",
)
This also lets you freely map target names if they differ. Alternatively, you could write cc_library wrappers (without srcs, just deps) to do things like combine multiple #openssl targets into a single one for the #boringssl equivalent.
With your approach, even if you use some hacks to reuse the repository directory or copy the source files, you will still have linking problems. Bazel will build all the #openssl source files twice and then try linking both copies in. The linker might just pick one and make everything work, or it might refuse to link, or you might get subtle problems at runtime with duplicated global state.
My project depends on some external libraries which I have to bazelfy myself. Thus, my WORKSPACE:
http_archive(
name = "external_lib_component1",
build_file = "//third_party:external_lib_component1.BUILD",
sha256 = "xxx",
urls = ["https://example.org/external_lib_component1.tar.gz"],
)
http_archive(
name = "external_lib_component2",
build_file = "//third_party:external_lib_component2.BUILD",
sha256 = "yyy",
urls = ["https://example.org/external_lib_component2.tar.gz"],
)
...
The two entries above are similar, and external_lib_component{1, 2}.BUILD share a lot of code.
What is the best way to share code (macros) between them?
Just putting a shared_macros.bzl file into third_party/ won't work, because it will not be copied into
the archive location on build (only the build_file is copied).
If you place a bzl file such a In your./third_party/shared_macros.bzl into your tree as you've mentioned.
Then in the //third_party:external_lib_component1.BUILD and //third_party:external_lib_component2.BUILD you provide for your external dependencies, you can load symbols from that shared file using:
load("#//third_party:shared_macros.bzl", ...)
Labels starting with #// refer to packages from the main repository, even when used in an external dependency (as they would otherwise be rooted when starting with //. You can for check docs on labels, in particular the last paragraph.
Alternatively you can also refer to the "parent" project by its name. If in your WORKSPACE file you've had:
workspace(name = "parent")
You could say:
load("#parent//third_party:shared_macros.bzl", ...)
Note: in versions prior to 2.0.0 you might want to add --incompatible_remap_main_repo if you mixed both of above approaches in your project.
We frequently need common combinations of rules per tech stack.
That currently wastes a lot of space in WORKSPACE - and they should be kept in sync over multiple repos. It's 50+ lines after buildifier and contains too many urls, versions and hashes.
Now say I have a "technology stack" repo and do something like
load("#techstack_repo//mylang.bzl", "load_rules")
load_rules()
where load_rules would load and initialize pinned versions of e.g. rules_go, bazel-gazelle, rules_docker, rules_proto and initialize all of them in the right order so they are visible in WORKSPACE?
I did not get this to work in my tests because load apparently can not be run in a function in a bzl file - it's not a function itself.
Is there a way to do this?
Here's an example of what I tested for Java:
load("#io_bazel_rules_docker//repositories:repositories.bzl", container_repositories = "repositories")
load("#io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
load("#io_bazel_rules_docker//container:container.bzl", "container_pull")
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
load(
"#io_grpc_grpc_java//:repositories.bzl",
"IO_GRPC_GRPC_JAVA_ARTIFACTS",
"IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS",
"grpc_java_repositories",
)
load("#rules_jvm_external//:defs.bzl", "maven_install")
def prepare_stack(maven_deps = []):
container_repositories()
container_deps()
container_pull(
name = "java_base",
# https://console.cloud.google.com/gcr/images/distroless/GLOBAL/java-debian10
# tag = "11", # OpenJDK 11 as of 2020-03-04
digest = "sha256:eda9e5ae2facccc9c7016f0c2d718d2ee352743bda81234783b64aaa402679b6",
registry = "gcr.io",
repository = "distroless/java-debian10",
)
rules_proto_dependencies()
rules_proto_toolchains()
maven_install(
artifacts = maven_deps + IO_GRPC_GRPC_JAVA_ARTIFACTS,
# for improved debugging in IDE
fetch_sources = True,
generate_compat_repositories = True,
override_targets = IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS,
repositories = [
"https://repo.maven.apache.org/maven2/",
"https://repo1.maven.org/maven2",
],
strict_visibility = True,
)
grpc_java_repositories()
... all http_archive calls for the rule repos are in WORKSPACE and I want to move them in here, but that did not work at all.
As is, I get this error:
ERROR: Failed to load Starlark extension '#rules_python//python:pip.bzl'.
Cycle in the workspace file detected. This indicates that a repository is used prior to being defined.
The following chain of repository dependencies lead to the missing definition.
- #rules_python
This could either mean you have to add the '#rules_python' repository with a statement like `http_archive` in your WORKSPACE file (note that transitive dependencies are not added automatically), or move an existing definition earlier in your WORKSPACE file.
also adding rules_python does not help either.
I found a solution:
Split it into two files.
One with imports like this:
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("#bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
def declare():
maybe(
git_repository,
name = "rules_cc",
commit = "34ca16f4aa4bf2a5d3e4747229202d6cb630ebab",
remote = "https://github.com/bazelbuild/rules_cc.git",
shallow_since = "1584036492 -0700",
)
# ... for me requires at least rules_cc, rules_python, bazel_skylib
# for later proto, docker, go, java support
and another using the declared external sources:
# go
load("#io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("#bazel_gazelle//:deps.bzl", "gazelle_dependencies")
# protobuf
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
# container
load("#io_bazel_rules_docker//container:container.bzl", "container_pull")
load("#io_bazel_rules_docker//repositories:repositories.bzl", container_repositories = "repositories")
load("#io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
load("#io_bazel_rules_docker//go:image.bzl", go_image_repositories = "repositories")
def init_rules():
go_rules_dependencies()
go_register_toolchains()
gazelle_dependencies()
rules_proto_dependencies()
rules_proto_toolchains()
container_repositories()
container_deps()
go_image_repositories()
container_pull(
name = "go_static",
digest = "sha256:9b60270ec0991bc4f14bda475e8cae75594d8197d0ae58576ace84694aa75d7a",
registry = "gcr.io",
repository = "distroless/static",
)
It's a bit of a hassle, but fetch this repo with http_archive or git_repository, load the first file and call declare and load the second for init_rules and call that.
It may be a little convoluted, but it still helps to unify the stack and simplify your WORKSPACE.
I'm building a C++ executable that needs to #include "Python.h" from the user's Python installation.
To express Python.h (and the various header files it includes) in Bazel, I need to know where the Python include directory is. This location will be different on Windows and Linux, and I'd like a single Bazel configuration to build them both.
What's the best Bazel practice for referencing software that exists outside of the WORKSPACE root?
So to tell Bazel about external dependencies you need to use one of the Workspace Rules to specify the location of the external dependency, as well as the BUILD file for Bazel to use with that external dependency.
To have something work cross-platform you need to use the select() function to have Bazel select the proper library to build against for your host operating system.
Here's a stab at accomplishing it:
First we have the WORKSPACE file in your project's root that defines the two libraries and the BUILD file to use for them. Here I'm just using build_file_content but if that becomes too complex you can put it in it's own file and reference it instead. The BUILD file here exposes the prebuild library shipped with Python along with the header files needed. It also adds an include path for any targets that depend on these libraries so you can do #include "Python.h"
new_local_repository(
name = "python_linux",
path = "/usr",
build_file_content = """
cc_library(
name = "python35-lib",
srcs = ["lib/python3.5/config-3.5m-x86_64-linux-gnu/libpython3.5.so"],
hdrs = glob(["include/python3.5/*.h"]),
includes = ["include/python3.5"],
visibility = ["//visibility:public"]
)
"""
)
new_local_repository(
name = "python_win",
path = "C:/Python35",
build_file_content = """
cc_library(
name = "python35-lib",
srcs = ["libs/python35.lib"],
hdrs = glob(["include/*.h"]),
includes = ["include/"],
visibility = ["//visibility:public"]
)
"""
)
Next the BUILD file for your application. Here you need to define some config_settings. This allows us to define platform dependent settings for our build. We use the cpu value to determine the host OS.
In the cc_binary rule we use the select() function to choose the correct host library to link against based on the configuration.
config_setting(
name = "linux_x86_64",
values = {"cpu": "k8"},
visibility = ["//visibility:public"],
)
config_setting(
name = "windows",
values = {"cpu": "x64_windows"},
visibility = ["//visibility:public"],
)
cc_binary(
name="python-test",
srcs = [
"main.c",
],
deps = select({
"//:linux_x86_64": [
"#python_linux//:python35-lib"
],
"//:windows": [
"#python_win//:python35-lib"
]
})
)
FWIW here's the main.c I was playing around with to get this working.
#include "Python.h"
int main(int argc, char *argv[])
{
Py_SetProgramName(argv[0]); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
"print('Today is',ctime(time()))\n");
Py_Finalize();
return 0;
}
Another way (and perhaps simpler) is checking the python headers and libraries into your repository. You will still need to use select() to choose the correct library to link against but at least you won't need to add anything to your WORKSPACE file and can just rely on another BUILD file in your repository. If you look at the Bazel repo they check in lots of external dependencies into the third_party directory, so it's a common practice.