Read file content in WORKSPACE - bazel

I'm using container_pull in my WORKSPACE file. (It's part of bazel docker rules)
Here is what it looks like:
container_pull(
name = "base-image",
registry = "registry:9999",
repository = "base-image",
digest = "sha256:e6f44554a270025c578c0f91160d809735c2589baae80bafcdeebefb0c0b04b6",
tag = "1.1.0"
)
Howeve, there is a file containing the version of base-image, and I want it to be read from that file, instead of hardcoding in WORKSPACE.
How can I read a file content in WORKSPACE?

There's no direct way to read a file from the workspace file. The container_pull rule would have to add support for reading from a file.
A workaround is to put the file that contains the information into .bzl format, and load that from the workspace file.
Something like this:
versions.bzl:
BASE_IMAGE_VERSION = "1.1.0"
WORKSPACE:
load("//:versions.bzl", "BASE_IMAGE_VERSION")
container_pull(
name = "base-image",
registry = "registry:9999",
repository = "base-image",
digest = "sha256:e6f44554a270025c578c0f91160d809735c2589baae80bafcdeebefb0c0b04b6",
tag = BASE_IMAGE_VERSION,
)
Bazel does a similar thing in its own workspace file:
https://github.com/bazelbuild/bazel/blob/669a1a2634bdf267f890cf88833c9712d4e75016/WORKSPACE#L589

Related

Bazel: How to pass relative path of a file as an "args" to java_binary?

I have a file hello.txt that lives in src/main/resources folder.
I am using a java_binary rule and need to pass this hello.txt file as an argument.
java_binary(
name = "Hello",
srcs = glob(["src/main/java/**"]),
args = ["/Users/jdoe/repo1/libraries/myproj/src/main/resources/hello.txt"],
deps = [...],
)
The above works when I provide the full path, however it fails if I try with a relative path like src/main/resources/hello.txt.
How do I provide a relative path to the args attribute?
Your binary depends on the resource file hello.txt, but Bazel is not aware of this. Make hello.txt a data dependency of your java_binary, i. e. add the attribute data = ["src/main/resources/hello.txt"]. Bazel runs your executable in a sandbox, i.e. somewhere where your hello.txt is not present. The data dependency makes sure that the file is copied to the place where it is needed.

Pull Git Repository using Bazel returns git_repository is not defined

I'm trying to build my project running Bazelbuild //... but I'm getting the following error
CPTI/WORKSPACE:1:1: name 'git_repository' is not defined
There's my WORKSPACE file:
git_repository(
name = "org_pubref_rules_protobuf",
remote = "https://github.com/pubref/rules_protobuf",
tag = "v0.8.1",
# commit = "d9523f3d443b6a4f3fabc72051d84eb5474d7745"
)
load("#org_pubref_rules_protobuf//cpp:rules.bzl", "cpp_proto_repositories")
#cpp_proto_repositories()
#BTW, #org_pubref_rules_protobuf already contains #com_google_googletest
load("//tools/build_defs:externals.bzl",
"new_patched_http_archive",
)
# The sparsehash BUILD is copied from https://github.com/livegrep/livegrep
new_patched_http_archive(
name = "com_github_sparsehash",
url = "https://github.com/sparsehash/sparsehash/archive/sparsehash-2.0.3.tar.gz",
sha256 = "05e986a5c7327796dad742182b2d10805a8d4f511ad090da0490f146c1ff7a8c",
build_file = "//third_party:BUILD.sparsehash",
strip_prefix = "sparsehash-sparsehash-2.0.3/",
patch_file = "//third_party:sparsehash.patch",
)
I found another post with the answer Solve dependency issue when using gRPC cpp in bazel
I added and it worked. Sorry, I didn't found it before.
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

Bazel get location of external dependency as command line arg for py_binary

I need the path to external (or internal) dependency to pass it as an argument to a function inside. We need the location to the folder, not specific files. Also, sometimes, we need the path to the folder where a shared library, generated by cc_library.
Python file
import cppyy
cppyy.add_include_path('path/to/external/dependency/1')
cppyy.add_library_path('path/to/another/external/dependency/2')
cppyy.add_include_path('path/to/another/internal/dependency')
cppyy.include('file/in/external/dependency')
BUILD file
py_binary(
name = "sample",
srcs = ["sample.py"],
deps = [
"#cppyy_archive//:cppyy",
],
data = [
"#external-dependency//location:target",
"//internal-dependency/location:target2"
]
)
From https://docs.bazel.build/versions/master/external.html#layout:
You can see the external directory by running:
ls $(bazel info output_base)/external
How the paths in external actually look like really depends on the rule used for the archive.
For example, if it's declared using an http_file in the WORKSPACE file:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
http_file(
name = "fenix",
urls = ["https://github.com/mozilla-mobile/fenix/archive/v76.0.0-beta.2.tar.gz"],
sha256 = "94050c664e5ec5b66cd2ca9f6a8b898987ab63d9602090533217df1a3f2dc5a9"
)
You will find that v76.0.0-beta.2.tar.gz file as external/fenix/file/downloaded:
user#host:~$ file $(bazel info output_base)/external/fenix/file/downloaded
/home/user/.cache/bazel/_bazel_user/761044447e04744e746cd54d0b4b5056/external/fenix/file/downloaded: gzip compressed data, from Unix, original size modulo 2^32 15759360

bazel with protobuf / gRPC-gateway / golang - getting started

So I am trying to convert a monorepo of micro services (C#, Go, NodeJS) to use bazel. Just playing with it for now.
I focus on one go service to get started and isolated it as a WORKSPACE.
The go service is gRPC service that uses protobuf obviously, grpc-gateway with the protoc-gen-swagger and also protoc-gen-gorm (this one does not support bazel).
The code builds using a command like go build cmd/server/server.go
I am hoping to get some guidance on how to get started to build this project with all the dependencies.
I see several rules available for protobuf/go and I am not yet comfortable browsing through them or deciding which is better (i cannot get any to work due to grpc gateway or protoc gen gorm)
- https://github.com/stackb/rules_proto
- https://github.com/bazelbuild/rules_go
- https://github.com/stackb/rules_proto/tree/master/github.com/grpc-ecosystem/grpc-gateway
Code structure looks like this:
/repo
svc1
svc2
svc3
cmd/server
BUILD.bazel
server.go
pkg
contains folders and some go files and a BUILD.bazel in each
proto
BUILD.bazel
test.proto
WORKSPACE
BUILD.bazel
Right now I only work on svc3. Later i will probably move the WORKSPACE to the parent folder.
My WORKSPACE looks like this:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_bazel_rules_go",
sha256 = "96b1f81de5acc7658e1f5a86d7dc9e1b89bc935d83799b711363a748652c471a",
urls = [
"https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/rules_go/releases/download/0.19.2/rules_go-0.19.2.tar.gz",
"https://github.com/bazelbuild/rules_go/releases/download/0.19.2/rules_go-0.19.2.tar.gz",
],
)
load("#io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
go_rules_dependencies()
go_register_toolchains()
http_archive(
name = "bazel_gazelle",
urls = [
"https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/bazel-gazelle/releases/download/0.18.1/bazel-gazelle-0.18.1.tar.gz",
"https://github.com/bazelbuild/bazel-gazelle/releases/download/0.18.1/bazel-gazelle-0.18.1.tar.gz",
],
sha256 = "be9296bfd64882e3c08e3283c58fcb461fa6dd3c171764fcc4cf322f60615a9b",
)
load("#bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository")
gazelle_dependencies()
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
name = "com_google_protobuf",
commit = "09745575a923640154bcf307fba8aedff47f240a",
remote = "https://github.com/protocolbuffers/protobuf",
shallow_since = "1558721209 -0700",
)
load("#com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
+ a bunch of go_repository() created by Gazelle
Running gazelle created a bunch of build.bazel files for my go project in each folder.
Next to the .proto, I have a generated build.bazel file:
load("#io_bazel_rules_go//go:def.bzl", "go_library")
load("#io_bazel_rules_go//proto:def.bzl", "go_proto_library")
proto_library(
name = "svc_proto",
srcs = ["test.proto"],
visibility = ["//visibility:public"],
deps = [
# the two github below are referenced as go_repository
"#com_github_infobloxopen_protoc_gen_gorm//options:proto_library", # not sure what to put after the colon
"#com_github_grpc_ecosystem_grpc_gateway//protoc-gen-swagger/options:proto_library",
"#go_googleapis//google/api:annotations_proto",
],
)
go_proto_library(
name = "svc_go_proto",
compilers = ["#io_bazel_rules_go//proto:go_grpc"],
importpath = "src/test/proto/v1",
proto = ":svc_proto",
visibility = ["//visibility:public"],
deps = [
"//github.com/infobloxopen/protoc-gen-gorm/options:go_default_library",
"//github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library",
"#go_googleapis//google/api:annotations_go_proto",
],
)
go_library(
name = "go_default_library",
embed = [":svc_go_proto"],
importpath = "src/test/proto/v1",
visibility = ["//visibility:public"],
)
Now the questions:
not sure what to put to reference other proto files: "#com_github_infobloxopen_protoc_gen_gorm//options:proto_library" ? and not sure this is the best way to reference other external libraries from git.
if i build the above using bazel build //proto/v1:svc_proto, i get: no such target '#com_github_grpc_ecosystem_grpc_gateway//protoc-gen-swagger/options:proto_library': target 'proto_library' not declared in package 'protoc-gen-swagger/options'. Probably linked to 1.
i am not sure which rule to use. As I need grpc gateway, I guess i
need to exclusively use
https://github.com/stackb/rules_proto/tree/master/github.com/grpc-ecosystem/grpc-gateway
but i can't make them to work either.
I use statik (https://github.com/rakyll/statik) to package the
swagger file in go to server the swagger. Is there any alternative
or if not, how can i call a custom bash/command as part of the build
process in the chain?
In summary, I am pretty sure my BUILD.bazel file to build the proto and library is structured wrong and would appreciate some up-to-date guidance (github is full of repos that are outdated, using outdated rules or simply not working).

bitbake SRC_URI file://

If I have a tarball, helloworld.tar.gz in a local directory, say /home/user/tarballs/, how can I make my bitbake recipe fetch from that directory?
my helloworld.bb is
SECTION = "examples"
LICENSE = "Proprietary"
LIC_FILES_CHKSUM = "file://COPYING; md5=1b1b8016e15e07a2fec59623ebf12345"
SRC_URI = "file://helloworld.tar.gz"
but when I bitbake, I get the below warning message:
WARNING: Unable to get checksum for helloworld SRC_URI entry helloworld.tar.gz: file could not be found
I read something about FILES and FILESEXTRAPATHS can influence the download path but not sure where/how to set them.
I did a bitbake -c show FILESEXTRAPATHS but get an error message:
ERROR: Nothing PROVIDES 'FILESEXTRAPATHS'
Well, if you want to fetch from a local directory, use e.g.:
SRC_URI = "file:///home/user/tarballs/helloworld.tar.gz"
The FILES and FILESEXTRAPATHS variables tells bitbake where to find files which are referenced as:
SRC_URI = "file://helloworld.tar.gz"
These files are searched for in the locations specified by those two variables. (Or rather, FILESEXTRAPATHS is searched and then some possible subdirectories of the directories specified in FILESEXTRAPATHS, amongst those the expanded values of DISTRO, MACHINE, ARCH, etc).
FILES (and FILESEXTRAPATHS) are used to find files stored together with the meta-data, i.e. under the paths meta-/recipes-/name/XXX.
See http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#var-FILES and http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#var-FILESEXTRAPATHS
Better you can keep your files in present(where the .bb file present) directory and give the below lines in your .bb file.
FILESEXTRAPATHS_prepend := "${THISDIR}:"
SRC_URI = "file://helloworld.tar.gz"
FILESEXTRAPATHS_prepend : tells to bitbake the files are present in where the .bb file is present.
Also you can edit .bb file as follows,
FILESEXTRAPATHS_prepend := "path_to_home_folder_of_source_folders:"
SRC_URI = "file://Source_floder/*"
as an example
FILESEXTRAPATHS_prepend := "/home/username/:"
SRC_URI = "file://tarballs/*"

Resources