Bazel for packages with a "bootstrap->configure->make" build? - bazel

I'm experimenting with using libhttpserver and, by extension, libmicrohttpd from inside a Bazel build. The build process for these libraries seems to go:
./bootstrap
mkdir build
cd build
../configure
make
which is a slight variation, that I haven't seen before, on the more classic configure && make workflow.
Has anyone managed to make such a library work under Bazel?
Does anyone have a public example I can crib from?
The closest thing I've found to supporting this is #rules_foreign_cc//tools/build_defs:configure.bzl#configure_make but that seems to have no concept of the bootstrap step. Even hacking it doesn't seem to work as the bootstrap script ends up failing with:
mkdir: cannot create directory 'tmpwrk23': Read-only file system
autopoint: *** cannot create directory tmpwrk23
autopoint: *** Stop.
autoreconf: autopoint failed with exit status: 1
I'm about ready to just reach for a genrule() but that seems very error prone...

I went down the same path as you and was able to get libhttpserver compiling with bazel using the rules_foreign_cc project after coming across the 6/23/20 update on this blog post. I added code below, but in general rules_foreign_cc has a make rule, which you can set override make_commands and call ./bootstrap.
WORKSPACE:
...
http_archive(
name = "rules_foreign_cc",
strip_prefix = "rules_foreign_cc-master",
url = "https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip",
)
load("#rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies")
rules_foreign_cc_dependencies(register_default_tools = True)
all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""
http_archive(
name = "rules_cc",
urls = ["https://github.com/bazelbuild/rules_cc/archive/262ebec3c2296296526740db4aefce68c80de7fa.zip"],
strip_prefix = "rules_cc-262ebec3c2296296526740db4aefce68c80de7fa",
)
http_archive(
name = "libgnutls",
build_file_content = all_content,
strip_prefix = "gnutls-3.6.15",
urls = ["https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.15.tar.xz"],
)
http_archive(
name = "libhttpserver",
build_file_content = all_content,
strip_prefix = "libhttpserver-master",
urls = ["https://github.com/etr/libhttpserver/archive/master.zip"],
)
http_archive(
name = "libmicrohttpd",
build_file_content = all_content,
strip_prefix = "libmicrohttpd-0.9.71",
urls = ["https://ftp.gnu.org/gnu/libmicrohttpd/libmicrohttpd-0.9.71.tar.gz"],
)
BUILD:
load("#rules_foreign_cc//tools/build_defs:configure.bzl", "configure_make")
load("#rules_foreign_cc//tools/build_defs:make.bzl", "make")
load("#rules_cc//cc:defs.bzl", "cc_binary", "cc_library")
package(default_visibility = ["//visibility:public"])
configure_make(
name = "libgnutls",
lib_source = "#libgnutls//:all",
configure_options = ["--with-included-unistring"],
out_include_dir = "include/gnutls",
shared_libraries = ["libgnutls.so"],
)
configure_make(
name = "libmicrohttpd",
lib_source = "#libmicrohttpd//:all",
deps = [":libgnutls"],
)
make(
name = "libhttpserver",
lib_source = "#libhttpserver//:all",
make_commands = [
"./bootstrap",
"mkdir build_dir",
"cd build_dir",
"../configure --prefix=${INSTALLDIR}",
"make",
"make install",
],
deps = [":libmicrohttpd", ":libgnutls"],
)
cc_binary(
name = "hello-world",
srcs = ["hello_world.cc"],
deps = [
":libhttpserver"
],
)
hello_world.cc (example on libhttpserver github page, run "curl -XGET -v http://localhost:8080/hello" to test)
#include <iostream>
#include <httpserver.hpp>
using namespace std;
using namespace httpserver;
class hello_world_resource : public http_resource {
public:
const std::shared_ptr<http_response> render(const http_request&) {
return std::shared_ptr<http_response>(new string_response("Hello, World!"));
}
};
int main(int argc, char** argv) {
cout << "hello!" << std::endl;
webserver web_server = create_webserver(8080);
hello_world_resource resource;
web_server.register_resource("/hello", &resource);
web_server.start(true);
return 0;
}

Related

How to get bazel genrule to access transitive dependencies?

I have the following in a BUILD file:
proto_library(
name = "proto_default_library",
srcs = glob(["*.proto"]),
visibility = ["//visibility:public"],
deps = [
"#go_googleapis//google/api:annotations_proto",
"#grpc_ecosystem_grpc_gateway//protoc-gen-openapiv2/options:options_proto",
],
)
genrule(
name = "generate-buf-image",
srcs = [
":buf_yaml",
":buf_breaking_image_json",
":protos",
],
exec_tools = [
":proto_default_library",
"//buf:generate-buf-image-sh",
"//buf:generate-buf-image",
],
outs = ["buf-image.json"],
cmd = "$(location //buf:generate-buf-image-sh) --buf-breaking-image-json=$(location :buf_breaking_image_json) $(location :protos) >$#",
)
While executing $(location //buf:generate-buf-image-sh), glob(["*.proto"]) of proto_default_library can be seen in the sandbox but the proto files of #go_googleapis//google/api:annotations_proto and #grpc_ecosystem_grpc_gateway//protoc-gen-openapiv2/options:options_proto cannot. The same goes for the dependencies of //buf:generate-buf-image-sh.
Do I need to explicitly list out all transitive dependencies so they can be processed by generate-buf-image? Is there a programmatic way to do that?
Since genrules are pretty generic, a genrule sees only the default provider of a target, which usually just has the main outputs of that target (e.g., for java_library, a jar of the classes of that library, for proto_library, the proto files of that library). So to get more detailed information, you would write a Starlark rule to access more specific providers. For example:
WORKSPACE:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_proto",
sha256 = "66bfdf8782796239d3875d37e7de19b1d94301e8972b3cbd2446b332429b4df1",
strip_prefix = "rules_proto-4.0.0",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_proto/archive/refs/tags/4.0.0.tar.gz",
"https://github.com/bazelbuild/rules_proto/archive/refs/tags/4.0.0.tar.gz",
],
)
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()
defs.bzl:
def _my_rule_impl(ctx):
output = ctx.actions.declare_file(ctx.attr.name + ".txt")
args = ctx.actions.args()
args.add(output)
inputs = []
for src in ctx.attr.srcs:
proto_files = src[ProtoInfo].transitive_sources
args.add_all(proto_files)
inputs.append(proto_files)
ctx.actions.run(
inputs = depset(transitive = inputs),
executable = ctx.attr._tool.files_to_run,
arguments = [args],
outputs = [output],
)
return DefaultInfo(files = depset([output]))
my_rule = rule(
implementation = _my_rule_impl,
attrs = {
"srcs": attr.label_list(providers=[ProtoInfo]),
"_tool": attr.label(default = "//:tool"),
},
)
ProtoInfo is here: https://bazel.build/rules/lib/ProtoInfo
BUILD:
load(":defs.bzl", "my_rule")
proto_library(
name = "proto_a",
srcs = ["proto_a.proto"],
deps = [":proto_b"],
)
proto_library(
name = "proto_b",
srcs = ["proto_b.proto"],
deps = [":proto_c"],
)
proto_library(
name = "proto_c",
srcs = ["proto_c.proto"],
)
my_rule(
name = "foo",
srcs = [":proto_a"],
)
sh_binary(
name = "tool",
srcs = ["tool.sh"],
)
proto_a.proto:
package my_protos_a;
message ProtoA {
optional int32 a = 1;
}
proto_b.proto:
package my_protos_b;
message ProtoB {
optional int32 b = 1;
}
proto_c.proto:
package my_protos_c;
message ProtoC {
optional int32 c = 1;
}
tool.sh:
output=$1
shift
echo input protos: $# > $output
$ bazel build foo
INFO: Analyzed target //:foo (40 packages loaded, 172 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
bazel-bin/foo.txt
INFO: Elapsed time: 0.832s, Critical Path: 0.02s
INFO: 5 processes: 4 internal, 1 linux-sandbox.
INFO: Build completed successfully, 5 total actions
$ cat bazel-bin/foo.txt
input protos: proto_a.proto proto_b.proto proto_c.proto

How to create a py_library from a generated source tree

I am trying to consume some Python code + C extensions that are produced by CMake.
I'm using rules_foreign_cc to build the code, and it puts the code into an output directory, but I'm stumped as to how I can turn that into a py_library. I tried the below:
Depend on the cmake rule as srcs
Set imports to try to point to the directory containing the Python packages
But it doesn't work. When I run the py_test, the constructed PYTHONPATH doesn't point to the directory containing the python package from the cmake rule.
Thanks for any help!
Here's what I've got in my third_party/BUILD.bazel:
load("#rules_foreign_cc//foreign_cc:defs.bzl", "cmake")
load("#rules_python//python:defs.bzl", "py_library", "py_test")
filegroup(
name = "torch_mlir_src",
srcs = glob(["torch-mlir/**"]),
)
make(
name = "torch_mlir",
...
install = False,
lib_source = ":torch_mlir_src",
out_data_dirs = ["python_packages"],
postfix_script = " && ".join([
"cp -r --dereference tools/torch-mlir/python_packages/torch_mlir $$INSTALLDIR$$/python_packages",
]),
targets = ["tools/torch-mlir/all"],
working_directory = "external/llvm-project/llvm",
)
# This doesn't seem to work.
py_library(
name = "torch_mlir_py",
srcs = [":torch_mlir"],
imports = ["$(BINDIR)/third_party/torch_mlir/python_packages"],
srcs_version = "PY3",
)
py_test(
name = "torch_mlir_annotations_sugar_test",
srcs = ["torch-mlir/python/test/annotations-sugar.py"],
main = "torch-mlir/python/test/annotations-sugar.py",
srcs_version = "PY3",
deps = [":torch_mlir_py"],
)

What Bazel incantation do I need to include gRPC reflection header file?

When I try to compile a bazel project that uses gRPC reflection, I get the following error.
fatal error: external/com_github_grpc_grpc/src/proto/grpc/reflection/v1alpha/reflection.grpc.pb.h: No such file or directory
In my WORKSPACE, I have the following bindings:
def _com_github_grpc_grpc():
external_http_archive("com_github_grpc_grpc")
external_http_archive("build_bazel_rules_apple")
# Rebind some stuff to match what the gRPC Bazel is expecting.
native.bind(
name = "protobuf_headers",
actual = "#com_google_protobuf//:protobuf_headers",
)
native.bind(
name = "libssl",
actual = "//external:ssl",
)
native.bind(
name = "cares",
actual = "//external:ares",
)
native.bind(
name = "grpc",
actual = "#com_github_grpc_grpc//:grpc++",
)
In my BUILD file, I have the following deps:
deps = [
"//external:protobuf_headers",
"//external:grpc",
],
What additional incantations do I need for the include at the top of the question?
After a spelunking through issues, I figured out that I need the following WORKSPACE
workspace(name = "xxx")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "com_github_grpc_grpc",
strip_prefix = "grpc-master",
urls = ["https://github.com/grpc/grpc/archive/master.tar.gz"],
)
load("#com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
grpc_deps()
load("#com_github_grpc_grpc//bazel:grpc_extra_deps.bzl", "grpc_extra_deps")
grpc_extra_deps()
and the following BUILD:
cc_binary(
name = "ReflectionPlay",
copts = ["-std=c++17"],
srcs = ["reflection_play.cc"],
deps = [
"#com_github_grpc_grpc//:grpc++",
"#com_github_grpc_grpc//:grpc++_reflection",
"#com_github_grpc_grpc//:grpcpp_admin",
],
)

How do I get my custom header template rule to pass it's output downstream cc_binary/cc_library dependency?

I'm trying to build a rule for bazel which emulates the CMake *.in template system.
This has two challenges, the first is generate the template output. The second is make the output available to both genrules, filegroups and cc_* rules. The third is to get that dependency to transitively be passed to further downstream rules.
I have it generating the output file version.hpp in genfiles (or bazel-bin), and I can get the initial library rule to include it, but I can't seem to figure out how to make my cc_binary rule, which depends on the cc_library and transitively the header_template rule to find the header file.
I have the following .bzl rule:
def _header_template_impl(ctx):
# this generates the output from the template
ctx.actions.expand_template(
template = ctx.file.template,
output = ctx.outputs.out,
substitutions = ctx.attr.vars,
)
return [
# create a provider which says that this
# out file should be made available as a header
CcInfo(compilation_context=cc_common.create_compilation_context(
headers=depset([ctx.outputs.out])
)),
# Also create a provider referencing this header ???
DefaultInfo(files=depset(
[ctx.outputs.out]
))
]
header_template = rule(
implementation = _header_template_impl,
attrs = {
"vars": attr.string_dict(
mandatory = True
),
"extension": attr.string(default=".hpp"),
"template": attr.label(
mandatory = True,
allow_single_file = True,
),
},
outputs = {
"out": "%{name}%{extension}",
},
output_to_genfiles = True,
)
elsewhere I have a cc_library rule:
load("//:tools/header_template.bzl", "header_template")
# version control
BONSAI_MAJOR_VERSION = '2'
BONSAI_MINOR_VERSION = '0'
BONSAI_PATCH_VERSION = '9'
BONSAI_VERSION = \
BONSAI_MAJOR_VERSION + '.' + \
BONSAI_MINOR_VERSION + '.' + \
BONSAI_PATCH_VERSION
header_template(
name = "bonsai_version",
extension = ".hpp",
template = "version.hpp.in",
vars = {
"#BONSAI_MAJOR_VERSION#": BONSAI_MAJOR_VERSION,
"#BONSAI_MINOR_VERSION#": BONSAI_MINOR_VERSION,
"#BONSAI_PATCH_VERSION#": BONSAI_PATCH_VERSION,
"#BONSAI_VERSION#": BONSAI_VERSION,
},
)
# ...
private = glob([
"src/**/*.hpp",
"src/**/*.cpp",
"proto/**/*.hpp",
])
public = glob([
"include/*.hpp",
":bonsai_version",
])
cc_library(
# target name matches directory name so you can call:
# bazel build .
name = "bonsai",
srcs = private,
hdrs = public,
# public headers
includes = [
"include",
],
# ...
deps = [
":bonsai_version",
# ...
],
# ...
)
When I build, my source files need to be able to:
#include "bonsai_version.hpp"
I think the answer involves CcInfo but I'm grasping in the dark as to how it should be constructed.
I've already tried add "-I$(GENDIR)/" + package_name() to the copts, to no avail. The generated header still isn't available.
My expectation is that I should be able to return some kind of Info object that would allow me to add the dependency in srcs. Maybe it should be a DefaultInfo.
I've dug through the bazel rules examples and the source, but I'm missing something fundamental, and I can't find documentation that discuss this particular.
I'd like to be able to do the following:
header_template(
name = "some_header",
extension = ".hpp",
template = "some_header.hpp.in",
vars = {
"#SOMEVAR#": "value",
"{ANOTHERVAR}": "another_value",
},
)
cc_library(
name = "foo",
srcs = ["foo.src", ":some_header"],
...
)
cc_binary(
name = "bar",
srcs = ["bar.cpp"],
deps = [":foo"],
)
and include the generated header like so:
#include "some_header.hpp"
void bar(){
}
The answer looks like it is:
def _header_template_impl(ctx):
# this generates the output from the template
ctx.actions.expand_template(
template = ctx.file.template,
output = ctx.outputs.out,
substitutions = ctx.attr.vars,
)
return [
# create a provider which says that this
# out file should be made available as a header
CcInfo(compilation_context=cc_common.create_compilation_context(
# pass out the include path for finding this header
includes=depset([ctx.outputs.out.dirname]),
# and the actual header here.
headers=depset([ctx.outputs.out])
))
]
elsewhere:
header_template(
name = "some_header",
extension = ".hpp",
template = "some_header.hpp.in",
vars = {
"#SOMEVAR#": "value",
"{ANOTHERVAR}": "another_value",
},
)
cc_library(
name = "foo",
srcs = ["foo.cpp"],
deps = [":some_header"],
...
)
cc_binary(
name = "bar",
srcs = ["bar.cpp"],
deps = [":foo"],
)
If your header has a generic name (eg config.h) and you want it to be private (ie srcs instead of hdrs), you might need a different approach. I've seen this problem for gflags, which "leaked" config.h and affected libraries that depended on it (issue).
Of course, in both cases, the easiest solution is to generate and commit header files for the platforms you target.
Alternatively, you can set copts for the cc_library rule that uses the generated private header:
cc_library(
name = "foo",
srcs = ["foo.cpp", "some_header.hpp"],
copts = ["-I$(GENDIR)/my/package/name"],
...
)
If you want this to work when your repository is included as an external repository, you have a bit more work cut out for you due to bazel issue #4463.
PS. You might want to see if cc_fix_config from https://github.com/antonovvk/bazel_rules works for you. It's just a wrapper around perl but I found it useful.

bazel build error in project containing pcl

My WORKSPACE file is below:
new_local_repository(
name = "opencv",
path = "/usr/local",
build_file = "opencv.BUILD",
)
new_local_repository(
name = "pcl",
path = "/usr/local",
build_file = "pcl.BUILD"
)
my pcl.BUILD file is below:
cc_library(
name = "pcl",
srcs = glob(["lib/libopencv_*.so*"]),
hdrs = glob(["include/pcl-1.7/pcl/**/*.hpp", "include/pcl-1.7/pcl/**/*.h"]),
includes = ["include"],
visibility = ["//visibility:public"],
linkstatic = 1,
)
my code is very sample, just like below:
#include <pcl/common/common_headers.h>
#include <pcl/visualization/pcl_visualizer.h>
int main() {
std::cout << "hello pcl" << std::endl;
return 0;
}
and i run bazel build command, using bazel query //... and bazel build target.
now I get the error like "fatal error: pcl/common/common_headers.h: No such file or directory". Why?
Someone can give suggestion for this error?
Normally in the #include you must use a path relative to the repository's root:
#include "include/pcl-1.7/pcl/common/common_headers.h"
In order to be able to include like so:
#include "pcl/common/common_headers.h"
...I believe you need to change cc_library.includes to this:
cc_library(
name = "pcl",
...
includes = ["include/pcl-1.7"],
...
)
Also, I believe you have to use quotes in #include, not angle brackets:
#include "pcl/common/common_headers.h"

Resources