I am trying to consume some Python code + C extensions that are produced by CMake.
I'm using rules_foreign_cc to build the code, and it puts the code into an output directory, but I'm stumped as to how I can turn that into a py_library. I tried the below:
Depend on the cmake rule as srcs
Set imports to try to point to the directory containing the Python packages
But it doesn't work. When I run the py_test, the constructed PYTHONPATH doesn't point to the directory containing the python package from the cmake rule.
Thanks for any help!
Here's what I've got in my third_party/BUILD.bazel:
load("#rules_foreign_cc//foreign_cc:defs.bzl", "cmake")
load("#rules_python//python:defs.bzl", "py_library", "py_test")
filegroup(
name = "torch_mlir_src",
srcs = glob(["torch-mlir/**"]),
)
make(
name = "torch_mlir",
...
install = False,
lib_source = ":torch_mlir_src",
out_data_dirs = ["python_packages"],
postfix_script = " && ".join([
"cp -r --dereference tools/torch-mlir/python_packages/torch_mlir $$INSTALLDIR$$/python_packages",
]),
targets = ["tools/torch-mlir/all"],
working_directory = "external/llvm-project/llvm",
)
# This doesn't seem to work.
py_library(
name = "torch_mlir_py",
srcs = [":torch_mlir"],
imports = ["$(BINDIR)/third_party/torch_mlir/python_packages"],
srcs_version = "PY3",
)
py_test(
name = "torch_mlir_annotations_sugar_test",
srcs = ["torch-mlir/python/test/annotations-sugar.py"],
main = "torch-mlir/python/test/annotations-sugar.py",
srcs_version = "PY3",
deps = [":torch_mlir_py"],
)
Related
When I try to compile a bazel project that uses gRPC reflection, I get the following error.
fatal error: external/com_github_grpc_grpc/src/proto/grpc/reflection/v1alpha/reflection.grpc.pb.h: No such file or directory
In my WORKSPACE, I have the following bindings:
def _com_github_grpc_grpc():
external_http_archive("com_github_grpc_grpc")
external_http_archive("build_bazel_rules_apple")
# Rebind some stuff to match what the gRPC Bazel is expecting.
native.bind(
name = "protobuf_headers",
actual = "#com_google_protobuf//:protobuf_headers",
)
native.bind(
name = "libssl",
actual = "//external:ssl",
)
native.bind(
name = "cares",
actual = "//external:ares",
)
native.bind(
name = "grpc",
actual = "#com_github_grpc_grpc//:grpc++",
)
In my BUILD file, I have the following deps:
deps = [
"//external:protobuf_headers",
"//external:grpc",
],
What additional incantations do I need for the include at the top of the question?
After a spelunking through issues, I figured out that I need the following WORKSPACE
workspace(name = "xxx")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "com_github_grpc_grpc",
strip_prefix = "grpc-master",
urls = ["https://github.com/grpc/grpc/archive/master.tar.gz"],
)
load("#com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
grpc_deps()
load("#com_github_grpc_grpc//bazel:grpc_extra_deps.bzl", "grpc_extra_deps")
grpc_extra_deps()
and the following BUILD:
cc_binary(
name = "ReflectionPlay",
copts = ["-std=c++17"],
srcs = ["reflection_play.cc"],
deps = [
"#com_github_grpc_grpc//:grpc++",
"#com_github_grpc_grpc//:grpc++_reflection",
"#com_github_grpc_grpc//:grpcpp_admin",
],
)
I have some html files in "root/html_files/*.html" directory. I want to iterate on these html files and run some bazel rules on them, from "root/tests/" directory.
So, I made a filegroup of those html files and tried to get access to them in "root/tests/" directory, but that is't working.
I wonder if it is possible?
My BUILD file in "root/" directory:
HTMLS_LIST = glob(["html_files/*.html",])
filegroup(
name = "html_files",
srcs = HTMLS_LIST,
visibility = [ "//visibility:public" ],)
My BUILD file in "root/tests/" directory:
load("//tests:automation_test.bzl", "make_and_run_html_tests")
make_and_run_html_tests(
name = 'test_all_htmls',
srcs = ['test/automation_test.py'],
html_files = '//:html_files')
My bzl file in "root/tests/" directory:
def make_and_run_html_tests(name, html_files, srcs):
tests = []
for html_file in html_files: # I want to iterate on sources of html filegroup here
folders = html_file.split("/")
filename = folders[-1].split(".")[0]
test_rule_name = 'test_' + filename + '_file'
native.py_test(
name = test_rule_name,
srcs = srcs,
main = srcs[0],
data = [
html_file,
],
args = [html_file],
)
testname = ":" + test_rule_name
tests.append(testname)
native.test_suite(
name = name,
tests = tests,
)
And my python unittest file in "root/tests/" directory:
import sys
import codecs
import unittest
class TestHtmlDocumetns(unittest.TestCase):
def setUp(self):
self.html_file_path = sys.argv[1]
def test_html_file(self):
fid = codecs.open(self.html_file_path, 'r')
print(fid.read())
self.assertTrue(fid)
if __name__ == '__main__':
unittest.main(argv=[sys.argv[1]])
You can't access the references to the files inside another filegroup / rule from within a macro like that. You'd need to create a rule and access them via the ctx.files attr
However, you can iterate over them if you were to remove the filegroup, and pass the glob directly to the macro:
HTMLS_LIST = glob(["html_files/*.html"])
make_and_run_html_tests(
name = 'test_all_htmls',
srcs = ['test/automation_test.py'],
html_files = HTMLS_LIST
)
The glob is resolved to an array before expanding the macro
I'm experimenting with using libhttpserver and, by extension, libmicrohttpd from inside a Bazel build. The build process for these libraries seems to go:
./bootstrap
mkdir build
cd build
../configure
make
which is a slight variation, that I haven't seen before, on the more classic configure && make workflow.
Has anyone managed to make such a library work under Bazel?
Does anyone have a public example I can crib from?
The closest thing I've found to supporting this is #rules_foreign_cc//tools/build_defs:configure.bzl#configure_make but that seems to have no concept of the bootstrap step. Even hacking it doesn't seem to work as the bootstrap script ends up failing with:
mkdir: cannot create directory 'tmpwrk23': Read-only file system
autopoint: *** cannot create directory tmpwrk23
autopoint: *** Stop.
autoreconf: autopoint failed with exit status: 1
I'm about ready to just reach for a genrule() but that seems very error prone...
I went down the same path as you and was able to get libhttpserver compiling with bazel using the rules_foreign_cc project after coming across the 6/23/20 update on this blog post. I added code below, but in general rules_foreign_cc has a make rule, which you can set override make_commands and call ./bootstrap.
WORKSPACE:
...
http_archive(
name = "rules_foreign_cc",
strip_prefix = "rules_foreign_cc-master",
url = "https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip",
)
load("#rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies")
rules_foreign_cc_dependencies(register_default_tools = True)
all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""
http_archive(
name = "rules_cc",
urls = ["https://github.com/bazelbuild/rules_cc/archive/262ebec3c2296296526740db4aefce68c80de7fa.zip"],
strip_prefix = "rules_cc-262ebec3c2296296526740db4aefce68c80de7fa",
)
http_archive(
name = "libgnutls",
build_file_content = all_content,
strip_prefix = "gnutls-3.6.15",
urls = ["https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.15.tar.xz"],
)
http_archive(
name = "libhttpserver",
build_file_content = all_content,
strip_prefix = "libhttpserver-master",
urls = ["https://github.com/etr/libhttpserver/archive/master.zip"],
)
http_archive(
name = "libmicrohttpd",
build_file_content = all_content,
strip_prefix = "libmicrohttpd-0.9.71",
urls = ["https://ftp.gnu.org/gnu/libmicrohttpd/libmicrohttpd-0.9.71.tar.gz"],
)
BUILD:
load("#rules_foreign_cc//tools/build_defs:configure.bzl", "configure_make")
load("#rules_foreign_cc//tools/build_defs:make.bzl", "make")
load("#rules_cc//cc:defs.bzl", "cc_binary", "cc_library")
package(default_visibility = ["//visibility:public"])
configure_make(
name = "libgnutls",
lib_source = "#libgnutls//:all",
configure_options = ["--with-included-unistring"],
out_include_dir = "include/gnutls",
shared_libraries = ["libgnutls.so"],
)
configure_make(
name = "libmicrohttpd",
lib_source = "#libmicrohttpd//:all",
deps = [":libgnutls"],
)
make(
name = "libhttpserver",
lib_source = "#libhttpserver//:all",
make_commands = [
"./bootstrap",
"mkdir build_dir",
"cd build_dir",
"../configure --prefix=${INSTALLDIR}",
"make",
"make install",
],
deps = [":libmicrohttpd", ":libgnutls"],
)
cc_binary(
name = "hello-world",
srcs = ["hello_world.cc"],
deps = [
":libhttpserver"
],
)
hello_world.cc (example on libhttpserver github page, run "curl -XGET -v http://localhost:8080/hello" to test)
#include <iostream>
#include <httpserver.hpp>
using namespace std;
using namespace httpserver;
class hello_world_resource : public http_resource {
public:
const std::shared_ptr<http_response> render(const http_request&) {
return std::shared_ptr<http_response>(new string_response("Hello, World!"));
}
};
int main(int argc, char** argv) {
cout << "hello!" << std::endl;
webserver web_server = create_webserver(8080);
hello_world_resource resource;
web_server.register_resource("/hello", &resource);
web_server.start(true);
return 0;
}
In my Java-based Bazel project I use a code generator written in Java. The generator is part of the root project, and used in sub-projects as well.
What I want to achieve is to include the output of the root project (a .jar file) as a dependency for the code generation in sub-projects to grant the code generator access to all compiled files of the root project (through the classpath). Is that possible in Bazel?
What I see is that the classpath for key generation in the child project only includes the dependencies of the code generator binary (//parent:SettingsGenerator in the script below).
In my custom rule I invoke the code generator basically like this:
def _generate_settings(ctx):
...
ctx.actions.run(
inputs = [ctx.file._src] + [ctx.files.deps],
outputs = [keys, settings, loader],
mnemonic = "GenKeysAndSettings",
arguments = [args],
executable = ctx.executable._tool,
)
return [DefaultInfo(
files=depset([keys, settings, loader]),
runfiles=ctx.runfiles(files=ctx.files.deps)
)]
gen_settings = rule(
implementation = _generate_settings,
attrs = {
"lang": attr.string(),
"deps": attr.label_list(
allow_files = True
),
"_tool": attr.label(
cfg = "host",
executable = True,
default = Label("//parent:SettingsGenerator"),
),
"_src": attr.label(
single_file = True,
default = Label("//parent:Name")
),
}
)
The parent project BUILD:
load("//parent:settings.bzl", "gen_settings")
gen_settings(
name = "GenerateSettings",
lang = ""
)
java_library(
name = "parent",
srcs = glob(["src/main/java/**/*.java"]) + [
":GenerateSettings",
],
...
)
java_binary(
name = "SettingsGenerator",
srcs = glob(["src/main/java/**/SettingsGenerator.java"]),
main_class = "my.company.SettingsGenerator",
...
)
The child project BUILD:
gen_settings(
name = "GenerateSettings",
lang = "Java",
deps = ["//parent"]
)
...
My workaround is to include the .jar file as input and use a custom classloader in the generator. But it would be nice if I could control the classpath directly from Bazel.
Any help would be appreciated. Thank you.
What I figured currently is creating a AllTest and run it with junit. But, I am not satisfied with it. I want this rule can create as many tests as many java test file in created in codebase.
def junit_suite_test(name, srcs, deps, size="small", resources=[], classpath_resources=[], jvm_flags=[], tags=[], data=[]):
tests = []
package = PACKAGE_NAME.replace("src/test/java/", "").replace("/", ".")
for src in srcs:
if src.endswith("Test.java"):
if "/" in src:
src = package + "." + src.replace("/", ".")
tests += [src.replace(".java", ".class")]
native.genrule(
name = name + "-AllTests-gen",
outs = ["AllTests.java"],
cmd = """
cat <<EOF >> $#
package %s;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({%s})
public class AllTests {}
EOF
""" % (package, ",".join(tests))
)
native.java_test(
name = name,
srcs = srcs + ["AllTests.java"],
test_class = package + ".AllTests",
resources = resources,
classpath_resources = classpath_resources,
data = data,
size = size,
tags = tags,
jvm_flags = jvm_flags,
deps = deps + [
],
)
Hi you can do something like that:
[java_test(name = s[:-5], srcs = s) for s in glob(["*.java"])]
That will create on test target per java file.
With that method, your macro would looks like:
def junit_suite_test(name, srcs, deps, size="small", resources=[], classpath_resources=[], jvm_flags=[], tags=[], data=[]):
[native.java_test(
name = name,
srcs = src,
resources = resources,
classpath_resources = classpath_resources,
data = data,
size = size,
tags = tags,
jvm_flags = jvm_flags,
deps = deps,
) for src in srcs if src.endswith("Test.java")]
Of course you probably needs some adaptation to feed in the good sources.
However, I would recommend against doing that over your solution as too much parallelisms can actually be slower in fine. The test log and the XML file will report the actual failing test case and you can use shard_count to increase parallelism is really needed.