I have an external source repository that apparently stores some source files using ISO-8859-1 character encoding. I am having trouble getting javac to change from default UTF-8 to ISO-8859-1 when invoked through Bazel.
I'm fetching the external repository via Bazel and can determine the charset of the fetched files:
> cd bazel-PROJECT/external/third-party/src
> file -i LibraryCode.java
LibraryCode.java: text/x-c; charset=iso-8859-1
Building the external sources via Bazel's java_library, or attempting to compile external repository source files via javac directly from the command line fails with (expected):
error: unmappable character for encoding UTF8
Attempting to use javac's -encoding argument solves the compilation issue when used from the command line against the external repository files fetched by Bazel:
> javac -encoding iso-8859-1 LibraryCode.java
However, I've been unable to successfully pass the -encoding option to javac via Bazel.
I've tried so far:
setting the javacopts in java_library rule
setting --javacopt from Bazel's command line
declaring java_toolchain rule with encoding ISO-8859-1 and using it with --java_toolchain from Bazel's command line.
None of these attempts got around the charset mismatch and compiler error.
1) repository_rule build_file: thirdparty.BUILD
java_library(
name = "thirdparty",
srcs = glob(["src/**/*.java"]),
javacopts = ["-encoding iso-8859-1"],
visibility = ["//visibility:public"]
)
2) Bazel command line:
> bazel build --javacopt="-encoding iso-8859-1" target
3) Defining Java toolchain target with encoding setting:
java_toolchain(
name = "toolchain",
bootclasspath = ["#bazel_tools//tools/jdk:bootclasspath"],
encoding = "iso-8859-1",
extclasspath = ["#bazel_tools//tools/jdk:extdir"],
forcibly_disable_header_compilation = 0,
genclass = ["#bazel_tools//tools/jdk:GenClass_deploy.jar"],
header_compiler = ["#bazel_tools//tools/jdk:turbine_deploy.jar"],
ijar = ["#bazel_tools//tools/jdk:ijar"],
javabuilder = ["#bazel_tools//tools/jdk:JavaBuilder_deploy.jar"],
javac = ["#bazel_tools//third_party/java/jdk/langtools:javac_jar"],
javac_supports_workers = 1,
jvm_opts = [
"-XX:+TieredCompilation",
"-XX:TieredStopAtLevel=1",
],
misc = [
"-XDskipDuplicateBridges=true",
],
singlejar = ["#bazel_tools//tools/jdk:SingleJar_deploy.jar"],
source_version = "8",
target_version = "8",
visibility = ["//visibility:public"]
)
All end up with error: unmappable character for encoding UTF8.
What is the mistake I am making in setting the javac encoding via Bazel?
I can try to work around the issue by converting the external repository source files via iconv but I'd prefer solving it via javac's encoding setting as intended.
Follow Up
The java_toolchain encoding not getting recognized appears to be a bug. I've a preliminary fix for this on my local copy of Bazel -- java_toolchain approach to changing charset (option #3 above) appears to work.
Tracking this issue and a proposed fix in: #2926
Unfortunately there is no good way to do that from the command line / target based. You have to write a java_toolchain and point to it. Deriving the one from bazel that would result in:
java_toolchain(
name = "toolchain",
bootclasspath = ["#bazel_tools//tools/jdk:bootclasspath"],
encoding = "iso-8859-1",
extclasspath = ["#bazel_tools//tools/jdk:extclasspath"],
forcibly_disable_header_compilation = 0,
genclass = ["#bazel_tools//tools/jdk:genclass"],
header_compiler = ["#bazel_tools//tools/jdk:turbine"],
ijar = ["#bazel_tools//tools/jdk:ijar"],
javabuilder = ["#bazel_tools//tools/jdk:javabuilder"],
javac = ["#bazel_tools//third_party/java/jdk/langtools:javac_jar"],
javac_supports_workers = 1,
jvm_opts = [
"-XX:+TieredCompilation",
"-XX:TieredStopAtLevel=1",
],
misc = [
"-XDskipDuplicateBridges=true",
],
singlejar = ["#bazel_tools//tools/jdk:SingleJar_deploy.jar"],
source_version = "8",
target_version = "8",
)
(you might want to change the singlejar target to the C++ binary for performance reason: #bazel_tools//tools/jdk:singlejar IIRC)
Then you can point to that toolchain with --java_toolchain=//my:toolchain (see the java_toolchain flag)
Related
I am trying to set up a simple Bazel workspace to build a pybind library and I am unable to get Bazel to use a specific non-system python (pulled using the rules_python Bazel pakage).
This is my setup:
.
├── BUILD
├── WORKSPACE
└── example.cpp
BUILD:
load("#pybind11_bazel//:build_defs.bzl", "pybind_extension")
pybind_extension(
name = "example",
srcs = ["example.cpp"],
deps = [ ],
)
WORKSPACE:
oad("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_python",
sha256 = "8c8fe44ef0a9afc256d1e75ad5f448bb59b81aba149b8958f02f7b3a98f5d9b4",
strip_prefix = "rules_python-0.13.0",
url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.13.0.tar.gz",
)
load("#rules_python//python:repositories.bzl", "python_register_toolchains")
python_register_toolchains(
name = "python3_9",
python_version = "3.9",
)
load("#python3_9//:defs.bzl", "interpreter")
load("#rules_python//python:pip.bzl", "pip_parse")
pip_parse(
python_interpreter_target = interpreter,
)
http_archive(
name = "pybind11_bazel",
strip_prefix = "pybind11_bazel-9a24c33cbdc510fa60ab7f5ffb7d80ab89272799",
urls = ["https://github.com/pybind/pybind11_bazel/archive/9a24c33cbdc510fa60ab7f5ffb7d80ab89272799.zip"],
)
http_archive(
name = "pybind11",
build_file = "#pybind11_bazel//:pybind11.BUILD",
strip_prefix = "pybind11-2.10.0",
urls = ["https://github.com/pybind/pybind11/archive/refs/tags/v2.10.0.zip"],
)
load("#pybind11_bazel//:python_configure.bzl", "python_configure")
python_configure(
name = "local_config_python",
python_interpreter_target = "**what goes here**",
)
example.cpp:
#include <pybind11/pybind11.h>
namespace py = pybind11;
void do_something() { }
PYBIND11_MODULE(example, m) {
m.def("do_something", &do_something, "A function that does something");
}
If I remove the python_interpreter_target = "**what goes here**" line from the WORKSPACE everything works, but it is using the system python which I am trying to avoid. But I can't find where/what bazel target the python3 binary is from the rules_bazel pakage pull.
In case it helps, these are all the targets I can see for #python3_9:
% bazel query #python3_9//...
#python3_9//:files
#python3_9//:includes
#python3_9//:libpython
#python3_9//:pip
#python3_9//:py3_runtime
#python3_9//:python3
#python3_9//:python_headers
#python3_9//:python_runtimes
Loading: 0 packages loaded
(None of these seem to be what python_interpreter_target wants)
Please try passing the actual path to your target python executable. Kindly check the python executable into the third_party dir and instead of invoking your rules_python using simply python pass the relative path to that executable relative to WORKSPACE. Thank you!
You're almost there, just need to pass in the interpreter:
load("#python3_9//:defs.bzl", "interpreter")
...
python_configure(
name = "local_config_python",
python_interpreter_target = interpreter,
)
I'm trying to compile my TS files to JS with SWC in Bazel. I'm unfortunately unable to use the rules_js library (which has rules_swc), so I need to hand roll it. So far I have:
load("#npm//#swc/cli:index.bzl", "swc")
SRC_FILES = glob(["**/*.ts"])
swc(
name = "compile_ts",
outs = [s.replace(".ts", ".js") for s in SRC_FILES],
args = [
"$(location %s)" % s
for s in SRC_FILES
],
data = SRC_FILES,
)
I get
Successfully compiled 96 files with swc.
but no output and the following error
ERROR: output {file} was not created
for each src file I passed in. How do I get the compiled files to output to the bazel-out folder so they can be used as inputs to other rules?
I need the path to external (or internal) dependency to pass it as an argument to a function inside. We need the location to the folder, not specific files. Also, sometimes, we need the path to the folder where a shared library, generated by cc_library.
Python file
import cppyy
cppyy.add_include_path('path/to/external/dependency/1')
cppyy.add_library_path('path/to/another/external/dependency/2')
cppyy.add_include_path('path/to/another/internal/dependency')
cppyy.include('file/in/external/dependency')
BUILD file
py_binary(
name = "sample",
srcs = ["sample.py"],
deps = [
"#cppyy_archive//:cppyy",
],
data = [
"#external-dependency//location:target",
"//internal-dependency/location:target2"
]
)
From https://docs.bazel.build/versions/master/external.html#layout:
You can see the external directory by running:
ls $(bazel info output_base)/external
How the paths in external actually look like really depends on the rule used for the archive.
For example, if it's declared using an http_file in the WORKSPACE file:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
http_file(
name = "fenix",
urls = ["https://github.com/mozilla-mobile/fenix/archive/v76.0.0-beta.2.tar.gz"],
sha256 = "94050c664e5ec5b66cd2ca9f6a8b898987ab63d9602090533217df1a3f2dc5a9"
)
You will find that v76.0.0-beta.2.tar.gz file as external/fenix/file/downloaded:
user#host:~$ file $(bazel info output_base)/external/fenix/file/downloaded
/home/user/.cache/bazel/_bazel_user/761044447e04744e746cd54d0b4b5056/external/fenix/file/downloaded: gzip compressed data, from Unix, original size modulo 2^32 15759360
So I am trying to convert a monorepo of micro services (C#, Go, NodeJS) to use bazel. Just playing with it for now.
I focus on one go service to get started and isolated it as a WORKSPACE.
The go service is gRPC service that uses protobuf obviously, grpc-gateway with the protoc-gen-swagger and also protoc-gen-gorm (this one does not support bazel).
The code builds using a command like go build cmd/server/server.go
I am hoping to get some guidance on how to get started to build this project with all the dependencies.
I see several rules available for protobuf/go and I am not yet comfortable browsing through them or deciding which is better (i cannot get any to work due to grpc gateway or protoc gen gorm)
- https://github.com/stackb/rules_proto
- https://github.com/bazelbuild/rules_go
- https://github.com/stackb/rules_proto/tree/master/github.com/grpc-ecosystem/grpc-gateway
Code structure looks like this:
/repo
svc1
svc2
svc3
cmd/server
BUILD.bazel
server.go
pkg
contains folders and some go files and a BUILD.bazel in each
proto
BUILD.bazel
test.proto
WORKSPACE
BUILD.bazel
Right now I only work on svc3. Later i will probably move the WORKSPACE to the parent folder.
My WORKSPACE looks like this:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_bazel_rules_go",
sha256 = "96b1f81de5acc7658e1f5a86d7dc9e1b89bc935d83799b711363a748652c471a",
urls = [
"https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/rules_go/releases/download/0.19.2/rules_go-0.19.2.tar.gz",
"https://github.com/bazelbuild/rules_go/releases/download/0.19.2/rules_go-0.19.2.tar.gz",
],
)
load("#io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
go_rules_dependencies()
go_register_toolchains()
http_archive(
name = "bazel_gazelle",
urls = [
"https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/bazel-gazelle/releases/download/0.18.1/bazel-gazelle-0.18.1.tar.gz",
"https://github.com/bazelbuild/bazel-gazelle/releases/download/0.18.1/bazel-gazelle-0.18.1.tar.gz",
],
sha256 = "be9296bfd64882e3c08e3283c58fcb461fa6dd3c171764fcc4cf322f60615a9b",
)
load("#bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository")
gazelle_dependencies()
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
name = "com_google_protobuf",
commit = "09745575a923640154bcf307fba8aedff47f240a",
remote = "https://github.com/protocolbuffers/protobuf",
shallow_since = "1558721209 -0700",
)
load("#com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
+ a bunch of go_repository() created by Gazelle
Running gazelle created a bunch of build.bazel files for my go project in each folder.
Next to the .proto, I have a generated build.bazel file:
load("#io_bazel_rules_go//go:def.bzl", "go_library")
load("#io_bazel_rules_go//proto:def.bzl", "go_proto_library")
proto_library(
name = "svc_proto",
srcs = ["test.proto"],
visibility = ["//visibility:public"],
deps = [
# the two github below are referenced as go_repository
"#com_github_infobloxopen_protoc_gen_gorm//options:proto_library", # not sure what to put after the colon
"#com_github_grpc_ecosystem_grpc_gateway//protoc-gen-swagger/options:proto_library",
"#go_googleapis//google/api:annotations_proto",
],
)
go_proto_library(
name = "svc_go_proto",
compilers = ["#io_bazel_rules_go//proto:go_grpc"],
importpath = "src/test/proto/v1",
proto = ":svc_proto",
visibility = ["//visibility:public"],
deps = [
"//github.com/infobloxopen/protoc-gen-gorm/options:go_default_library",
"//github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library",
"#go_googleapis//google/api:annotations_go_proto",
],
)
go_library(
name = "go_default_library",
embed = [":svc_go_proto"],
importpath = "src/test/proto/v1",
visibility = ["//visibility:public"],
)
Now the questions:
not sure what to put to reference other proto files: "#com_github_infobloxopen_protoc_gen_gorm//options:proto_library" ? and not sure this is the best way to reference other external libraries from git.
if i build the above using bazel build //proto/v1:svc_proto, i get: no such target '#com_github_grpc_ecosystem_grpc_gateway//protoc-gen-swagger/options:proto_library': target 'proto_library' not declared in package 'protoc-gen-swagger/options'. Probably linked to 1.
i am not sure which rule to use. As I need grpc gateway, I guess i
need to exclusively use
https://github.com/stackb/rules_proto/tree/master/github.com/grpc-ecosystem/grpc-gateway
but i can't make them to work either.
I use statik (https://github.com/rakyll/statik) to package the
swagger file in go to server the swagger. Is there any alternative
or if not, how can i call a custom bash/command as part of the build
process in the chain?
In summary, I am pretty sure my BUILD.bazel file to build the proto and library is structured wrong and would appreciate some up-to-date guidance (github is full of repos that are outdated, using outdated rules or simply not working).
Is there any way to tell HHVM to output Hacklang warnings and errors into the browser? Something like PHP does with enabled display_errors, display_startup_errors and error_reporting set to E_ALL
HHVM version:
$ php -v
HipHop VM 3.1.0-dev+2014.04.09 (rel)
Compiler: heads/master-0-g4fc811c64c23a3686f66a2bea80ba47f3eaf9f3d
Repo schema: 79197c935790c0b9c9cb13566c3e727ace368117
I've tried the following config:
$ cat /etc/hhvm/php.ini
; php options
display_startup_errors = On
error_reporting = E_ALL
display_errors = On
; hhvm specific
hhvm.log.level = Warning
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.runtime_error_reporting_level = 8191
hhvm.mysql.typed_results = false
And :
$ cat /etc/hhvm/server.ini
; php options
pid = /var/run/hhvm/pid
; hhvm specific
hhvm.server.port = 9000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.level = Warning
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.runtime_error_reporting_level = 8191
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.mysql.typed_results = false
hhvm.debug.full_backtrace = true
hhvm.debug.server_stack_trace = true
hhvm.debug.server_error_message = true
hhvm.debug.translate_source = true
tl;dr: You can't.
The thing to keep in mind here is that the typechecker does a static analysis of your code while the PHP errors you talk about show up at runtime. If this was C++, you could compare the Hack typechecker errors with the errors during the compile step - so Hack tells you things that are wrong before the code even runs.
The trick is to use either the vim or emacs plugins which warn you of errors as you save the file, or use hh_client from the terminal, or build a plugin for your favorite IDE (feel free to send pull requests!). hh_client --json gives an easy to parse output if you want to build a plugin for Sublime Text, or Eclipse or whatever you want.
Note that some errors are runtime errors, while some aren't. Function args as well as return types should throw exceptions at runtime for the latest HHVM build for example. The problem there is that you only see those errors when you hit a certain code-path. The beauty of Hack is that it errors for all the problems in your code, even if it's a code-path you may not test at runtime.