I'm trying to compile my TS files to JS with SWC in Bazel. I'm unfortunately unable to use the rules_js library (which has rules_swc), so I need to hand roll it. So far I have:
load("#npm//#swc/cli:index.bzl", "swc")
SRC_FILES = glob(["**/*.ts"])
swc(
name = "compile_ts",
outs = [s.replace(".ts", ".js") for s in SRC_FILES],
args = [
"$(location %s)" % s
for s in SRC_FILES
],
data = SRC_FILES,
)
I get
Successfully compiled 96 files with swc.
but no output and the following error
ERROR: output {file} was not created
for each src file I passed in. How do I get the compiled files to output to the bazel-out folder so they can be used as inputs to other rules?
Related
I need the path to external (or internal) dependency to pass it as an argument to a function inside. We need the location to the folder, not specific files. Also, sometimes, we need the path to the folder where a shared library, generated by cc_library.
Python file
import cppyy
cppyy.add_include_path('path/to/external/dependency/1')
cppyy.add_library_path('path/to/another/external/dependency/2')
cppyy.add_include_path('path/to/another/internal/dependency')
cppyy.include('file/in/external/dependency')
BUILD file
py_binary(
name = "sample",
srcs = ["sample.py"],
deps = [
"#cppyy_archive//:cppyy",
],
data = [
"#external-dependency//location:target",
"//internal-dependency/location:target2"
]
)
From https://docs.bazel.build/versions/master/external.html#layout:
You can see the external directory by running:
ls $(bazel info output_base)/external
How the paths in external actually look like really depends on the rule used for the archive.
For example, if it's declared using an http_file in the WORKSPACE file:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
http_file(
name = "fenix",
urls = ["https://github.com/mozilla-mobile/fenix/archive/v76.0.0-beta.2.tar.gz"],
sha256 = "94050c664e5ec5b66cd2ca9f6a8b898987ab63d9602090533217df1a3f2dc5a9"
)
You will find that v76.0.0-beta.2.tar.gz file as external/fenix/file/downloaded:
user#host:~$ file $(bazel info output_base)/external/fenix/file/downloaded
/home/user/.cache/bazel/_bazel_user/761044447e04744e746cd54d0b4b5056/external/fenix/file/downloaded: gzip compressed data, from Unix, original size modulo 2^32 15759360
So I am trying to convert a monorepo of micro services (C#, Go, NodeJS) to use bazel. Just playing with it for now.
I focus on one go service to get started and isolated it as a WORKSPACE.
The go service is gRPC service that uses protobuf obviously, grpc-gateway with the protoc-gen-swagger and also protoc-gen-gorm (this one does not support bazel).
The code builds using a command like go build cmd/server/server.go
I am hoping to get some guidance on how to get started to build this project with all the dependencies.
I see several rules available for protobuf/go and I am not yet comfortable browsing through them or deciding which is better (i cannot get any to work due to grpc gateway or protoc gen gorm)
- https://github.com/stackb/rules_proto
- https://github.com/bazelbuild/rules_go
- https://github.com/stackb/rules_proto/tree/master/github.com/grpc-ecosystem/grpc-gateway
Code structure looks like this:
/repo
svc1
svc2
svc3
cmd/server
BUILD.bazel
server.go
pkg
contains folders and some go files and a BUILD.bazel in each
proto
BUILD.bazel
test.proto
WORKSPACE
BUILD.bazel
Right now I only work on svc3. Later i will probably move the WORKSPACE to the parent folder.
My WORKSPACE looks like this:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_bazel_rules_go",
sha256 = "96b1f81de5acc7658e1f5a86d7dc9e1b89bc935d83799b711363a748652c471a",
urls = [
"https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/rules_go/releases/download/0.19.2/rules_go-0.19.2.tar.gz",
"https://github.com/bazelbuild/rules_go/releases/download/0.19.2/rules_go-0.19.2.tar.gz",
],
)
load("#io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
go_rules_dependencies()
go_register_toolchains()
http_archive(
name = "bazel_gazelle",
urls = [
"https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/bazel-gazelle/releases/download/0.18.1/bazel-gazelle-0.18.1.tar.gz",
"https://github.com/bazelbuild/bazel-gazelle/releases/download/0.18.1/bazel-gazelle-0.18.1.tar.gz",
],
sha256 = "be9296bfd64882e3c08e3283c58fcb461fa6dd3c171764fcc4cf322f60615a9b",
)
load("#bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository")
gazelle_dependencies()
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
name = "com_google_protobuf",
commit = "09745575a923640154bcf307fba8aedff47f240a",
remote = "https://github.com/protocolbuffers/protobuf",
shallow_since = "1558721209 -0700",
)
load("#com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
+ a bunch of go_repository() created by Gazelle
Running gazelle created a bunch of build.bazel files for my go project in each folder.
Next to the .proto, I have a generated build.bazel file:
load("#io_bazel_rules_go//go:def.bzl", "go_library")
load("#io_bazel_rules_go//proto:def.bzl", "go_proto_library")
proto_library(
name = "svc_proto",
srcs = ["test.proto"],
visibility = ["//visibility:public"],
deps = [
# the two github below are referenced as go_repository
"#com_github_infobloxopen_protoc_gen_gorm//options:proto_library", # not sure what to put after the colon
"#com_github_grpc_ecosystem_grpc_gateway//protoc-gen-swagger/options:proto_library",
"#go_googleapis//google/api:annotations_proto",
],
)
go_proto_library(
name = "svc_go_proto",
compilers = ["#io_bazel_rules_go//proto:go_grpc"],
importpath = "src/test/proto/v1",
proto = ":svc_proto",
visibility = ["//visibility:public"],
deps = [
"//github.com/infobloxopen/protoc-gen-gorm/options:go_default_library",
"//github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library",
"#go_googleapis//google/api:annotations_go_proto",
],
)
go_library(
name = "go_default_library",
embed = [":svc_go_proto"],
importpath = "src/test/proto/v1",
visibility = ["//visibility:public"],
)
Now the questions:
not sure what to put to reference other proto files: "#com_github_infobloxopen_protoc_gen_gorm//options:proto_library" ? and not sure this is the best way to reference other external libraries from git.
if i build the above using bazel build //proto/v1:svc_proto, i get: no such target '#com_github_grpc_ecosystem_grpc_gateway//protoc-gen-swagger/options:proto_library': target 'proto_library' not declared in package 'protoc-gen-swagger/options'. Probably linked to 1.
i am not sure which rule to use. As I need grpc gateway, I guess i
need to exclusively use
https://github.com/stackb/rules_proto/tree/master/github.com/grpc-ecosystem/grpc-gateway
but i can't make them to work either.
I use statik (https://github.com/rakyll/statik) to package the
swagger file in go to server the swagger. Is there any alternative
or if not, how can i call a custom bash/command as part of the build
process in the chain?
In summary, I am pretty sure my BUILD.bazel file to build the proto and library is structured wrong and would appreciate some up-to-date guidance (github is full of repos that are outdated, using outdated rules or simply not working).
I have an external source repository that apparently stores some source files using ISO-8859-1 character encoding. I am having trouble getting javac to change from default UTF-8 to ISO-8859-1 when invoked through Bazel.
I'm fetching the external repository via Bazel and can determine the charset of the fetched files:
> cd bazel-PROJECT/external/third-party/src
> file -i LibraryCode.java
LibraryCode.java: text/x-c; charset=iso-8859-1
Building the external sources via Bazel's java_library, or attempting to compile external repository source files via javac directly from the command line fails with (expected):
error: unmappable character for encoding UTF8
Attempting to use javac's -encoding argument solves the compilation issue when used from the command line against the external repository files fetched by Bazel:
> javac -encoding iso-8859-1 LibraryCode.java
However, I've been unable to successfully pass the -encoding option to javac via Bazel.
I've tried so far:
setting the javacopts in java_library rule
setting --javacopt from Bazel's command line
declaring java_toolchain rule with encoding ISO-8859-1 and using it with --java_toolchain from Bazel's command line.
None of these attempts got around the charset mismatch and compiler error.
1) repository_rule build_file: thirdparty.BUILD
java_library(
name = "thirdparty",
srcs = glob(["src/**/*.java"]),
javacopts = ["-encoding iso-8859-1"],
visibility = ["//visibility:public"]
)
2) Bazel command line:
> bazel build --javacopt="-encoding iso-8859-1" target
3) Defining Java toolchain target with encoding setting:
java_toolchain(
name = "toolchain",
bootclasspath = ["#bazel_tools//tools/jdk:bootclasspath"],
encoding = "iso-8859-1",
extclasspath = ["#bazel_tools//tools/jdk:extdir"],
forcibly_disable_header_compilation = 0,
genclass = ["#bazel_tools//tools/jdk:GenClass_deploy.jar"],
header_compiler = ["#bazel_tools//tools/jdk:turbine_deploy.jar"],
ijar = ["#bazel_tools//tools/jdk:ijar"],
javabuilder = ["#bazel_tools//tools/jdk:JavaBuilder_deploy.jar"],
javac = ["#bazel_tools//third_party/java/jdk/langtools:javac_jar"],
javac_supports_workers = 1,
jvm_opts = [
"-XX:+TieredCompilation",
"-XX:TieredStopAtLevel=1",
],
misc = [
"-XDskipDuplicateBridges=true",
],
singlejar = ["#bazel_tools//tools/jdk:SingleJar_deploy.jar"],
source_version = "8",
target_version = "8",
visibility = ["//visibility:public"]
)
All end up with error: unmappable character for encoding UTF8.
What is the mistake I am making in setting the javac encoding via Bazel?
I can try to work around the issue by converting the external repository source files via iconv but I'd prefer solving it via javac's encoding setting as intended.
Follow Up
The java_toolchain encoding not getting recognized appears to be a bug. I've a preliminary fix for this on my local copy of Bazel -- java_toolchain approach to changing charset (option #3 above) appears to work.
Tracking this issue and a proposed fix in: #2926
Unfortunately there is no good way to do that from the command line / target based. You have to write a java_toolchain and point to it. Deriving the one from bazel that would result in:
java_toolchain(
name = "toolchain",
bootclasspath = ["#bazel_tools//tools/jdk:bootclasspath"],
encoding = "iso-8859-1",
extclasspath = ["#bazel_tools//tools/jdk:extclasspath"],
forcibly_disable_header_compilation = 0,
genclass = ["#bazel_tools//tools/jdk:genclass"],
header_compiler = ["#bazel_tools//tools/jdk:turbine"],
ijar = ["#bazel_tools//tools/jdk:ijar"],
javabuilder = ["#bazel_tools//tools/jdk:javabuilder"],
javac = ["#bazel_tools//third_party/java/jdk/langtools:javac_jar"],
javac_supports_workers = 1,
jvm_opts = [
"-XX:+TieredCompilation",
"-XX:TieredStopAtLevel=1",
],
misc = [
"-XDskipDuplicateBridges=true",
],
singlejar = ["#bazel_tools//tools/jdk:SingleJar_deploy.jar"],
source_version = "8",
target_version = "8",
)
(you might want to change the singlejar target to the C++ binary for performance reason: #bazel_tools//tools/jdk:singlejar IIRC)
Then you can point to that toolchain with --java_toolchain=//my:toolchain (see the java_toolchain flag)
How to read a data file at a package path on Lua 5.1?
What I'm looking for is something like a io.read but at the package directory instead of the working directory (arg[0]), and without using hardcoded absolute paths. That would be something like dofile does, but without running the code, only reading it as a string.
Example:
I have a test folder, the current working directory of the test.lua script.
There's a package luapackage in another folder, somewhere specified at the LUA_PATHenviroment variable.
luapackage can:
require("luapackage.other_module");
dofile("other_module.lua").
But luapackage can't do:
io.read("data.txt")
io.read("luapackage/data.txt")
Sample structure:
+-test/
|
+-test.lua
+-luamodule/
|
+-data.txt
|
+-luamodule.lua
|
+-other_module.lua
For this example, test.lua only requires luamodule:
-- test.lua
local luamodule = require("luamodule")
And luamodule needs to read its modules and data files:
-- luamodule.lua
local other_module= dofile("other_module.lua") -- works
-- local other_module= require("luamodule.other_module") -- also works
local data = io.open("data.txt") -- fails
-- local data = io.open("luamodule/data.txt") -- also fails
It doesn't work because it searches for the file at the working directory (test) and not the package directory.
If I place a copy of the package at running script's folder, io.read(luapackage/data.txt) is possible. But every script would have to carry its own local copy of luapackage.
Note: I'm looking for a Lua solution, avoiding binary packages that could compromise cross-compatibility.
You can use debug.getinfo(1,"S").source to get the location of the current (module) file. Replace luamodule.lua with data.txt, remove leading #, and you should get the path you need.
If I have a tarball, helloworld.tar.gz in a local directory, say /home/user/tarballs/, how can I make my bitbake recipe fetch from that directory?
my helloworld.bb is
SECTION = "examples"
LICENSE = "Proprietary"
LIC_FILES_CHKSUM = "file://COPYING; md5=1b1b8016e15e07a2fec59623ebf12345"
SRC_URI = "file://helloworld.tar.gz"
but when I bitbake, I get the below warning message:
WARNING: Unable to get checksum for helloworld SRC_URI entry helloworld.tar.gz: file could not be found
I read something about FILES and FILESEXTRAPATHS can influence the download path but not sure where/how to set them.
I did a bitbake -c show FILESEXTRAPATHS but get an error message:
ERROR: Nothing PROVIDES 'FILESEXTRAPATHS'
Well, if you want to fetch from a local directory, use e.g.:
SRC_URI = "file:///home/user/tarballs/helloworld.tar.gz"
The FILES and FILESEXTRAPATHS variables tells bitbake where to find files which are referenced as:
SRC_URI = "file://helloworld.tar.gz"
These files are searched for in the locations specified by those two variables. (Or rather, FILESEXTRAPATHS is searched and then some possible subdirectories of the directories specified in FILESEXTRAPATHS, amongst those the expanded values of DISTRO, MACHINE, ARCH, etc).
FILES (and FILESEXTRAPATHS) are used to find files stored together with the meta-data, i.e. under the paths meta-/recipes-/name/XXX.
See http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#var-FILES and http://www.yoctoproject.org/docs/1.7/mega-manual/mega-manual.html#var-FILESEXTRAPATHS
Better you can keep your files in present(where the .bb file present) directory and give the below lines in your .bb file.
FILESEXTRAPATHS_prepend := "${THISDIR}:"
SRC_URI = "file://helloworld.tar.gz"
FILESEXTRAPATHS_prepend : tells to bitbake the files are present in where the .bb file is present.
Also you can edit .bb file as follows,
FILESEXTRAPATHS_prepend := "path_to_home_folder_of_source_folders:"
SRC_URI = "file://Source_floder/*"
as an example
FILESEXTRAPATHS_prepend := "/home/username/:"
SRC_URI = "file://tarballs/*"