I'm using some macro which generates *.kt files in bazel-bin folder.
My plan was to encapsulate those kt files in a kt_jvm_library.
I am trying this but it's not working:
kt_jvm_library(
name = "generated-stuff",
srcs = ["bazel-bin/src/main/java/com/example/Hello.kt"],
deps = [
...
],
)
In Gradle I could just do:
sourceSets {
main {
java {
srcDir ('${buildDir.absolutePath}/generated/...')
}
resources {
srcDir ('config')
}
}
}
Trying to find an equivalent in Bazel.
You need to use a label to refer to outputs from other rules, so that Bazel understands where they're coming from to set up the correct dependencies. Depending on your BUILD file layout, that would be something like //src/main/java/com/example:Kello.kt or //src/main/java:com/example/Hello.kt. If it's in the same BUILD file, then just :Hello.kt would work too. It depends on where the package boundary is (basically where the deepest-nested BUILD file is).
Also, depending on the rule, you may not be able to refer to Hello.kt directly, you may have to use the label of the rule instead. Using the label of a rule in srcs will typically use its default outputs.
Related
Bazel has been working great for me recently, but I've stumbled upon a question for which I have yet to find a satisfactory answer:
How can one collect all files bearing a certain extension from the workspace?
Another way of phrasing the question: how could one obtain the functional equivalent of doing a glob() across a complete Bazel workspace?
Background
The goal in this particular case is to collect all markdown files to run some checks and generate a static site from them.
At first glance, glob() sounds like a good idea, but will stop as soon as it runs into a BUILD file.
Current Approaches
The current approach is to run the collection/generation logic outside of the sandbox, but this is a bit dirty, and I'm wondering if there is a way that is both "proper" and easy (ie, not requiring that each BUILD file explicitly exposes its markdown files.
Is there any way to specify, in the workspace, some default rules that will be added to all BUILD files?
You could write an aspect for this to aggregate markdown files in a bottom-up manner and create actions on those files. There is an example of a file_collector aspect here. I modified the aspect's extensions for your use case. This aspect aggregates all .md and .markdown files across targets on the deps attribute edges.
FileCollector = provider(
fields = {"files": "collected files"},
)
def _file_collector_aspect_impl(target, ctx):
# This function is executed for each dependency the aspect visits.
# Collect files from the srcs
direct = [
f
for f in ctx.rule.files.srcs
if ctx.attr.extension == f.extension
]
# Combine direct files with the files from the dependencies.
files = depset(
direct = direct,
transitive = [dep[FileCollector].files for dep in ctx.rule.attr.deps],
)
return [FileCollector(files = files)]
markdown_file_collector_aspect = aspect(
implementation = _file_collector_aspect_impl,
attr_aspects = ["deps"],
attrs = {
"extension": attr.string(values = ["md", "markdown"]),
},
)
Another way is to do a query on file targets (input and output files known to the Bazel action graph), and process these files separately. Here's an example querying for .bzl files in the rules_jvm_external repo:
$ bazel query //...:* | grep -e ".bzl$"
//migration:maven_jar_migrator_deps.bzl
//third_party/bazel_json/lib:json_parser.bzl
//settings:stamp_manifest.bzl
//private/rules:jvm_import.bzl
//private/rules:jetifier_maven_map.bzl
//private/rules:jetifier.bzl
//:specs.bzl
//:private/versions.bzl
//:private/proxy.bzl
//:private/dependency_tree_parser.bzl
//:private/coursier_utilities.bzl
//:coursier.bzl
//:defs.bzl
I've got a Java project I'm converting to Bazel.
As is typical with Java projects, there are property files with placeholders that need to be resolved/substituted at build time.
Some of the values can be hardcoded in a BUILD or BZL file:
BUILD_PROPERTIES = { "pom.version": "1.0.0", "pom.group.id": "com.mygroup"}
Some of the variables are "stamps" (e.g. BUILD_TIMESTAMP, GIT_REVISION, etc): The source for these variables are volatile-status.txt and stable-status.txt
I must generate a POM for publish, so I use #bazel_common//tools/maven:pom_file in BUILD
(assume that I need ALL the values described above for my pom template):
_local_build_properties = {}
_local_build_properties.update(BUILD_PROPERTIES)
# somehow add workspace status properties?
# add / override
_local_build_properties.update({
"pom.project.name": "my-submodule",
"pom.project.description": "My submodule description",
"pom.artifact.id": "my-submodule",
})
# Variable placeholders in the pom template are wrapped with {}
_pom_substitutions = { '{'+k+'}':v for (k,v) in _local_build_properties.items()}
pom_file(
name = "my_submodule_pom",
targets = [
"//my-submodule",
],
template_file = "//:pom_template.xml",
substitutions = _pom_substitutions,
)
So, my questions are:
How do I get key-value pairs from volatile/stable -status.txt into the
dictionary I need for pom_file.substitutions?
pom_file depends on java_library so that it can write its dependencies
into the POM. How do I update the jar generated by java_library with the
pom?
Once I have the pom and the updated jar containing the pom, how do I publish to a Maven repo?
When I look at existing code, for example rules_docker, it seems that the implementation always bails to a local executable (shell | python | go) to do the real work of substitution, jar manipulation and image publication. Am I trying to do too much in BUILD and BZL files? Should I be thinking, "Ultimately, what do I need to pass to local shell/python/go scripts to get real build work done?
(Answered on bazel-discuss group)
Hi,
You can't get these values from Starlark. You need a genrule to read the stable/volatile files and do the substitutions using an
external tool like 'sed'.
A file cannot be both an input and output of an action, i.e. you can't update the .jar from which you generate the pom. The action has
to produce a new .jar file.
I don't know -- how would you publish outside of Bazel, is there a tool to do so? Can you write a genrule / Starlark rule to wrap this
tool?
Cheers, László
Starting with Bazel v0.19, if you have Starlark (formerly known as "Skylark") code that references #bazel_tools//tools/jdk:jar, you see messages like this at build time:
WARNING: <trimmed-path>/external/bazel_tools/tools/jdk/BUILD:79:1: in alias rule #bazel_tools//tools/jdk:jar: target '#bazel_tools//tools/jdk:jar' depends on deprecated target '#local_jdk//:jar': Don't depend on targets in the JDK workspace; use #bazel_tools//tools/jdk:current_java_runtime instead (see https://github.com/bazelbuild/bazel/issues/5594)
I think I could make things work with #bazel_tools//tools/jdk:current_java_runtime if I wanted access to the java command, but I'm not sure what I'd need to do to get the jar tool to work. The contents of the linked GitHub issue didn't seem to address this particular problem.
I stumbled across a commit to Bazel that makes a similar adjustment to the Starlark java rules. It uses the following pattern: (I've edited the code somewhat)
# in the rule attrs:
"_jdk": attr.label(
default = Label("//tools/jdk:current_java_runtime"),
providers = [java_common.JavaRuntimeInfo],
),
# then in the rule implementation:
java_runtime = ctx.attr._jdk[java_common.JavaRuntimeInfo]
jar_path = "%s/bin/jar" % java_runtime.java_home
ctx.action(
inputs = ctx.files._jdk + other inputs,
outputs = [deploy_jar],
command = "%s cmf %s" % (jar_path, input_files),
)
Additionally, java is available at str(java_runtime.java_executable_exec_path) and javac at "%s/bin/javac" % java_runtime.java_home.
See also, a pull request with a simpler example.
Because my reference to the jar tool is inside a genrule within top-level macro, rather than a rule, I was unable to use the approach from Rodrigo's answer. I instead explicitly referenced the current_java_runtime toolchain and was then able to use the JAVABASE make variable as the base path for the jar tool.
native.genrule(
name = genjar_rule,
srcs = [<rules that create files being jar'd>],
cmd = "some_script.sh $(JAVABASE)/bin/jar $# $(SRCS)",
tools = ["some_script.sh", "#bazel_tools//tools/jdk:current_java_runtime"],
toolchains = ["#bazel_tools//tools/jdk:current_java_runtime"],
outs = [<some outputs>]
)
I need to declare specific resources for some of sub-projects and I'm doing it following way
filegroup(
name = "some_resources",
visibility = ["//:app"],
srcs = glob([
"src/my/resources/**/*.resources",
]),
)
In any subproject however the path where one can find resources is the same. My question would be - what is the most bazelian (bazelish?) way to minimize code duplication in this particular case?
Basically I want to have something like
expose_some_resources() in relevant subprojects.
and then make this resources visible for every app.
You can put the filegroup into a macro in a .bzl file, and load and run that macro in the relevant subprojects.
so something like:
workspace/resources.bzl:
def expose_some_resources():
native.filegroup(
name = "some_resources",
visibility = ["//:app"],
srcs = native.glob([
"src/my/resources/**/*.resources",
]),
)
workspace/subproject/BUILD:
load("//:resources.bzl", "expose_some_resources")
expose_some_resources()
You might also consider adding some error checking to the macro, like checking that the macro is called only once per package using native.existing_rule, or checking that the glob returns 1 or more files.
I'm building a C++ executable that needs to #include "Python.h" from the user's Python installation.
To express Python.h (and the various header files it includes) in Bazel, I need to know where the Python include directory is. This location will be different on Windows and Linux, and I'd like a single Bazel configuration to build them both.
What's the best Bazel practice for referencing software that exists outside of the WORKSPACE root?
So to tell Bazel about external dependencies you need to use one of the Workspace Rules to specify the location of the external dependency, as well as the BUILD file for Bazel to use with that external dependency.
To have something work cross-platform you need to use the select() function to have Bazel select the proper library to build against for your host operating system.
Here's a stab at accomplishing it:
First we have the WORKSPACE file in your project's root that defines the two libraries and the BUILD file to use for them. Here I'm just using build_file_content but if that becomes too complex you can put it in it's own file and reference it instead. The BUILD file here exposes the prebuild library shipped with Python along with the header files needed. It also adds an include path for any targets that depend on these libraries so you can do #include "Python.h"
new_local_repository(
name = "python_linux",
path = "/usr",
build_file_content = """
cc_library(
name = "python35-lib",
srcs = ["lib/python3.5/config-3.5m-x86_64-linux-gnu/libpython3.5.so"],
hdrs = glob(["include/python3.5/*.h"]),
includes = ["include/python3.5"],
visibility = ["//visibility:public"]
)
"""
)
new_local_repository(
name = "python_win",
path = "C:/Python35",
build_file_content = """
cc_library(
name = "python35-lib",
srcs = ["libs/python35.lib"],
hdrs = glob(["include/*.h"]),
includes = ["include/"],
visibility = ["//visibility:public"]
)
"""
)
Next the BUILD file for your application. Here you need to define some config_settings. This allows us to define platform dependent settings for our build. We use the cpu value to determine the host OS.
In the cc_binary rule we use the select() function to choose the correct host library to link against based on the configuration.
config_setting(
name = "linux_x86_64",
values = {"cpu": "k8"},
visibility = ["//visibility:public"],
)
config_setting(
name = "windows",
values = {"cpu": "x64_windows"},
visibility = ["//visibility:public"],
)
cc_binary(
name="python-test",
srcs = [
"main.c",
],
deps = select({
"//:linux_x86_64": [
"#python_linux//:python35-lib"
],
"//:windows": [
"#python_win//:python35-lib"
]
})
)
FWIW here's the main.c I was playing around with to get this working.
#include "Python.h"
int main(int argc, char *argv[])
{
Py_SetProgramName(argv[0]); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
"print('Today is',ctime(time()))\n");
Py_Finalize();
return 0;
}
Another way (and perhaps simpler) is checking the python headers and libraries into your repository. You will still need to use select() to choose the correct library to link against but at least you won't need to add anything to your WORKSPACE file and can just rely on another BUILD file in your repository. If you look at the Bazel repo they check in lots of external dependencies into the third_party directory, so it's a common practice.