On successful build of a go binary (cli) I would like to publish the binary to a repository manager e.g. Artifactory. I have found various references to uploading jar dependencies but nothing specific to the rules_go
Can anyone point me in the right direction?
I don't think there is a publish story for go.
It took 4 years to implement #1372, it's probably easier to leave the publish part out of the Bazel whose mission is to build.
Thanks for the followup. I kept digging and did not find a solutions but came up with the following.
def _local_deploy_impl(ctx):
target = ctx.attr.target
shell_commands = ""
for s in ctx.files.srcs:
shell_commands += "sudo cp %s %s\n" % (s.short_path, target) # <2>
ctx.actions.write(
output = ctx.outputs.executable,
is_executable = True,
content = shell_commands,
)
runfiles = ctx.runfiles(files = ctx.files.srcs)
return DefaultInfo(
executable = ctx.outputs.executable,
runfiles = runfiles,
)
local_deploy = rule(
executable = True,
implementation = _local_deploy_impl,
attrs = {
"srcs": attr.label_list(allow_files = True),
"target": attr.string(default = "/usr/local/bin", doc = "Deployment target directory"),
},
)
Including in a build file:
local_deploy(
name = "install",
srcs = [":binary"],
)
srcs references the binary outputs of another rule that will be installed locally.
Any suggestions to improve?
Related
I'm trying to consume the output of https://github.com/OpenAPITools/openapi-generator-bazel, which happily takes my openapi.json file and generates a tree of source code for python.
Unfortunately I can't use it directly, because I need only the python files in a subdirectory to be used as sources in a py_library rule.
...but it seems to produce a single generated file, which is a ctx.actions.declare_directory sort of artifact.
I'm flummoxed. I can make a rule that extracts a subdirectory from that in the same way using ctx.actions.run_shell, but since you have to declare every output file in a rule, and the directory is opaque to bazel, I can't declare every output file with an action, so I can't find any way to iterate across the input directory.
Surely, surely, surely there is a way to filter a TreeArtifact by inspection. Any ideas?
It's not totally clear how the authors intended for openapi_generator to be used, because in general directory outputs are not well supported as outputs of targets themselves. E.g. py_library and java_library don't know to look inside the directory outputs of other targets. At least for now, typically directory outputs are more for passing things between actions within the implementations of rules.
Indeed there's an open issue for this on OpenAPI: https://github.com/OpenAPITools/openapi-generator-bazel/issues/22
And there's a related Bazel bug about taking directory outputs as srcs, at least for the Java rules: https://github.com/bazelbuild/bazel/issues/11996
Compare to protocol buffers, for example, where (usually) a .proto file corresponds to 1 output python file (foo.proto -> foo_pb2.py), so the outputs can be derived from the py_proto_library's srcs directly.
Anyway, one workaround is to explicitly list the expected output files:
defs.bzl:
def _get_openapi_files(ctx):
for out in ctx.outputs.outs:
ctx.actions.run_shell(
inputs = ctx.files.src,
outputs = [out],
command = "cp {src} {dst}".format(
src = ctx.files.src[0].path + "/" + out.short_path,
dst = out.path,
),
)
return DefaultInfo(files = depset(ctx.outputs.outs))
get_openapi_files = rule(
implementation = _get_openapi_files,
attrs = {
"src": attr.label(mandatory = True),
"outs": attr.output_list(mandatory = True),
},
)
BUILD:
load("#openapi_tools_generator_bazel//:defs.bzl", "openapi_generator")
load(":defs.bzl", "get_openapi_files")
openapi_generator(
name = "gen_petstore_python",
generator = "python",
spec = "petstore.yaml",
)
get_openapi_files(
name = "get_petstore_python_files",
src = ":gen_petstore_python",
outs = [
"openapi_client/models/__init__.py",
"openapi_client/apis/__init__.py",
"openapi_client/__init__.py",
"openapi_client/model_utils.py",
"openapi_client/api/__init__.py",
"openapi_client/api/pets_api.py",
"openapi_client/rest.py",
"openapi_client/configuration.py",
"openapi_client/exceptions.py",
"openapi_client/api_client.py",
"openapi_client/model/__init__.py",
"openapi_client/model/pets.py",
"openapi_client/model/pet.py",
"openapi_client/model/error.py",
],
)
py_library(
name = "petstore_python",
srcs = [":get_petstore_python_files"],
)
py_binary(
name = "petstore_main",
srcs = [":petstore_main.py"],
deps = [":petstore_python"],
)
petstore_main.py:
from openapi_client.model import pet
p = pet.Pet(123, "lassie")
print(p)
petstore.yaml is https://github.com/OpenAPITools/openapi-generator-bazel/blob/fb7e302de4597277bea12757836f2ce988c805ee/internal/test/petstore.yaml
$ bazel run petstore_main
INFO: Analyzed target //:petstore_main (45 packages loaded, 600 targets configured).
INFO: Found 1 target...
Target //:petstore_main up-to-date:
bazel-bin/petstore_main
INFO: Elapsed time: 2.150s, Critical Path: 1.42s
INFO: 20 processes: 5 internal, 15 linux-sandbox.
INFO: Build completed successfully, 20 total actions
INFO: Build completed successfully, 20 total actions
{'id': 123, 'name': 'lassie'}
The obvious downside is that any time you make a modification to the API definition that changes the what files are created, you have to go and update the BUILD file. And creating the list of output files might be tedious if you have a lot of api definitions.
Another workaround is to take advantage of the fact that Python doesn't really get compiled in the build system, and play some symlink tricks. However, this requires setting --experimental_allow_unresolved_symlinks (which can be added to the .bazelrc file):
defs.bzl:
def _symlink_openapi_files_impl(ctx):
symlink = ctx.actions.declare_symlink("openapi_client")
ctx.actions.symlink(
output = symlink,
target_path = ctx.files.src[0].path + "/openapi_client")
return [
DefaultInfo(
default_runfiles = ctx.runfiles(files = ctx.files.src + [symlink])),
PyInfo(transitive_sources = depset(ctx.files.src)),
]
symlink_openapi_files = rule(
implementation = _symlink_openapi_files_impl,
attrs = {
"src": attr.label(mandatory = True),
},
)
BUILD:
load("#openapi_tools_generator_bazel//:defs.bzl", "openapi_generator")
load(":defs.bzl", "symlink_openapi_files")
openapi_generator(
name = "gen_petstore_python",
generator = "python",
spec = "petstore.yaml",
)
symlink_openapi_files(
name = "symlink_petstore_python_files",
src = ":gen_petstore_python",
)
py_binary(
name = "petstore_main",
srcs = [":petstore_main.py"],
deps = [":symlink_petstore_python_files"],
)
$ bazel run petstore_main --experimental_allow_unresolved_symlinks
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed target //:petstore_main (48 packages loaded, 634 targets configured).
INFO: Found 1 target...
Target //:petstore_main up-to-date:
bazel-bin/petstore_main
INFO: Elapsed time: 1.797s, Critical Path: 1.38s
INFO: 7 processes: 6 internal, 1 linux-sandbox.
INFO: Build completed successfully, 7 total actions
INFO: Build completed successfully, 7 total actions
{'id': 123, 'name': 'lassie'}
Another alternative is to use a repository rule to generate the files. A repository rule can do things outside the regular rule rule / target execution model like generating BUILD files, however this amounts to basically reimplementing OpenAPI's Bazel integration.
Good response from ahumesky! It turns out at least in the Python case that there's another way. One way is to do as ahumesky says and explicitly declare all the output files in a separate rule, and that works well. Another way is to declare your own rule which can accept a TreeArtifact, but the sneaky way to do it is wrap the output in a PyInfo provider, as is done at https://github.com/jvolkman/rules_pycross/blob/main/pycross/private/wheel_library.bzl, which I cannibalized for this, although I will probably change to using the symlink approach above:
def _python_client_from_openapi_impl(ctx):
"""Rule that generates the python library from the openapi-generator source
directory (a tree)"""
output = ctx.actions.declare_directory(ctx.attr.package_name) # label.name)
# because the openapi generator imports the package directly ("import [packagename]")
# and names things appropriately, we can't so easily just use it fully-qualified;
# we'll have to add it to the import path.
# If the package is 'jwt_generator/service', this adds '__main__/jwt_generator/service'
# to the import path, so if the client packagename was "jwt_client", you can just
# import jwt_client
imp = paths.join(
ctx.label.workspace_name or ctx.workspace_name,
ctx.label.package,
)
print(imp)
imports = depset(
direct=[imp], transitive=[d[PyInfo].imports for d in ctx.attr.deps]
)
ctx.actions.run_shell(
outputs=[output],
inputs=ctx.files.srcs,
command="cp -r {}/{}/* {}".format(
ctx.files.srcs[0].path, ctx.attr.package_name, output.path
),
)
# now all the relative python sources are in our output directory
transitive_sources = depset(direct=[output], transitive=[])
# runfiles (https://bazel.build/rules/lib/runfiles)
# "a set of files required at runtime execution" -- which in the case of
# interpreted languages generally means much of the source
runfiles = ctx.runfiles(files=[output])
return [
DefaultInfo(files=depset([output]), runfiles=runfiles),
PyInfo(
has_py2_only_sources=False,
has_py3_only_sources=True,
imports=imports,
transitive_sources=transitive_sources,
),
]
python_client_from_openapi = rule(
implementation=_python_client_from_openapi_impl,
attrs={
"srcs": attr.label(allow_files=True, doc="Output of openapi_client rule"),
"package_name": attr.string(),
"deps": attr.label_list(
doc="A list of the client's dependencies; typically just urllib3 and python-dateutil",
providers=[DefaultInfo, PyInfo],
),
},
)
This actually works pretty well although it was a sort of first-try hack and not robust/efficient, so I'm going to probably use a combination of the two.
I have two repositories with Bazel builds. The one uses the Bazel imports for building protobuf types. When I make this repo/workspace a dependency of my other repo/workspace I have to re-add all of the protobuf imports for rules_proto and rules_cc to this new dependent workspace. Is there a way to not have to do this, or is this expected behavior with Bazel builds?
Example:
WORKSPACE A:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "bazel_skylib",
strip_prefix = "bazel-skylib-67bfa0ce4de5d4b512178d5f63abad1696f6c05b",
urls = ["https://github.com/bazelbuild/bazel-skylib/archive/67bfa0ce4de5d4b512178d5f63abad1696f6c05b.tar.gz"],
)
http_archive(
name = "rules_cc",
sha256 = "d1886f0ea5b6cfe7519b87030811f52620db31bcca7ef9964aa17af2d14f23c4",
strip_prefix = "rules_cc-cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be.tar.gz",
"https://github.com/bazelbuild/rules_cc/archive/cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be.tar.gz",
],
)
http_archive(
name = "rules_proto",
sha256 = "fb7f1959d2d2bf4d7a1f4f29d650845a9a2303b7879c6792320ba8244910ab01",
strip_prefix = "rules_proto-3212323502e21b819ac4fbdd455cb227ad0f6394",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_proto/archive/3212323502e21b819ac4fbdd455cb227ad0f6394.tar.gz",
"https://github.com/bazelbuild/rules_proto/archive/3212323502e21b819ac4fbdd455cb227ad0f6394.tar.gz",
],
)
load("#rules_cc//cc:repositories.bzl", "rules_cc_dependencies")
rules_cc_dependencies()
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()
WORKSPACE B (has dependency on WORKSPACE A):
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
name = "workspace-a",
branch = "bazel",
remote = "url/to/my/git/repo",
)
# WHY DO I HAVE TO DO THE FOLLOWING WHEN IT'S DONE IN WORKSPACE A ALREADY?
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "bazel_skylib",
strip_prefix = "bazel-skylib-67bfa0ce4de5d4b512178d5f63abad1696f6c05b",
urls = ["https://github.com/bazelbuild/bazel-skylib/archive/67bfa0ce4de5d4b512178d5f63abad1696f6c05b.tar.gz"],
)
http_archive(
name = "rules_cc",
sha256 = "d1886f0ea5b6cfe7519b87030811f52620db31bcca7ef9964aa17af2d14f23c4",
strip_prefix = "rules_cc-cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be.tar.gz",
"https://github.com/bazelbuild/rules_cc/archive/cb6d32e4d1ae29e20dd3328109d8cb7f8eccc9be.tar.gz",
],
)
http_archive(
name = "rules_proto",
sha256 = "fb7f1959d2d2bf4d7a1f4f29d650845a9a2303b7879c6792320ba8244910ab01",
strip_prefix = "rules_proto-3212323502e21b819ac4fbdd455cb227ad0f6394",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_proto/archive/3212323502e21b819ac4fbdd455cb227ad0f6394.tar.gz",
"https://github.com/bazelbuild/rules_proto/archive/3212323502e21b819ac4fbdd455cb227ad0f6394.tar.gz",
],
)
load("#rules_cc//cc:repositories.bzl", "rules_cc_dependencies")
rules_cc_dependencies()
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()
It's mostly because bazel wasn't really originally designed to be a package manager. Bazel comes from a monorepo design where everything (your applications, libraries, third-party dependencies, data, tools, toolchains, compilers, Bazel itself, etc) is checked into a single repository with a single version of everything. So there's no question of where things come from and how to integrate them beyond the domain-specific rulesets (java, c++, python, etc). It's all just there in BUILD files. The WORKSPACE design and external dependencies are for bridging the monorepo approach and non-monorepo approach.
Some mitigations of the inconveniences are the macros like rules_proto_dependencies() and rules_proto_toolchains(), but you still have to load many things at the top-level, and resolve diamond-dependency problems manually.
There's a new system call "bzlmod" that's being developed to replace all of this. It was recently experimentally released in Bazel 5.0:
https://bazel.build/docs/bzlmod
We frequently need common combinations of rules per tech stack.
That currently wastes a lot of space in WORKSPACE - and they should be kept in sync over multiple repos. It's 50+ lines after buildifier and contains too many urls, versions and hashes.
Now say I have a "technology stack" repo and do something like
load("#techstack_repo//mylang.bzl", "load_rules")
load_rules()
where load_rules would load and initialize pinned versions of e.g. rules_go, bazel-gazelle, rules_docker, rules_proto and initialize all of them in the right order so they are visible in WORKSPACE?
I did not get this to work in my tests because load apparently can not be run in a function in a bzl file - it's not a function itself.
Is there a way to do this?
Here's an example of what I tested for Java:
load("#io_bazel_rules_docker//repositories:repositories.bzl", container_repositories = "repositories")
load("#io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
load("#io_bazel_rules_docker//container:container.bzl", "container_pull")
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
load(
"#io_grpc_grpc_java//:repositories.bzl",
"IO_GRPC_GRPC_JAVA_ARTIFACTS",
"IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS",
"grpc_java_repositories",
)
load("#rules_jvm_external//:defs.bzl", "maven_install")
def prepare_stack(maven_deps = []):
container_repositories()
container_deps()
container_pull(
name = "java_base",
# https://console.cloud.google.com/gcr/images/distroless/GLOBAL/java-debian10
# tag = "11", # OpenJDK 11 as of 2020-03-04
digest = "sha256:eda9e5ae2facccc9c7016f0c2d718d2ee352743bda81234783b64aaa402679b6",
registry = "gcr.io",
repository = "distroless/java-debian10",
)
rules_proto_dependencies()
rules_proto_toolchains()
maven_install(
artifacts = maven_deps + IO_GRPC_GRPC_JAVA_ARTIFACTS,
# for improved debugging in IDE
fetch_sources = True,
generate_compat_repositories = True,
override_targets = IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS,
repositories = [
"https://repo.maven.apache.org/maven2/",
"https://repo1.maven.org/maven2",
],
strict_visibility = True,
)
grpc_java_repositories()
... all http_archive calls for the rule repos are in WORKSPACE and I want to move them in here, but that did not work at all.
As is, I get this error:
ERROR: Failed to load Starlark extension '#rules_python//python:pip.bzl'.
Cycle in the workspace file detected. This indicates that a repository is used prior to being defined.
The following chain of repository dependencies lead to the missing definition.
- #rules_python
This could either mean you have to add the '#rules_python' repository with a statement like `http_archive` in your WORKSPACE file (note that transitive dependencies are not added automatically), or move an existing definition earlier in your WORKSPACE file.
also adding rules_python does not help either.
I found a solution:
Split it into two files.
One with imports like this:
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("#bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
def declare():
maybe(
git_repository,
name = "rules_cc",
commit = "34ca16f4aa4bf2a5d3e4747229202d6cb630ebab",
remote = "https://github.com/bazelbuild/rules_cc.git",
shallow_since = "1584036492 -0700",
)
# ... for me requires at least rules_cc, rules_python, bazel_skylib
# for later proto, docker, go, java support
and another using the declared external sources:
# go
load("#io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("#bazel_gazelle//:deps.bzl", "gazelle_dependencies")
# protobuf
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
# container
load("#io_bazel_rules_docker//container:container.bzl", "container_pull")
load("#io_bazel_rules_docker//repositories:repositories.bzl", container_repositories = "repositories")
load("#io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
load("#io_bazel_rules_docker//go:image.bzl", go_image_repositories = "repositories")
def init_rules():
go_rules_dependencies()
go_register_toolchains()
gazelle_dependencies()
rules_proto_dependencies()
rules_proto_toolchains()
container_repositories()
container_deps()
go_image_repositories()
container_pull(
name = "go_static",
digest = "sha256:9b60270ec0991bc4f14bda475e8cae75594d8197d0ae58576ace84694aa75d7a",
registry = "gcr.io",
repository = "distroless/static",
)
It's a bit of a hassle, but fetch this repo with http_archive or git_repository, load the first file and call declare and load the second for init_rules and call that.
It may be a little convoluted, but it still helps to unify the stack and simplify your WORKSPACE.
I've got two repositories, Client and Library.
Inside of Client's WORKSPACE file Client imports Library as a http_archive with the name "foo".
Inside of Client, I want to use Library macros that reference targets inside Library. My problem is that the Library macros don't know that were imported as "foo", so when the macro is expanded the targets are not found.
library/WORKSPACE:
workspace(name = "library")
library/some.bzl:
def my_macro():
native.java_library(name = "my_macro_lib",
deps = ["#library//:my_macro_lib_dependnecy"]
)
library/BUILD.bazel:
java_library(name = "my_macro_lib_dependnecy",
...
)
client/WORKSPACE:
workspace(name = "client")
http_archive(
name = "library",
urls = [...],
strip_prefix = ...,
sha256 = ...,
)
Because both workspaces use the same name for library workspace (name = "library") and because the macro refers to the workspace name in its dependencies (#library//:my_macro_lib_dependnecy) this works.
Note this works but has some quirks which will be resolved in 0.17.0
I'm building ARM Cortex-M firmware from Bazel with a custom CROSSTOOL. I'm successfully building elf files and manually objcopying them to binary files with the usual:
path/to/my/objcopy -o binary hello.elf hello.bin
I want to make a Bazel macro or rule called cc_firmware that:
Adds the -Wl,-Map=hello.map flags to generate a mapfile
Changes the output elf name from hello to hello.elf
Invokes path/to/my/objcopy to convert the elf to a bin.
I don't know how to get the name of a CROSSTOOL tool (objcopy) to invoke it, and it feels wrong to have the rule know the path to the tool executable.
Is there a way to use the objcopy that I've already told Bazel about in my CROSSTOOL file?
You can actually access this from a custom rule. Basically you need to tell Bazel that you want access to the cpp configuration information (fragments = ["cpp"]) and then access its path via ctx.fragments.cpp.objcopy_executable, e.g.,:
def _impl(ctx):
print("path: {}".format(ctx.fragments.cpp.objcopy_executable))
# TODO: actually do something with the path...
cc_firmware = rule(
implementation = _impl,
fragments = ["cpp"],
attrs = {
"src" : attr.label(allow_single_file = True),
"map" : attr.label(allow_single_file = True),
},
outputs = {"elf" : "%{name}.elf"}
)
Then create the output you want with something like (untested):
def _impl(ctx):
src = ctx.attr.src.files.to_list()[0]
m = ctx.attr.map.files.to_list()[0]
ctx.action(
command = "{objcopy} -Wl,-Map={map} -o binary {elf_out} {cc_bin}".format(
objcopy=ctx.fragments.cpp.objcopy_executable,
map=m,
elf_out=ctx.outputs.elf.path,
cc_bin=src,
),
outputs = [ctx.outputs.elf],
inputs = [src, m],
)