We have a bunch of targets being built by Bazel in our build environment today, targets being cc_binary/cc_test/cc_library etc. Is there some Bazel option that dumps build time for each of these into a file so we can analyze it after the build is done. While building Bazel dumps some timing information that disappears fast but I am looking for something (either text/XML or any other readable format) that can be used in post-build analysis.
To some extent this is possible. Look at the documentation: Performance profiling
There is the build flag --experimental_generate_json_trace_profile that generates a file named command.profile.gz in our outputBase dir (details about the outputBase dir can be fond here).
Example:
bazel build --experimental_generate_json_trace_profile --experimental_profile_cpu_usage //:HelloWorld
command.profile.gz contains a file named command.profile. This can be view in Chrome (chrome://tracing/).
(Tested with Bazel 2.0.0)
Related
I observe that my Bazel build agent frequently builds the project from scratch (including compiling grpc, which keeps unchanged) instead of taking results from cache. Is there a way, like query or cquery (pardon my ignorance) to determine why is the cache considered invalid for particular target? Or any techniques to tackle cache invalidation problem?
This is How the bazel build works :
When running a build or a test, Bazel does the following: Loads the BUILD files relevant to the target. Analyzes the inputs and their dependencies, applies the specified build rules, and produces an action graph. Executes the build actions on the inputs until the final build outputs are produced.
If you are having any clear assumptions can you please share the complete details!
This is most likely due to the rebuild sensitivity to particular environment variables. Many build actions will read from environment variables and use them to change the outputs. Bazel keeps track of this and will rebuild seemingly unchanged remote targets when your env changes.
To demonstrate this;
Build grpc (2x ensure it is cached the second time)
Change the PATH environment variable (your IDE may do this without you knowing)
mkdir ~/bin && export PATH=$PATH:~/bin
Rebuild grpc (This should trigger a complete rebuild)
There are a couple helpful flags to combat this rebuild sensitivity, and I'd recommend adding them to your bazelrc.
incompatible_strict_action_env: Freezes your environment and doesn't source environment variables from your shell.
action_env modify environment variables as needed for you build.
# file //.bazelrc
# Don't source environment from shell
build --incompatible_strict_action_env
# Use action_env as needed for your project
build --action_env=CC=clang
For some reasons, practical or not, rxjs npm package stores BAZEL.build configuration in the package, so when I'm trying to build my project (which has node_modules folder) bazel tries automatically to build something that it's not supposed to build at all.
My question would be - what is canonical way of ignoring some specific folder while building bazel project recursively?
The only way to achieve what I'm looking for that I know of is to point to it explicitly in the command line
bazel build //... --deleted_packages=node_modules/rxjs/src (see user manual)
But I don't want to type this every time.
Bazel recently added a feature for ignoring folders (similar to gitignore).
Simply add node_modules to the .bazelignore file in the root of your project.
Yes, this is expressible as a bazel target pattern:
bazel build -- //... -//node_modules/rxjs/src/...
Full documentation is available at https://docs.bazel.build/versions/master/user-manual.html#target-patterns
As part of our efforts to create a bazel-maven transition interop tool (that creates maven sized jars from more granular sized bazel jars),
we have written an aspect that runs on bazel build of the entire bazel repo and writes txt files outputs.
We want to write these aspect outputs only for non-cashed targets.
Even better will be to have a list at the end of the run that contains all the targets that were run (not skipped due to them being cached)
Are 1. and 2. possible?
We want to write these aspect outputs only for non-cashed targets.
I am not 100% sure what you mean here. It is undetectable if the target was cached or not. However, output files will be cached. If you run the same build with the same aspect, only the files that are not up-to-date will be updated.
... to have a list at the end of the run that contains all the targets that were run
We have a flag --experimental_show_artifacts that prints you all the artifacts that were built.
As part of our efforts to create a bazel-maven transition interop tool (that creates maven sized jars from more granular sized bazel jars),
we have written an aspect that runs on bazel build of the entire bazel repo and writes important information to txt files outputs (e.g.: jar file paths, compile deps targets and runtime deps targets, etc.)
We ran across an issue where the repo's code was changed such that some of the txt file were not written anymore. But the old txt file from previous runs (before the code change) remained!
Is there a way to know that these txt files are no longer relevant?
You should be able to run with --build_event_json_file=file.json and try to locate generated artifacts. For example we use it on ci.bazel.io to locate actual test.xml file that were generated: https://github.com/bazelbuild/continuous-integration/blob/09975cbb487a84a62ca1e43aa43e7c6fe078f058/jenkins/lib/src/build/bazel/ci/BazelUtils.groovy#L218
The definition of the protocol can be found in build_event_stream.proto
I am trying to follow the instructions for contributors here:
https://bazel.build/contributing.html
I have a successful build off of master (i.e. bazel build //src:bazel), but the doc suggests also "you might want to build the various tools Bazel uses." I am trying to do that, for example:
cd src/java_tools/singlejar
bazel build //...
but it fails with:
ERROR: /Users/.../bazel/third_party/protobuf/3.2.0/BUILD:621:1: no such target '//external:gtest': target 'gtest' not declared in package 'external' defined by /Users/plaird/scone/public/bazel/WORKSPACE and referenced by '//third_party/protobuf/3.2.0:test_plugin'.
Do I need to build gtest locally, and then add it to the WORKSPACE file?
bazel build //..., no matter where you invoke it, will build everything in the project. It looks like what you probably want is bazel build //src/java_tools/singlejar/..., which will build all targets under that directory.
In general, though, you probably don't need to compile singlejar separately. I've been working on Bazel for several years and 99% of the time you don't have to build the tools separately.
In terms of the error you're getting, it would be nice if we could get //... building, but it hasn't been a huge priority. The protobuf code build is weird and I don't recommend trying to debug it, just jump into whatever you want to actually work on.