I observe that my Bazel build agent frequently builds the project from scratch (including compiling grpc, which keeps unchanged) instead of taking results from cache. Is there a way, like query or cquery (pardon my ignorance) to determine why is the cache considered invalid for particular target? Or any techniques to tackle cache invalidation problem?
This is How the bazel build works :
When running a build or a test, Bazel does the following: Loads the BUILD files relevant to the target. Analyzes the inputs and their dependencies, applies the specified build rules, and produces an action graph. Executes the build actions on the inputs until the final build outputs are produced.
If you are having any clear assumptions can you please share the complete details!
This is most likely due to the rebuild sensitivity to particular environment variables. Many build actions will read from environment variables and use them to change the outputs. Bazel keeps track of this and will rebuild seemingly unchanged remote targets when your env changes.
To demonstrate this;
Build grpc (2x ensure it is cached the second time)
Change the PATH environment variable (your IDE may do this without you knowing)
mkdir ~/bin && export PATH=$PATH:~/bin
Rebuild grpc (This should trigger a complete rebuild)
There are a couple helpful flags to combat this rebuild sensitivity, and I'd recommend adding them to your bazelrc.
incompatible_strict_action_env: Freezes your environment and doesn't source environment variables from your shell.
action_env modify environment variables as needed for you build.
# file //.bazelrc
# Don't source environment from shell
build --incompatible_strict_action_env
# Use action_env as needed for your project
build --action_env=CC=clang
Related
How is it possible to access which features the package is being built with, inside the build.rs script? There is an incredibly expensive step in the script which is only needed for a particular cargo feature, but I can't see any way to access config features inside the build script.
Is there any way to read whether or not a given feature is enabled in the build.rs script?
I haven't been able to find documentation here, but was able to figure out one solution by guessing.
Cargo features are available as build features not just in the main source files, but inside the build.rs script as well. So you can use any of the standard ways to check configuration, like the cfg! and #[cfg(feature = "...")] macros, as mentioned in https://doc.rust-lang.org/reference/conditional-compilation.html and How do I use conditional compilation with `cfg` and Cargo?
Cargo sets a number of environment variables when the build scripts are run:
https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts
Including an environment variable for each feature:
CARGO_FEATURE_<name> — For each activated feature of the package being built, this environment variable will be present where <name> is the name of the feature uppercased and having - translated to _.
As part of our efforts to create a bazel-maven transition interop tool (that creates maven sized jars from more granular sized bazel jars),
we have written an aspect that runs on bazel build of the entire bazel repo and writes txt files outputs.
We want to write these aspect outputs only for non-cashed targets.
Even better will be to have a list at the end of the run that contains all the targets that were run (not skipped due to them being cached)
Are 1. and 2. possible?
We want to write these aspect outputs only for non-cashed targets.
I am not 100% sure what you mean here. It is undetectable if the target was cached or not. However, output files will be cached. If you run the same build with the same aspect, only the files that are not up-to-date will be updated.
... to have a list at the end of the run that contains all the targets that were run
We have a flag --experimental_show_artifacts that prints you all the artifacts that were built.
I am trying to follow the instructions for contributors here:
https://bazel.build/contributing.html
I have a successful build off of master (i.e. bazel build //src:bazel), but the doc suggests also "you might want to build the various tools Bazel uses." I am trying to do that, for example:
cd src/java_tools/singlejar
bazel build //...
but it fails with:
ERROR: /Users/.../bazel/third_party/protobuf/3.2.0/BUILD:621:1: no such target '//external:gtest': target 'gtest' not declared in package 'external' defined by /Users/plaird/scone/public/bazel/WORKSPACE and referenced by '//third_party/protobuf/3.2.0:test_plugin'.
Do I need to build gtest locally, and then add it to the WORKSPACE file?
bazel build //..., no matter where you invoke it, will build everything in the project. It looks like what you probably want is bazel build //src/java_tools/singlejar/..., which will build all targets under that directory.
In general, though, you probably don't need to compile singlejar separately. I've been working on Bazel for several years and 99% of the time you don't have to build the tools separately.
In terms of the error you're getting, it would be nice if we could get //... building, but it hasn't been a huge priority. The protobuf code build is weird and I don't recommend trying to debug it, just jump into whatever you want to actually work on.
For my release process I need several different "modes." However, if I use the --mode option for pub build and set it to any value other than release, it forces un-minified javascript.
I know I can configure the dart2js transformer in my pubspec.yaml, but if I set minify: true under the $dart2js heading in my pubspec.yaml I am then forcing them to be minified, and then cannot produce un-minified debug builds.
What I'm really looking for is a way to configure arbitrary dart2js options (minified, checked, etc.) in pub build via the CLI (so that I don't have to hardcode in pubspec.yaml), or, failing that, to be able to specify additional arbitrary flags from the pub build CLI so that I can reserve --mode for debug and release. The asPlugin() transformer constructor takes a BarbackSettings object, but I can't see how to see arbitrary params in that via the command line.
I have never seen anything like that mentioned (for example in any of the bug reports) and I'm pretty sure this is not supported. I suggest to just create a feature request at http://dartbug.com.
One way I can think of is to create a script which manipulates the pubspec.yaml file before executing pub build. This should be easy using the https://pub.dartlang.org/packages/yaml package.
I am writing a script to automate my iOS building. It will be run outside of Xcode, either via Terminal or from build automating software. Is there any way to have access to Xcode environment variables in my script, so I don't have to try and derive them myself?
For example, can I get access to PROJECT_DIR instead of assuming I'm in the current directory and running pwd?
I am currently hardcoding the product names for my different build configurations. (I'm also hard coding the build configs, but I could parse them them from xcodebuild -list.) Is there a way to get the app if you know the build config name?
(I saw this related question but it doesn't have an answer for me.)
The Xcode environment variables are only defined for child processes of the xcodebuildcommand-line tool.
One solution I used is to have a very simple script as part of my build process (Project->New Build Phase->Add Run Script Build Phase). All this script does is export the necessary variables and call a script in my path somewhere.
That script could be generated by your build script before calling xcodebuild and voilà! you have an external script that has access to Xcode build variables.