I want to compile a large set of different bazel targets lets say [A, B, C, D] and understand which are broken.
I have notice that if I launch:
bazel build A
bazel build B
bazel build C
bazel build D
It takes a lot of time, not because of compilation time, but because of bazel setup time.
Is there an alternative solution for making this compilation process faster, but still having the ability to understand which targets are broken and which not?
Thanks.
You can do
bazel build --keep_going A B C D
Then bazel won't stop at the first broken target, and will try to build as much as it can.
That said, if the various targets don't share a lot of the same dependencies, I wouldn't expect a huge difference, since bazel should still be caching between different invocations. Also, if you're changing build flags between invocations, then bazel may have to rerun the analysis phase (this is what I assume you mean by 'bazel setup time'). And if you need different build flags for the different targets, then you might not be able to build them all in one invocation.
Related
Looking at the outputs from Bazel's build event protocol, I can see that the BEP output only has one unique invocation_id (which makes sense) and also only one unique build_id (even when building multiple targets at once).
In that case, what's the point of build_id?
A higher-level system that uses Bazel (e.g., a CI system) may have a concept of a build that is broader than one Bazel invocation. For instance, the higher-level system may retry entire Bazel invocations under certain circumstances or allow a "build" to contain multiple Bazel build or test steps. The build id allows multiple Bazel invocations composing a high-level build to be correlated in Bazel's metadata emissions (most notably, the Build Event Protocol).
New to Bazel. Looking to see if there is a way to create a Bazel rule that allows me to get a list of all targets and then feed that data into a kotlin file or something like that.
I was able to run bazel query //... --export xml > temp.xml this gives me all the targets and their build files info but I would like to retrieve this info using a bazel rule, any ideas of how I could go about this?
The short answer is no. The closest thing to this in Bazel is a genquery, though it's worth noting that there are some caveats to this approach as mentioned in the docs;
In order to keep the build consistent, the query is allowed only to visit the transitive closure of the targets specified in the scope attribute. Queries violating this rule will fail during execution if strict is unspecified or true (if strict is false, the out of scope targets will simply be skipped with a warning). The easiest way to make sure this does not happen is to mention the same labels in the scope as in the query expression.
If you are happy to blow past the warnings around 'build consistency' it might be possible to achieve this using a similar approach to the buildifier rules, where you would determine the path of the workspace and run Bazel query as a subprocess of Bazel. Personally, I wouldn't recommend this and instead would suggest that you just use the output of bazel query directly.
I am currently trying to measure the time bazel build //api/... takes to build our "api" project with different --spawn_strategys.
I am having a hard time doing so because Bazel doesn't rebuild anything as long as I don't touch the source files.
I was able to force rebuilds by editing all files inside our "api" project, but doing this repeatedly is cumbersome.
What is the best way to force Bazel to rebuild so that I can measure build times for our repository?
Preferably, I would like to use something like bazel build //api/... --some_option_which_forces_rebuilding or something similar.
A bit dirty, but you could use --action_env to change the build environment and invalidate all the actions. From the docs:
Environment variables are considered an essential part of an action. In other words, an action is expected to produce a different output, if the environment it is invoked in differs; in particular, a previously cached value cannot be taken if the effective environment changes.
also (from this page):
[...] The value of those environment variable can be enforced from the command line with the --action_env flag (but this flag will invalidate every action of the build).
Just setting a random variable should be enough:
> bazel build --action_env="avariable=1" :mytarget
> bazel build --action_env="avariable=2" :mytarget
> ...
If you delete all the stuff in the output directory for the api package and its subpackages (should be under the bazel-bin symlink), bazel will rerun all the actions that produced those things, and not run any actions that produced things in other packages. This should also avoid rerunning the analysis phase, which changing config_settings or --action_env will do.
originally, I use gradle to build my android project, but recently, I migrate it to bazel, and I find that bazel is truly fast than gradle, so I want to know why, but the doc of bazel doesn't give much idea about this, can anyone help me？
Thanks very much!
Full disclosure: I work on Bazel.
That's not an easy question to answer for two reasons. First, performance is highly dependent on the scenario. For example, we'd generally expect a clean build to be slower than a build where only a single file has changed. Second, I don't know how Gradle works internally, and they've done a lot of work recently to improve Gradle performance.
But I can talk about Bazel and what we're doing to make it fast. We've been working on build performance for ~10 years, starting long before we made it public.
The key feature is that we require all dependencies to be declared, and we track them explicitly. If you use a header file in C++, or depend on a Java library, you must declare this dependency in your BUILD file (and we enforce that these are declared by sandboxing individual actions). There are three effects from this:
First, we can heavily parallelize the build, because we know which things depend on which other things.
Second, we can make incremental builds very fast, because we can tell what parts of the build have to be re-done when you change a specific file (BUILD file, header file, source file, ...).
Third, we almost never have to do clean builds. Other build tools often require 'make clean' to get into a predictable state - since Bazel knows all the dependencies, it can get to a predictable state on every single build.
Another effect is that we can cache remotely (i.e., across users), and even execute on another machine, although neither of these are fully supported at the time of this writing.
i have this ant build.xml file with 3 targets in it:
target1, target2 and target3.
If the user simply runs ant and not an explicit ant target1 or something like that, i want to prompt the user asking which target he would like to call.
Remember, the user should only be prompted for this only if he doesnt explicitly call a target while running ant
Ant is not a programming language, it's a dependency matrix language. There's a big difference between the two.
In a program language, you can specify the absolute order of sequence. Plus, you have a lot more flexibility in doing things. In Ant, you don't specify the execution order. You specify various short how to build this steps and then specify their dependencies. Ant automatically will figure out the execution order needed.
It's one of the hardest things for developers to learn about Ant. I've seen too many times when developers try to force execution order and end up executing the same set of targets dozens of times over and over. I recently had a build her that took almost 10 minutes to build, and I rewrote the build.xml to produce the same build in under 2 minutes.
You could use <input/> to get the user input, then use <exec> or <java> to execute another Ant process to execute the requested target. However, this breaks the way Ant is suppose to work.
The default target should be the default target that developers would want to execute on a regular basis while they program. It should not clean the build. It should not run 10 minutes of testing. It should compile any changed files, and rebuild the war or jar. That's what I want about 99% of the time. The whole process takes 10 seconds.
I get really, really pissed when someone doesn't understand this. I hate it when I type ant and I get directions on how to execute my build. I get really irritated when the default target cleans out my previous compiles. And, I get filled with the deadly desire to pummel the person who wrote the damn build file with a large blunt object if I am prompted for something. That's because I will run Ant, do something else while the build happens, then come back to that command window when I think the build is done. Nothing makes me angrier to come back to a build only to find out it's sitting there waiting for me to tell it which target.
If you really, really need to do this. Use a shell script called build.sh. Don't futz with the build.xmlto do this because that affects development.
What you really need to do is teach everyone how to use Ant:
Ant will list user executable targets when you type in ant -p. This will list all targets, and their descriptions. If a target doesn't have a description, it won't list it. This is great for internal targets that a user shouldn't execute on their own. (For example, a target that merely does some sort of test to see if another target should execute). To make this work, make sure your targets have descriptions. I get angry when the person who wrote the Ant file puts a description for some minor target that I don't want, but forgets the description of the target I do want (like compile). Don't make David angry. You don't want to make David angry.
Use default target names for your group. That way, I know what targets do what across the entire project instead of one project using BUILD vs. build-programs vs. Compile vs build-my-stuff vs. StuffBuild. We standardized on Maven lifecycle names names. They're documented and there's no arguments or debates.
Do not use <ant/> or <antcall> to enforce build order. Do not divide your build.xml into a dozen separate build.xml programs. All of these probably break Ant's ability to build a target dependency matrix. Besides, many Ant tools that show dependency hierarchy in a build and they can't work across multiple build files.
Do not wrap your builds inside a shell script. If you do this, you're probably not understanding how builds work.
The build should not update any files in my working directory that were checked out by me. It shouldn't polute my working directory with all sorts of build artifacts spread out all over the place. It shouldn't do anything outside of the working directory (except maybe do some sort of deploy, but only when I run the deploy target). In fact, all build processing should take place in a sub-directory INSIDE my working directory. A clean should merely delete this one directory. Sometimes, this is called build, sometimes dist. I usually call it target because I've adopted Maven naming conventions.
Your build script should be a build script. It shouldn't do checkouts or updates -- at least not automatically. I know that if you use CruiseControl as a continuous build process, you have to have update and checkout functionality inside your build.xml. It's one of the reasons I now use Jenkins.
Sorry about this answer not necessarily being the one you're looking for. You didn't really state what you're doing with Ant. If you're doing builds, don't do what you're trying to do. If you're writing some sort of program, use a real programming language and not Ant.
An Ant build should typically finish in under a minute or two, and redoing a build because you changed a file shouldn't take more than 30 seconds. This is important to understand because I want to encourage my developers to build with Ant, and to use the same targets that my Jenkins server uses. That way, they can test out their build the same way my Jenkins server will do the official build.
you may use the input task provided by ant and make it your default target.
<input
message="Please enter Target ID (1,2 or 3):"
validargs="1,2,3"
addproperty="targetID"
/>
Use the value of this property to decide which target to execute.
From the ant documentation:
message : The Message which gets displayed to the user during the build run.
validargs: Comma separated String containing valid input arguments. If set, input task will reject any input not defined here.
You may pass any arguments according to your needs.
addproperty : The name of a property to be created from input.Behaviour is equal to property task which means that existing
properties cannot be overridden.