I would like to version build artefacts with build number for CI passed to bazel via workspace_status_command. Sometimes I would like to include build number to the name of the artefact.
Is there a way how do I access ctx when writing a macro(as I was trying to use ctx.info_file)? So far it seems that I am able to access such info just in new rule when creating a new rule which in this case is a bit awkward.
I guess that having a build number or similar info is pretty common use case so I wonder if thre is a simpler way how to access such info.
No, you really need to define a custom rule to be able to consume information passed from workspace_status_command through info_file and version_file file and even then you cannot just access it's values from Starlark, you can pass the file to your tooling (wrapper) and process the inputs there. After all, (build) rules do not execute anything, they emit actions to be executed at a later phase.
Be careful though, because if you depend on info_file (STABLE_* entries), changes to the file invalidate targets depending on it. For something like CI build number, it's usually not what you want and version_file is more likely what you are after. You may want to record the id, but you usually do not want to rebuild stuff just because the build ID has changed (it's a new CI run). However, even simple inclusion of IDs could be considered problematic, if you want your results to be completely reproducible.
Having variable artifact names is a whole new problem and there would be good reasons why not to. But generally since as proposed the name would be decided during execution of actions (reading in version_file in your tool), you're past the analysis phase to decide what comes out of the action. The only way I am currently aware of (that is for out of tree source of variable input, you can of course always define a Starlark variable and load it from your BUILD file) to be able to do that is to use tree artifacts (using declare_directory in your rule.
Related
We have several automated build scripts, some of which are run automatically every 2 hours, and some of which are only ever run manually.
If a build script is started manually while another is already running, it can cause...problems. Such as merging untested branches into the production branch.
I'd like to prevent this happening again, and the simplest solution in my mind is to have each build script start by checking that another is not currently running.
Is there a way in ant to directly check if another ant instance/script is currently running?
If not, what's the simplest way to add such a check? My first thought is a file created at the beginning and deleted at the end of a build. I'd prefer a way that handles user-cancelled builds nicely, but it's not necessary. It needs to work if a build succeeded and if a build failed (but was not killed by the user).
If these are separate Ant processes, then I think the only solution is to define a lockfile of some sort that each Ant process needs to acquire before it can continue.
Perhaps the tempfile task could be used for this?
Actually, a sort-of semaphore based on a directory might be better because the tempfile really is a unique tempfile. The first thing your script does is use mkdir to create a shared resource directory name, but it only does this if the directory does not exist.
Upon exit it invokes delete on this shared resource name.
The idea is that the content and name of the directory is meaningless -- it only serves as an "IPC" cooperative locking mechanism.
This isn't particularly elegant, but I think your only other option is to set up a build server that handles scheduled and continuous builds based on various triggers. One that many people use is Jenkins (or has it been renamed?)
[Update]
Perhaps Do I have a way to check the existence of a directory in Ant (not a file)? would do the trick?
To be honest, this approach may work in the short term, but it just moves the problem around. Instead of resetting unit test results you'll be removing lockfiles by hand to get builds working again. My advice is to set up a CI build system, but I recognize this is a fair amount of work (and introduces a whole different set of future problems.)
Long story short,
I was wondering if anyone ever felt the need for (and knows of any implementation of) the possibility of "instantiating" (OO terminology) a parametrized build.
What I mean is treating a parametrized build as a template, from which many "instances" can be generated.
Each instance is supposed to define a different combination of values for the parameters.
The final goal is twofold:
DRY (which is given simply by the parametrized build concept)
having separate build histories / test reports for each instance (otherwise it would be a mess)
the instances would be schedulable directly in jenkins UI (while a parametrized build is not)
The template would then be used only for:
manual builds
changing the config for all of the instances at once
Now, time for some context, as I may be missing something in my overall approach.
You are welcome to point me in the right direction :)
I have a maven project with a suite of selenium tests that I want jenkins to run.
The suite is parametrized: browser, OS, test environment.
So, I can run it e.g. with mvn test -Dbrowser=chrome -Dplatform=win [..].
I want a separate test report for each combination of my parameters.
As a newbie, my first solution was "Copy existing job".
Quick and dirty. But effective.
As you will know, problems arise when you need to make a change to the configuration of the job, and you want to keep in sync all of these copy&pasted jobs.
Then I found the parametrized build feature.
It's very cool (code reuse/maintainability++), but the test report and the build history is shared among all of the actual builds, therefore I can not rely on them for a tidy reporting like "this test is always failing on IE; but it isn't on chrome", and so on.
Thank you very much in advance
I think what you are describing is the matrix project
There are also selenium plugins, I put one together to work with matrix jobs https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Axis+Plugin
One lack I can see: you can't build a single combination, as the build btn is present only at the "top level".
Have you tried the Matrix Combination plugin
https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Combinations+Plugin
Although this question specifically involves Gradle and Bamboo, it really is a question about any build system (Ant/Maven/Gradle/etc.) and any CI tool (Bamboo/Jenkins/Hudson/etc.).
I was always under the impression that the purpose of a CI build is to:
Check out code from VCS
Run a buildscript (Gradle, etc.)
Deploy a binary (WAR, etc.) to an environment
Hence, all the guts and heavy-lifting (running automated tests, code analysis, test coverage, compiling, Javadocs, packaging, etc.) was all to be done from inside the buildscript.
But Bamboo seems to allow you to break this heavy-lifting out of the buildscript and into Bamboo itself. In Bamboo, you can add build stages and decompose the stages into tasks. Each task is something just as atomic/fundamental as an Ant task.
So it got me thinking: how much should one empower the CI tool? What typical buildscript functionality should be transferred over to Bambooo/CI? For instance, should I be compiling from a Gradle task, or from a Bamboo task? Same goes for all tasks/stages.
For some reason, I view this as the same problem as to whether or not to use stored procedures or put the data processing all at the application layer. What are the pros/cons of each approach?
TL;DR at the bottom
My experience is with Jenkins, so examples will relate to that.
One thing with any build system (be it CI server or a buildscript), is that it should be stable, simple and self-contained so that an untrained receptionist (with printed instructions and proper credentials) could do it.
Ease of use and re-use
Based on the above, one would think that a buildscript wins. Not always. As with the receptionist example, it's about easy of use and easy of reproducibility.
If a buildscript has interdependent build targets that only work in correct order, dependence on pre-supplied property files that have to be adjusted for the correct branch ahead of build, reliance on environment variables that no-one remembers who created in the first place, and a supply of SCM revision numbers that have to be obtained by looking at the log of the commits for the last month... This is in no way better than a Jenkins job that can be triggered with a single button.
Likewise, a Jenkins workflow could be reliant on multiple dependant jobs, each being manually pre-configured before the build, and need artifacts uploaded from one place to another... which no receptionist will do.
So, at this point, a self-contained good buildscript that only requires ant build command to do everything from beginning to end, is just as good as a Jenkins job that only required build now... button to be pressed.
Self-contained
It is easy to think that since Jenkins will (at some point) end up calling at least a portion of a buildscript (say ant compile), that Jenkins is "compartmentalizing" the buildscript into multiple steps, thus breaking away from being self-contained.
However, instead you should zoom out by one level, and treat the whole Jenkins job configuration as a single XML file (which, by the way, can be stored and versioned through an SCM just like the buildscript)
So, at this point, it doesn't matter if the whole build logic is inside a single buildfile, or a single XML job configuration file. Both can be self-contained when done right.
The devil you know
In majority of cases, it comes down to what you know.
Some people find it easier to use Jenkins UI to visually arrange their build workflow, reporting, emailing, and archiving (and for anything that doesn't fit as wanted, find a plugin). For them, figuring out a build script language is more time consuming then simply trying it in UI.
Others prefer to know exactly what every single line of their build script does, and don't like giving control to some piece of foreign code obfuscated by UI.
Both points have merits from all sides Quality-Time-Budget triangle
The presentation
So far, things have been more or less balanced. However:
My Jenkins will email a detailed HTML report with a link to a job page and send it straight up to the (non tech-savvy) CEO. He can look at the list of latest builds, along with SCM changes for each build, linking him to JIRA issues fixed for each build (all hyperlinks to relevant places). He can select the build with the set of changes that he wants, and click "install iOS package" right off his iPad that he just used to view all this information. Meanwhile I can go to the same job page, and review the build logs and artifacts of each log, check the build time trends and compare the parameters that were used between the failing and succeeding jobs (and I didn't have to write any echos to display that, it's just all there, cause Jenkins does that for you)
With a buildscript, even if you piped the output to a file, would you send that to your (non tech-savvy) CEO? Unlikely. But wait, you know this devil very well. A few quick changes and hacks, couple Red Bulls... and months of thankless work (mostly after-hours) later... you've created a buildscript that will create and start a webserver, prepare HTML reports, collect statistics and history, email all the relevant people, and publish everything on a webpage, just like Jenkins did. (Ohh, if people could only see all the magic you did escaping and sanitizing all that HTML content in a buildscript). But wait... this only works for a single project.
So, a full case of Red Bulls later, you've managed to make it general enough to build any project, and you've created...
Another Jenkins/Bamboo/CI-server
Congratulations. Come up with a name, market it, and make some cash of it, cause this ultimate buildscript just became another CI solution a la Jenkins.
TL;DR:
Provided the CI-server can be configured simply and intuitively so that a receptionist could run the build, and provided the configuration can be self-contained (through whatever storage method the CI-server uses) and versioned in SCM, it all comes down to the Quality-Time-Budget triangle.
If you have little time and budget to learn the CI server, you can still greatly increase the quality (at least of the presentation) by embracing the CI-server's way of organizing stuff.
If you have unlimited time and budget, by all means, make your own Jenkins with the buildscript.
But considering the "unlimited" part is rather unrealistic, I would embrace the CI-server as much as possible. Yes, it's a change. However a little time invested in learning the CI-server and how it compartmentalizes or breaks into tasks the different parts of the build flow, this time spent can go a long way to increasing the quality.
Likewise, if you have no time and/or budget, figuring out the quirks of all the plugins/tasks/etc and how it all comes together will only bring your overall quality down, or even drag the time/budget down with it. In such cases, use the CI-server for bare minimum needed to trigger your existing buildscripts. However, in some cases, the "bare minimum" is no better than not using the CI-server in the first place. And when you are at this place... ask yourself:
Why do you want a CI-server in the first place?
Personally (and with today's tools), I'd take a pragmatic approach. I'd do as much as feasible on the build side (clearly better from an automation perspective), and the rest (e.g. distribution of work across machines) on the CI server. Anything that a developer might want to do on his own machine should definitely be automated on the build level. As to the concrete steps you gave, I'd generally check out code from the CI server, and deploy binaries from the build. I'd try to make every CI job look the same, invoking the build tool in the same way (e.g. gradlew ciBuild).
In Bamboo, you can add build stages and decompose the stages into tasks. Each task is something just as atomic/fundamental as an Ant task.
To some extent, this overlap in functionality is natural, as neither build tool nor CI server can assume existence of the other, and both want to provide as complete a solution as possible.
For some reason, I view this as the same problem as to whether or not to use stored procedures or put the data processing all at the application layer.
It's not an unfair comparison, and hence opinions will be as diverse, contextual, and nuanced.
Disclaimer: I'm a Gradle(ware) developer.
i have this ant build.xml file with 3 targets in it:
target1, target2 and target3.
If the user simply runs ant and not an explicit ant target1 or something like that, i want to prompt the user asking which target he would like to call.
Remember, the user should only be prompted for this only if he doesnt explicitly call a target while running ant
Ant is not a programming language, it's a dependency matrix language. There's a big difference between the two.
In a program language, you can specify the absolute order of sequence. Plus, you have a lot more flexibility in doing things. In Ant, you don't specify the execution order. You specify various short how to build this steps and then specify their dependencies. Ant automatically will figure out the execution order needed.
It's one of the hardest things for developers to learn about Ant. I've seen too many times when developers try to force execution order and end up executing the same set of targets dozens of times over and over. I recently had a build her that took almost 10 minutes to build, and I rewrote the build.xml to produce the same build in under 2 minutes.
You could use <input/> to get the user input, then use <exec> or <java> to execute another Ant process to execute the requested target. However, this breaks the way Ant is suppose to work.
The default target should be the default target that developers would want to execute on a regular basis while they program. It should not clean the build. It should not run 10 minutes of testing. It should compile any changed files, and rebuild the war or jar. That's what I want about 99% of the time. The whole process takes 10 seconds.
I get really, really pissed when someone doesn't understand this. I hate it when I type ant and I get directions on how to execute my build. I get really irritated when the default target cleans out my previous compiles. And, I get filled with the deadly desire to pummel the person who wrote the damn build file with a large blunt object if I am prompted for something. That's because I will run Ant, do something else while the build happens, then come back to that command window when I think the build is done. Nothing makes me angrier to come back to a build only to find out it's sitting there waiting for me to tell it which target.
If you really, really need to do this. Use a shell script called build.sh. Don't futz with the build.xmlto do this because that affects development.
What you really need to do is teach everyone how to use Ant:
Ant will list user executable targets when you type in ant -p. This will list all targets, and their descriptions. If a target doesn't have a description, it won't list it. This is great for internal targets that a user shouldn't execute on their own. (For example, a target that merely does some sort of test to see if another target should execute). To make this work, make sure your targets have descriptions. I get angry when the person who wrote the Ant file puts a description for some minor target that I don't want, but forgets the description of the target I do want (like compile). Don't make David angry. You don't want to make David angry.
Use default target names for your group. That way, I know what targets do what across the entire project instead of one project using BUILD vs. build-programs vs. Compile vs build-my-stuff vs. StuffBuild. We standardized on Maven lifecycle names names. They're documented and there's no arguments or debates.
Do not use <ant/> or <antcall> to enforce build order. Do not divide your build.xml into a dozen separate build.xml programs. All of these probably break Ant's ability to build a target dependency matrix. Besides, many Ant tools that show dependency hierarchy in a build and they can't work across multiple build files.
Do not wrap your builds inside a shell script. If you do this, you're probably not understanding how builds work.
The build should not update any files in my working directory that were checked out by me. It shouldn't polute my working directory with all sorts of build artifacts spread out all over the place. It shouldn't do anything outside of the working directory (except maybe do some sort of deploy, but only when I run the deploy target). In fact, all build processing should take place in a sub-directory INSIDE my working directory. A clean should merely delete this one directory. Sometimes, this is called build, sometimes dist. I usually call it target because I've adopted Maven naming conventions.
Your build script should be a build script. It shouldn't do checkouts or updates -- at least not automatically. I know that if you use CruiseControl as a continuous build process, you have to have update and checkout functionality inside your build.xml. It's one of the reasons I now use Jenkins.
Sorry about this answer not necessarily being the one you're looking for. You didn't really state what you're doing with Ant. If you're doing builds, don't do what you're trying to do. If you're writing some sort of program, use a real programming language and not Ant.
An Ant build should typically finish in under a minute or two, and redoing a build because you changed a file shouldn't take more than 30 seconds. This is important to understand because I want to encourage my developers to build with Ant, and to use the same targets that my Jenkins server uses. That way, they can test out their build the same way my Jenkins server will do the official build.
you may use the input task provided by ant and make it your default target.
<input
message="Please enter Target ID (1,2 or 3):"
validargs="1,2,3"
addproperty="targetID"
/>
Use the value of this property to decide which target to execute.
From the ant documentation:
message : The Message which gets displayed to the user during the build run.
validargs: Comma separated String containing valid input arguments. If set, input task will reject any input not defined here.
You may pass any arguments according to your needs.
addproperty : The name of a property to be created from input.Behaviour is equal to property task which means that existing
properties cannot be overridden.
I have a Jenkins build that can either be triggered via scheduling, by a user requesting it, or by being called as a build step from other builds. If this build is called as part of another build, it needs to save some information for the larger build to use. I want to pass this information back up by writing to a file. The only problem is having the builds agree on a location to write to.
One approach is to write it to a well known location, but this does not allow several builds to be run in parallel since one will clobber the other.
Another is to add a build parameter to the build that other builds will fill in with a file location to write to. This, to me, seems like a bit of a hack since it means that whenever the build is run, it will need to have a parameter passed in, even if it is just starting with the default value.
The final approach that I considered was having the parent build set an environment variable in the build and having the child check for the existence and content of the variable and act appropriately. Unfortunately, I cannot find a way to set this up in Jenkins.
It seems to me that a combination of archiving artifacts in post build and the Copy Artifact Plugin would do the job.
It sounds like you need the Parameterized Trigger Plugin.