We are in the process of switching from VSoft Continua CI to Jenkins for our build management environment. As we use a slightly modified Gitflow process we would like Jenkins to be able to build from any feature, release or hotfix branch and pull requests, we decided to go for the Jenkins Pipeline.
The version number for builds from the release and hotfix branches are based on the branch name (e.g. release/2.1.0) while builds from any other branch or pull request is based on the date (e.g. September 6th 2018 resolves to 18.9.6). Continua CI provides a auto increment build number across all build configurations that is why we use this build number as the final part of our build number (e.g. 2.1.0.10, 18.9.6.11, 2.1.0.12, ...). This generated version number is passed as parameter to MSBuild using this version number as the file version and assembly version of our .NET binaries.
I'm looking for a similar solution in Jenkins. The Jenkins Pipeline assigns a separate auto increment build number per branch and pull request which might lead to two builds from different branches having the same version. I already tried using global environment variables to store the version and increase the value with every build but it seems that global enviroment variables cannot be set from Pipeline tasks.
Is there a way for a Jenkins Pipeline project to share a build number across all branches/pull requests?
Here are a few ideas:
have it file based: have your stage execute on, say, the master node; pick a file and decide on the format (properties file can be a good start); lock, read, update, write, unlock.
delegate this to an external service (for instance, a service with a
REST endpoint that you use to request an ID).
write a plugin for it.
Related
I want to have one Jenkins job control the build number of another job but without the inconvenience of reloading the entire project configuration from disk. I have seen that it's easily possible to directly update the nextBuildNumber file of the target job (I can do this as a build step of Job A) but this does not take effect immediately. Restarting Jenkins or even reloading the Jenkins configs from disk takes way too long and can only be done when there are no builds in progress.
I have tried the groovy script mentioned in the below post by running it from the Manage Jenkins > Script Console. The same post also suggests the script can be saved as a file and run from the CLI. Can it be run from a build step?
I want Job A to determine Job B's next build number and set it so that Job B can run (later in the same day) with the desired build number.
https://stackoverflow.com/a/20077362/4306857
Perhaps I should clarify. I'm not familiar with Groovy so I'm looking at the various build step options like "Execute Windows batch command" which I have a lot of experience with. I can see an "Invoke Gradle script" option so I was wondering if there may be a plugin that can run Groovy scripts perhaps?
The reason this requirement has arisen is that we are compiling a product for two different platforms. We want to compile the codebase almost simultaneously for both platforms with two jobs (A & B) which will both update the JIRA cases included in the builds. We feel it will be less confusing to have both these jobs running with the same build number so that when we talk about a particular issue being addressed in build #75, say, we won't have to qualify that by stating the full job name. If JOB-A build #75 and JOB-B build #75 are both compiled on the same day from the same codebase we can test and compare the results of both builds with far less confusion than if the build numbers go out of sync.
Obviously, in the short term we will use the Set Next Build Number plugin to manually keep the build numbers in step but we want to automate this if possible.
Depends on whether or not you are using Version Number plugin:
[ X ] Create a formatted version number
Build Display Name [ X ] Use the formatted version number for build display name.
Assuming you are NOT, this groovy script will do:
def NextNumber=42
job=Jenkins.instance.getItemByFullName('path/to/jobName')
job.nextBuildNumber = NextNumber
job.save();
You will need groovy plugin for that. Place that in an "Execute system Groovy script" step. Make sure to choose system groovy. That will execute on the master, where the job config and metadata is stored so you have access to the Jenkins internals and data.
I'd suggest you should really be using the above options rather than relying on "keeping both jobs in sync" via a script or manually. You can then pass the label to be used from the first job as a parameter to the second job. That would also require Parameterized Trigger as well as Version Number plugins.
You can even use ${BUILD_DATE_FORMATTED} or ${BUILD_TIMESTAMP}, etc.
Postdate: thinking about the problemspace from a different perspective, that of running 2+ builds on different platforms (simultaneously), there's a plugin for that: Matrix project. You can run it as a freeatyle job on multiple nodes or is excellently described as Matrix building in scripted pipeline. Not sure how that would tie in to JIRA.
We have a small java dev environment which uses gradle, Jenkins, and Git. We use an in-house built Gradle plugin to increment build numbers using a file to store the current number. The build number is baked into each build as part of its version data. The build number file is checked into the git workspace for the project.
We are now adding Jenkins to the environment for CI. Jenkins has its own build number which we can access via env var $BUILD_NUMBER.
The downside to our in-house Gradle build number plugin is that it uses a local file and thus builds by multiple developers do not sync build numbers. If we use Jenkins BUILD_NUMBER than that is completely different sequence than the Gradle build number plugin.
What is the best practice for this type of scenario?
If you state that only builds provided by your CI are valid for future usage it seems that you have to rely on Jenkins BUILD_NUMBER.
If you want Jenkins job BUILD_NUMBER to be started from specific value do the following:
Manage Jenkins -> Script Console
Jenkins.instance.getItemByFullName("YOUR_JOB_NAME").updateNextBuildNumber(YOUR_BUILD_NUMBER)
I was hoping someone could help with ideas in Jenkins for sharing a build increment across multibranch pipelines on multiple build machines. Looking through similar questions, I didn't see anything better than timestamp instead of build number but it's not quite what we were looking for.
I am using Jenkinsfiles to define multiple pipelines, then 'multibranch' to instantiate them across all branches. Currently these just call shared freestyle jobs to implement the stages. Every pipeline invokes the same job to do a build, across a pool of build machines, so we can just use that build number to increment the version. For example I have v1.2 being worked on by several branches, each having a CI, Nightly, and Release pipelines. They all invoke the same build sub-job so you might have CI/BranchA run sub-job #503 on buildVM1 so will have version string 1.2.503, then Nightly/BranchB will run sub-job #504 on buildVM2 so will have version string 1.2.504. This works great as long as I am invoking the same build sub-job.
The next step is to implement the builds as part of the Jenkinsfile pipeline, but then I lose my convenient build increment. How can I define a shared build increment for this component across all branches defining the same major.minor version?
Timestamp is a bit unwieldy since the multiple branches/teams/build machines means I need to go to seconds. Do I really need versions like 1.2.20180118165007? There's got to be a better way.
How else can I manage this?
the company where I work for is evaluating jenkins 2.71, in particular the pipeline and blue ocean plugins. We already tested also GoCD and we need, as in GoCD, a way for a pipeline to automatically fetch the artifacts from 2 other pipelines (taking the last successful result of each one of them), here our case.
We have these initial pipelines (build & run tests), which reflect 2 projects:
frontend, ~ 15 minutes
backend, ~10 minutes
I created a pipeline called configure (~1 minute), with e.g. a parameter called customer-name, which takes backend and frontend files and puts them together, then applies specific customer specific configurations and customizations and produces deployable artifacts. Instead of "customer-name" I could also parallelize this job to create all the artifacts for each customer at once, separated in different directories.
The next pipeline would be to deploy them on different test servers separated for each customer. This could be also part of the same configure pipeline, we still have to see how to put things together in jenkins...
Ideally, I need configure pipeline to be triggered automatically (or also on demand) after each frontend or backend success and take as input the last successful artifacts from these 2 pipelines, but not just having the last successful build, we need as dependency the git branch name.
E.g. we have:
backend branches:
master
release/2017.2
frontend braches:
master
release/2017.2
In the pipeline editor, I found a Build Triggers option and set it as follows: Build after other projects are built > Projects to watch: frontend, backend > Check Trigger only if build is stable or better in my test environment full of failures Trigger even if the build is unstable.
Searching further, I found Copy Artifact Plugin
But now the big question, how to fetch the last successful artifacts from these pipelines with the same git branch name?
Because we don't want to mix e.g. a backend build of "release/2017.2" with frontend "master", it has to find as the last successful build having the same relationship or parameter or whatever you wanna call it, in our case the association is the git branch name.
Is it possible to achieve this? If yes, how?
The copy artifact plugin seems to work in a freestyle project. Would it work in a pipeline? That's also a concern...
Thanks
Yes, the Copy Artifact plugin does work in both freestyle and pipeline projects; pipeline uses the copyArtifact function that I referenced in my comment. Note that if you go to the Pipeline Syntax link, it's kind of hidden: you have to first select "step: General Build Step" from the drop-down, then it will give you the Copy Artifact pipeline command builder.
I'm going to assume that your frontend and backend projects are built as multi-branch pipelines, as that would probably be easiest to maintain so that you don't have to keep creating new projects for every release. You can reference these projects from other projects by referencing <project name>/<branch name> (sometimes I've had to replace the / with %2f instead, I think mostly on freestyle projects). You could then set up your configure project as a parameterized build (either pipeline or freestyle), say with a string parameter of PROJECT_BRANCH_NAME. Then put in the following in your frontend/backend project pipeline scripts to trigger a build of your configure project
build job: 'configure', parameters: [[$class: 'StringParameterValue', name: 'PROJECT_BRANCH_NAME', value: ${env.BRANCH_NAME}]]
Then you should just be able to make your configure project reference the frontend/%PROJECT_BRANCH_NAME% and backend/%PROJECT_BRANCH_NAME% (or ${env.PROJECT_BRANCH_NAME} in a pipeline script) when copying the artifacts.
Also, is there a particular reason why you're evaluating specifically Jenkins 2.7? 2.7 is a year old now, and there have been a few new LTS releases since then. I'd recommend staying reasonably up-to-date unless you know there's a specific reason you want 2.7.
I have a project where part of the build process is to create a native library on a remote machine. This is currently a manual process outside of the CI builds made by Jenkins.
The setup in question is that the Jenkins master server build a GIT based maven project, which has a dependency to a native library which can only be built on a specific machine. Jenkins can't compile this module, and because of this, it is currently a manual process.
I would like to install a Jenkins slave on the machine that creates the native library, and returns the compiled files to the Jenkins master, without handling any other parts of the build.
I am having trouble figuring out if this is even possible. The number of articles i have found on the subject discusses Jenkins slaves as a means of distributing the build, but i want the slave to take responsibility for a small part of the build process, and nothing else. The Jenkins master should just send the build request to the slave and wait for the result, instead of trying to compile the code itself.
I do exactly the same. My setup, very similar to what Mark O'Connor and gaige are advising, and I am using the Copy Artifact plugin.
job A: produces a zip file on a Mac
job B, runs on slave B - Windows machine, takes the zip as input and produces an MSI
Here's the important part in the config of job B:
restrict the job B on the proper slave using labels
make sure job B happens after job A
make sure artifacts from job A are sent to job B before your build
build your stuff
archive artifacts produced by job B
Delegating part of a job to a slave is something that would have to be done external to Jenkins, for example, using ssh.
However, as #kan indicates, you most likely want to extract the native library build as a separate job and then have that job execute on a particular slave, or any slave that meets a specific criteria.
To do this, my suggestion would be to use Labels in the node configurations to determine which slaves can be used for building that particular job.
In Jenkins > nodes > <slave node>, use the Labels property to set one-word labels that indicate your specific requirements, such as the OS or processor type.
Then, in the jobs that are node-specific, check Restrict where this project can be run and set the Label Expression to something that meets your criteria. If the criteria is simple, it will just be a single word, if you need a boolean, you can use those as well (such as OSX&&Lion in our case).
I believe this is all in the standard version of Jenkins, without need for a special plugin. Leave me a comment if it isn't and I'll try and diagnose which plugin enables this functionality.
This is problem is solved by using a binary repository manager to centralize your software artifacts. Personally I use Nexus, but it could be something as dumb as a remote file system.
The idea is to publish the built artifact after each Jenkins job (if you don't like Nexus, you could use one of the Publish over plugins) and retrieve it as a build dependency in the next job.
This approach means it longer matters where the build executes, and has the added advantage of decoupling the build of each module component.