I have this scenario:
I have a Jenkins Job which basically performs the build and followed to that it starts deploying the artifacts into the virtual machines.
Here is my problem. At the moment I deploy artifacts one by one sequentially i.e. via ant task from the same build.xml file.
Deploy and install Artifact 1
Deploy and install Artifact 2
Deploy and install Artifact 3
Deploy and install Artifact 4
Each of these deployments take 30 mins.
So I am interested to know how I can plan to parallelize this.
As the deployment is not that resource intensive I am not interested to use slave Jenkins. So it will probably be a bit overkill.
You'll want four executors on your jenkins master. You can do this by going to manage jenkins > Manage Nodes > Master > Configure, and change the "# of executors" to 4
Then you'll want to split your jenkins job into four jobs - each calling one of your ant targets. The build flow plugin is genius, however you may not need it.
Related
Have seen the following graph representing Jenkins pipeline:
git push --> Git repo --> Jenkins CI server --> Maven build server --> Test server --> Deliver build artifacts --> Deploy
Given that is correct, I am struggling to understand how the above different(?) servers work under the hood and I need some clarification so that Jenkins procedures are not just a black box.
Jenkins CI server, Maven build server and Test server are in reality one physical server?
If the answer to the previous question is yes, these 3 servers are different logical servers?
In my understanding and my case (Java Spring project), Maven build server executes mvn install and since pom.xml contains npm install plus npm run test commands, it is Maven build server that executes the UI tests and not the Test server. Am I right?
Does the Test server execute only the back-end Java acceptance tests?
It could be one physical or virtual server server.
no
To use maven you just need to install maven plugin in jenkins. Configure which version of maven you want to use in "Manage jenkins" -> "Global tools configuration" -> add needed maven version.
Depends on which kind of tests you want to run - you can repeat steps from step 3 for the tool you need or you can install needed tools on the jenkins server manually and add it to the PATH.
At the end just use needed tools in your pipeline or other king of job in jenkins.
Without further context, we must put some points clear:
Jenkins may work on multiples nodes (could be real/virtual/pods)
Jenkins is an orchestrator... This means that it uses different tools in a order given by the pipeline. This order is pretty much: git > build > test > delivery > deploy
Testing server are used for install the software an run a variety of tests, pretty much for several teams.
Build servers are used for running commands to build software
With those clear, these are the questions:
1.- Depends on your infrastructure. Jenkins works with executors that may run on master or in nodes. If your Jenkins is just one server, then yes... It is one physical server. If you use nodes, then it is more probable than you may found one node handling the building, and another one the tests. The definite answer lies within your Jenkins and your pipeline
2.- No... Even with a Jenkins master-agent infrastructure, the building and testing occurs within Jenkins.
3.- Depends on your concept of test server. If you define it as a server where you use as a target for testing purposes, then the answer is no... If you define a test server as a machine that executes test, then the answer is yes.
4.- Depends on what you type or test do you have automatized. You can run unit tests, regression tests, smoke test, etc. For example, you may have some unit test for your back-end, and have some test in karma for your UI. Again, in your case you must check on your pipeline, and the code to check what kind of tests are you running.
Is the anyway to make a Jenkins Multibranch Pipeline better support incremental builds by preferring the last build node when it build a branch instead of choosing an available node more or less randomly?
Details:
We are setting up a Jenkins multibranch pipeline for a large Git project, where we use Make to build and test a lot of code. A full build takes 6-8 hours, but the dependency tracking in Make is good enough for us to use incremental builds, shortening most our build times a lot. For this to work, Jenkins have to pick the same workspace for changes to the same branch again. Luckily it does so - but only on the same build node.
We have some identical Jenkins slave nodes available. Each time a build job is started due to a change on a branch in Git, Jenkins apparently pick a random free build node with a fresh, clean workspace meaning no incremental build speedup.
We have tried to build via NFS, such all the build nodes can share the workspaces, but at least the NFS server we have available is way too slow to make this work.
Is there anyway to make Jenkins choose the node a little less randomly and prefer the latest node on which the branch was build the last time?
I have configured a multibranch-pipeline project in Jenkins. This project run integration test on all my feature branches (git). For each job in the pipeline project it creates an instance of my webapp (start tomcat and other dependencies). Because of port binding issues this result in many broken jobs.
Can I throttle the builds in the multibranch-pipeline project, so that the jobs for each feature branch run sequentially instead of parallel?
Or if there any more elegant solution?
Edit:
Situation and problem:
I want to have a multibranch pipeline project in Jenkins (because I have many feature branches in git)
The jobs which are created from the multibranch pipeline (for each feature branch in git), run in parallel
Polling scm is at midnight (commits on x branches are new, so the related jobs started at midnight)
every job started one instance of my webapp (and other dependencies) which bind to some ports
The problem is, that there can start many of these jobs at midnight. Every job will try to start an instance of my webapp. The first job can start the webapp without any problem. The second job cannot start the webapp because the ports are already taken from the first instance.
I don't want to configure a new port binding for each feature branch in my git repository. I need a solution to throttle the builds in the multibranch pipeline, so that only on "feature" can run concurrently.
From what I've read in other answers the disableConcurrentBuilds command only prevents multiple builds on the same branch.
If you want only one build running at a time, period, go to your Nodes/Build Executor configuration for the specific VM that your app is running on, drop the number of executors to 1 and configure the node labels so that only jobs from your multibranch pipeline can run on that VM.
My project has strict memory, licensing and storage constraints, so with this setup, all the jobs on the master and feature branches start, but only one can run at a time until the executor becomes available.
The most elegant solution would be to make your Integration Tests to be able to run concurrently.
One solution would be to use an embedded tomcat with a dynamic port. In that way each job instance would run in tomcat with different ports.
This is also a better solution than relying on an external server.
If this is too much work, you can always use the following code in your "jenkinsfile" pipeline:
node {
// This limits build concurrency to 1 per branch
properties([disableConcurrentBuilds()])
// continue your pipeline ...
}
The solution comes from this SO answer.
I would like to raise from scratch a build server for .NET applications using Jenkins, please note that i'm new to Jenkins CI.
Several Questions:
1) How should I decide on the build server specs? except for the OS which would be windows server 2012, how should I decide on the RAM and the CPU and HD space?
2) Should the Jenkins sit in the build machine or not, what is the recommended approach? I understood that the build server should be isolated from the Jenkins master
3) How do I decide on the Master/Slave approach, when should I use only Master and when should I use master and slave or slaves?
4) How would you recommend me to run the build and deployment tasks in the Jenkins CI, using NAnt/Python or any other scripting language ?
10x, and sorry for the igonrance :)
Responding to each in turn:
You can run Jenkins as a windows service (instructions here) and the machine can be a VM, so it doesn't have to be huge.
a) RAM and CPU: I'll put these together and will depend on how many jobs you plan to have running at the same time. The default number of build executors is 3 but can be increased as a global config change.
b) HDD: This depends on how many jobs you plan to have. Jenkins will checkout the source code (as well as the compiled output) to its home directory on a per job basis. This can get big. I would also recommend using the ThinBackup plugin to backup the Jenkins configuration.
Jenkins is the build machine. A vanilla installation of Jenkins is the master. In my experience you will not need a separate slave machine unless you're needing to do native builds on other platforms or have LOTS and LOTS of jobs. I've seen single masters running happily with hundreds of jobs.
Further to 2. above, suggest you start with a master and set up a slave later if you really need one.
As you have stated you are building .NET applications, you can simply install the MSBuild plugin which should serve you well. Builds for .NET applications in Jenkins are Freestyle builds so you will be using Windows Batch build steps often as well. This also is a great blog on Jenkins in a .NET environment.
Is it possible to iterate on slaves with a particular label in a jenkins job?
For example, lets say I have few slave with label "redhat". I have a job in which I want to logically do something like:
for slave in slave_list_with_label_redhat do
ssh someuser#${slave.hostname}
done
Thanks in advance!!
Edit use case in detail:
So this is to workaround a bug in jenkins where archiving artifacts fails from AIX slave. https://issues.jenkins-ci.org/browse/JENKINS-13614
So what we really want to do is once the build is complete on build slave, we would like to scp "build files" to available jenkins aix slaves and install and run few tests on test slaves.
You might find https://wiki.jenkins-ci.org/display/JENKINS/Elastic+Axis to fit your needs
I think this is used as part of the multi-configuration (Axis) create job and you can then select this as one of the axes
I usually let my jobs download all necessary artifacts. So your option would be to add another build step to your build job and archive the artifacts manually outside of Jenkins. You can use SCM tools (not perfect for binary artifacts but offer great auditing features) or binary artifact repositories (e.g. nexus - best, but I have no Idea about the auditing features) or just a remote filesystem (you can mount it locally on your slave machines). The build job just needs to pass some information to the test jobs so that these can get the right artifact. This way, Jenkins still decides where the test job(s) will run but the slaves have always access to the artifacts.