I am relatively new to Jenkins.
I created a declarative pipeline in Jenkins where users are asked to enter their branch name and then Jenkins builds that specific branch (for example, origin/mybranch).
This allows me to run a quick set of tests for specific branches.
The developers can run the pipeline multiple times and today I block multiple such pipelines from running simultaneously because if they do, one overwrites the other.
This happens because the first pipeline writes to c:\Jenkins\workspace\QuickBuild and when another such job run is writes to that exact same folder, killing the original run.
Blocking was the solution I found to prevent this but I would like it so that when one run is finishing up (using less than 8 cores) the next run in queue will already start running with whatever cores are freed up.
I would have though this would be a basic concept of Jenkins.
Am I missing something? Am I doing it wrong?
Following MaratC and Zett42's suggestions, I ended up adding this to my script:
agent
{
node {
customWorkspace "${params.Branch}"
}
}
This causes Jenkins to create each build in a different folder and they don't step on each others' toes.
The only downside is that you can't build the same branch simultaneously but that's a corner case.
Also, I could add a random number to the workspace to enable this as well.
Related
Following the Jenkins Best Practices, I want to avoid that Build Jobs/Pipelines could be executed into my Jenkins Master.
To do so, I've installed the Job Restrictions Plugin, using it to configure the Master to run only some Maintenance Pipelines.
The problem is that now Build Pipelines that are configured to run on specific Agents, are not executed anymore. I see that the Build Queue continuously grows, and the Pipelines are not runned. I think that this behaviour could be related to Flyweight Executors of the Master.
So, the question is the following: How can I execute on Master just a little subset of Maintenance Pipelines and, in the mean time, execute Build Pipelines only on specific Agent?
You can configure the master node to only be used when explicitly named. Just click the master node > go to configure and change Use this node as much as possible to Only build jobs with label expressions matching this node
I found the solution that perfectly fits with my needs, here.
To quickly sum up the solution, I was to able to exclude all the user Builds from Master and run on it only the Jobs/Pipelines of a specific Jenkins folder (IuA in my case), configuring the Job Restrictions Plugin in the following way:
In order to better understand the logic behind this solution, I recommend you to give a look at link that I posted above.
I have Multi branch build Jenkins pipeline which uses docker containers. The issue I am having is I don't want more than one branch to triggered the build at a time. Because I use postgres DB in a container and when more than one branch starts localhost port 5432 gets occupied with branch build which kicks in first then second branch fails.
Is there a way to avoid this in Jenkinsfile or any other way ?
pipeline {
options { lock resource: 'build-lock' }
stages {...}
}
use this in your pipelines. At any given point in time, only one instance of the pipeline will execute, even in a multi-branch pipeline.
For more info Lockable resources
I would perhaps tackle your postgres db rather than try and solve it this way. Could your build pick a random port and spin up the required db on alt ports instead?
If you do want to try limit concurrent builds...
1) You could limit the agent this repo runs on and provide it only 1 executor. This would cause builds to queue whilst they waited
2) If you wanted to do it programmatically, you would need to put in a check in the pipeline to abort/wait the build if it finds current executions that match.. I don't recommend this, if your running in sandbox you will likely need to approve script access. Plus it seems like you would be digging under the hood and it might cause issues with upgrade paths if the interface gets refactored... but you would be digging around in https://javadoc.jenkins-ci.org/hudson/model/Executor.html getCurrentExecutable() or perhaps something like this https://github.com/cloudbees/jenkins-scripts/blob/master/get-build-information.groovy#L24
I have configured a multibranch-pipeline project in Jenkins. This project run integration test on all my feature branches (git). For each job in the pipeline project it creates an instance of my webapp (start tomcat and other dependencies). Because of port binding issues this result in many broken jobs.
Can I throttle the builds in the multibranch-pipeline project, so that the jobs for each feature branch run sequentially instead of parallel?
Or if there any more elegant solution?
Edit:
Situation and problem:
I want to have a multibranch pipeline project in Jenkins (because I have many feature branches in git)
The jobs which are created from the multibranch pipeline (for each feature branch in git), run in parallel
Polling scm is at midnight (commits on x branches are new, so the related jobs started at midnight)
every job started one instance of my webapp (and other dependencies) which bind to some ports
The problem is, that there can start many of these jobs at midnight. Every job will try to start an instance of my webapp. The first job can start the webapp without any problem. The second job cannot start the webapp because the ports are already taken from the first instance.
I don't want to configure a new port binding for each feature branch in my git repository. I need a solution to throttle the builds in the multibranch pipeline, so that only on "feature" can run concurrently.
From what I've read in other answers the disableConcurrentBuilds command only prevents multiple builds on the same branch.
If you want only one build running at a time, period, go to your Nodes/Build Executor configuration for the specific VM that your app is running on, drop the number of executors to 1 and configure the node labels so that only jobs from your multibranch pipeline can run on that VM.
My project has strict memory, licensing and storage constraints, so with this setup, all the jobs on the master and feature branches start, but only one can run at a time until the executor becomes available.
The most elegant solution would be to make your Integration Tests to be able to run concurrently.
One solution would be to use an embedded tomcat with a dynamic port. In that way each job instance would run in tomcat with different ports.
This is also a better solution than relying on an external server.
If this is too much work, you can always use the following code in your "jenkinsfile" pipeline:
node {
// This limits build concurrency to 1 per branch
properties([disableConcurrentBuilds()])
// continue your pipeline ...
}
The solution comes from this SO answer.
Use case: Using Jenkinsfile to auto create builds for branches
Summary:
For a variety of reasons sometimes the Jenkins master fails to connect to the SCM server. When this occurs Jenkins deletes that job directory on master, because it no longer sees the branches. However, the slaves are not cleaned up and so they still have the old workspace paths (which are uniquely named based on the build # in my setup). When the Jenkins master reconnects to the SCM server, it recreates a new job folder on master, and the build counter is reset to #1.
This creates the following issues:
When a build starts, it executes on a slave. Since master has a new counter the job is #1. But this path may already exist from a previous build on that slave, so the artifact is built with content that was checked out for the original old build (i.e. maven uses the /target directory inside the workspace which already existed from previous build). So the end result is an artifact that potentially has the wrong code.
This can create build storms. After the connection issues are resolved, Jenkins will see all the repositories and branches with Jenkinsfiles and start to build them. So in a setup of let's say 20 repositories with 10 branches each, this will create 200 new builds. This increases with additional repositories and branches. This is obviously not desired.
Solutions:
One quick solution I can think of is to update the Jenkinsfile to delete the workspace if it exists before running the job inside of it. But this is just a work around. I would not want to mask the connection issues and would like to retain the actual build history of a pipeline (not have it keep erasing itself).
Minimize connection issues. This obviously will not always be guaranteed though. Plus sometimes maintenance must force servers offline. While I can construct maintenance in a way to limit or work around such issues, there still will be rare cases where downtime is required across the board. It would be best if Jenkins could handle this use case.
I'm curious if anyone has ran into this issue and what the thoughts are on this problem?
I have Jenkins setup of 6 Slaves and master, all windows machines. Now I have a housekeeping Jenkins job which I want to periodically run on all the slaves and master, as this job does following tasks
Delete unused temporary files.
Delete unwanted processes, as some of the tests are leaking processes (why leak is different question).
Set certain environment variables, as sometimes I want to push environment variable changes to all machines.
Any idea how can I force Jenkins to run this one job on all slaves and master once every day? As a work around I can create multiple Jenkins job and mark each one to run on one particular slave or master, but I would rather avoid having so many duplicate jobs.
The Node and Label Parameter plugin allows you to parameterize where a job should be run. The job can be run on more than one node -- each node shows up as a separate execution in the job's build history. When multiple nodes are selected, you can configure whether the job should continue to run on other nodes if an execution fails.
I had a similar need, but using the Node and Label Parameter Plugin didn't seem quite right, as I do not want to parameterize my cleanup jobs.
I found a more satisfying answer in this post and thought it would also benefit to this question: Jenkins - Running a single job in master as well as slave.
Here is some documentation on how to configure a "Matrix project": https://wiki.jenkins.io/display/JENKINS/Building+a+matrix+project.
What you are looking for is the "Slave axis". It's not very well documented in the page above, but it appears as an option of the "Add axis" menu whenever there are more than one node. Here's a screenshot of the interesting part:
Updates according to recent Jenkins
Pipeline type:
On "Configuration" page for pipeline: