I have multiple jobs set on Jenkins. There are only one executor on masters, there are many other slaves. I want to create a job which would run on a separate job queue or a separate executor concurrently. How can I achieve this in simplest way? Can it be achieved without modifying slaves?
The next thing is how can I achieve running on any new build for this job in parallel. Rest of the jobs set should not be disturbed and interfered.
As I can understand you need to configure a particular job to run on a slave node...
In order to achieve that you can 1st add a node to the Jenkins master. There you have a place to add a label to that slave node. The in the pipeline script of your job you can edit it as follows.
pipeline {
agent {
node {
label '<YOUR SLAVE NODE LABEL>'
}
}
}
In the latter part of the question, you have asked to run parallel jobs. Please refer to the parallel job documentation If you need to add sequential stages refer to the sequential stage documentation If you want to create a dynamic parallel job this link would help you.
Related
I am trying to setup various Jenkins pipelines whose last stage is always to run some acceptance tests. To cut a long story short, acceptance tests and test data (much of which is shared) for all products are checked into the same repository which is about 0.5 GB in size. It therefore seemed best to have a separate job for the acceptance tests and trigger it with a "build" step from each pipeline with the appropriate arguments to run the relevant tests. (It is also sometimes useful to rerun these tests without rebuilding the product)
stage('AcceptanceTest') {
steps {
build job: 'run-tests', parameters: ..., wait: true
}
}
So far I have seen that I can either:
trigger the job as normal. But this uses an extra agent/executor,
there doesn't seem to be a way to tell it to reuse the one from the
build (main pipeline). Both pipelines start with "agent { label 'master' }" but that
seems to mean "allocate a new agent on a node matching master".
trigger the job with the "wait: false" argument. This doesn't
block an executor but it does mean I can't report the results of the
tests in the main pipeline. It gives the impression that the test
stage has always succeeded.
Is there a better way?
I seem to have solved this, by adding "agent none" at the top of my main pipeline and moving "agent { label 'master' }" into the build stage. I can then leave my 'AcceptanceTest' stage without an agent and define it in the 'run-tests' job as before. I was under the impression from the docs that if you put agents in stages then all stages needed to have one, but it seems not to be the case. Which is lucky for this usecase...
I don't think that there's another way for declarative pipeline.
On the other hand for scripted pipeline you could execute this outside of node {} and it would just hold onto one executor on master releasing the one on slave.
stage("some") {
build job: 'test'
node {
...
Related question: Jenkis - Trigger another pipeline job in same machine - without creating new "Executor"
I am trying to run a Jenkins job on a slave. An excel file is created as a result of this first job.
I want to run a second parametrized job on the master after the first job is completed depending on the value from the excel.
I have tried the following options till now:
1. Using the Join Plugin. This doesn't work because the second job is parametrized and I have to take an input from the excel file. there is no option to provide options or read the parameter from a file.
2. Pipeline on master- For some reason when I create a pipeline on the master and execute the first slave job, the slave job waits for a slot to run since one job is already running and the main job is waiting for the job on the slave to run. So it results in a deadlock.
Pipeline (scripted, not declarative) sounds like the way to go.
Something like:
node('MySlaveLabel') {
...do your stuff here...
stash includes: 'myExcelFile.xls', name: 'myExcelFile'
}
node('MyMasterLabel') {
unstash 'myExcelFile'
...examine your Excel file here..
...add conditional statements...
}
As long as the node blocks are not nested, you will need only 1 executor on the slave and 1 on the master.
If for some reason you actually need the jobs to call each other:
Use the build 'anotherProject' syntax.
Make sure you have enough executors on the slave.
I have setup a slave.
For a job that executes a shell script, I configured to run either on the slave and on the master.
If I launch 2 instances of the same job, I observe that the job is run only by the master, and the 2nd instance waits to the 1st one to finish, and it will be run also by the master.
I expect master and slave to work simultaenously.
Why is the slave always idle?
Is there a way to priorize one slave?
UPDATE: In my use case, a job uses the database for destructive tests, so is not good for reliability to have more than one instance of the same job in a node. Each node has a copy of the database.
First, go to the job configuration page and check "Execute concurrent builds if necessary". This will allow multiple instances of your job to execute at the same time.
Next, go to the configuration pages of your build nodes (via the link "Build Executor Status" on the main page) and set "# of executors" to 1 for each one (both master and slave) . This will prevent one build node from running multiple jobs at the same time.
The result should be that if you launch 2 instances of the same job, one will execute on the master and one will execute on the slave.
The solution with a jenkins pipeline script:
node("master") {
parallel (
"masterbuild" : {
node ("master") {
mybuild()
}
},
"slavebuild" : {
node ("slave") {
mybuild()
}
}
)
}
def mybuild() {
sh 'echo build on `hostname`'
}
This is an improvement on Wim answer:
Go to the job configuration page and check "Execute concurrent builds if necessary". This will allow multiple instances of your job to execute at the same time.
Next, use the Throttle Concurrent Builds Plug-in.
In this way, only one execution per node is allowed, and the load is balanced between different nodes.
In this way, a node doesn't loose the ability to run simultaneously several unrelated jobs.
I'm new to Jenkins and I'm trying to setup a project which will use few build executors.
The flow shall be as follows:
two build executors with webservice label return their IP addresses and wait for the third build executor to finish its job
third build executor with tester label collects those IP addresses and performs some long running job (e.x. sends HTTP requests to the webservices deployed on those two agents)
How to achieve such behavior in Jenkins?
I've found that when an build executor finishes its job it is immediately released and I don't know how to make it wait for other build executors to finish their jobs.
Edit:
I forgot to mention that I want the build executors with the webservice label to be reserved (not available for other jobs) till the build executor with the tester label will finish its long-running job.
Also all these build executors should be on separate slaves each. That means each slave has only one build executor.
I've finally managed to do this using Pipeline and below script:
node('webservice') {
def firstHostname = getHostname()
node('webservice') {
def secondHostname = getHostname()
node('tester') {
println 'Running tests against ' + firstHostname + ' and ' + secondHostname
// ...
}
}
}
def getHostname() {
sh 'hostname > output'
readFile('output').trim()
}
It acquires two build executors with webservice label. I'm getting their hostnames (I'm using them instead of the IP addresses) and pass them to the build executor with a tester label. Finally the tester runs some long-running tests.
Those two webservice build executors are blocked till the tester finishes its job, and no other project may use them during that time.
As Alex O mentioned, you can configure the master and slave relationship between the projects /executors inside the Jenkins projects /executors. There is option for that, "Build Triggers" -> Build after other projects are built
or use plugin to achieve it
https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin
or
https://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin
What you actually want is probably that your job uses three slaves at the same time.
Re-thinking the setup in that way, it won't be necessary to consider the collection of IPs and the subsequent usage of the slaves as three different steps that must be aligned in some way.
Unfortunately, Jenkins does not support using multiple slaves for one build out-of-the box, but it will be possible to achieve what you want e.g. using the Multijob plugin and the Join plugin that Aaron mentioned already.
See also this question for information on how to use two slaves at the same time.
I have many Jenkins Jobs that I need to run on every Build,
At present time I have 4 slave servers.
I would like the jobs to run in parallel as much as possible, hence I defined the jobs as follow:
Execute concurrent builds if necessary - Disabled
Restrict where this project can be run - Enabled with the following values SalveLinux1HT||SalveLinux2HT||SalveLinux3HT||SalveLinux4HT
To my understanding if Job A and B are triggered at the same time, one should use 1HT and the other should use 2HT and they can run in parallel
however Jenkins build job A on all 4 slaves and only after it's finished he will build job B on all 4 slaves
This is the opposite of my goal
Any ideas?
Thanks in advance
You can use
Build Flow Plugin
You can find both installation and configuration instructions of this plugin at the above mentioned link.
If you want to run any jobs in parallel you can use following scripts:
parallel (
// job A and B will be scheduled in parallel.
{ build("jobA") },
{ build("jobB") }
)
// jobC will be triggered after jobs A and B are completed
build("jobC")