I need to get some advice about the way to control the way Jenkins Slaves are used / Jobs are being triggered.
Background / Constraints:
I have a sequence of 10 jobs that run one after each other using the "Trigger parameterized build on other projects" option Parameterized Trigger Plugin.
Each Build of these jobs must run on the same node (I am doing it by using "Build on the same node" which is also configured in the parameterized build plugin and comes from the NodeLabel Plugin).
I have 5 Slaves (current number of executors per slave is 1 but i am open for suggestions here...)
Once Slave is occupied by a Build Sequence, no other job can run on it. When I had only 1 slave, the way I enforced it was using the "Block build when downstream project is building"
The way I configured the slave to be chosen when the first Job is triggered is one of the following: (None of them solved my problem)
a. Using the "Restrict where this project can be run" and put there a label that all relevant slaves will point to.
b. Using the option of "This build is parameterized" (Parameterized Trigger Plugin) and then add a "Node" parameter with the list of Slaves that the User can choose from.
What I want to achieve?
When a User triger the build of the First Job in the Build Sequence, this Build will be done on once of the idle Slaves. (I mean a slave that is doing nothing at the moment)
If there are no idle slaves, then it will join a queue of one of them (doesn't matter which)
Any suggestions how to solve it?
Thanks!
Try passing ${NODE_NAME} as a NodeLabel as a post-build trigger to the downstream jobs. If that works, you may need to pass it every job.
Try node-label parameter plugin. you can make Jobs to run slave node which is free at that movement
Related
Let's say I have a job A and also a agent configured. I want to run build 1 of Job A on master and build 2 of Job A on agent node.
Is there an option to achieve that ?
OR
Is there a way where my job looks at controller and if it already finds a build running, then start the next build on agent ?
Are you intending to run in parallel or just alternate? (Not a good idea to run jobs on master; could configure a node to run on same host as "master".). Seems to be parallel and you have restricted to one executor each on master and agent (you can have more, in which case any advice may be moot).
Nevertheless, Jenkins queue job allocation to executors is "sticky"; it tries to run where last run, unless unavailable. This can lead to overloading in nodes. So the M,A,M,A pattern is unnatural.
There are plugins that might help: Least Load, Scoring Load Balancer, but maybe not.
Perhaps an approach would be to restrict your job using a label and have a post-build groovy step that moves the label to the other upon success for the next run or two labels and the job self-modifies the label to match the other.
I am new in Jenkins and looking for an easy solution:
I have about 10 jobs and about 4 nodes and would like to run all the jobs on different nodes. I want the jobs to run simultaneously, when a node is free then a job that didn't run yet should run on it.
What I did is that I gave a label to each node and on the jobs I set to run on label. I added a job X that not do anything, and in the jobs that I want to run I set to run after the job X finished. This solved the issue.
My problem now is I want to send report in mail after all the jobs are done but I dont have a way knowing when they are done. I tried using pipeline with stages, but even with pipeline it does only trigger the jobs but I cannot know when they are done. Any suggestions?
You can use this plugin to do that. After you add that plugin to Jenkins just do a "New Item -> Multijob Project" from Jenkins home and configure the jobs to run parallel and then you can send mail using this plugin
I have Jenkins installed on my computer which configure to have just the master as a node (no other nodes) and number of executes is 5. I created a Job named "myJob" and I want to run it on the master 2 times at the simultaneously (meaning that if I run Builds 90 and 91, I don't want to get the message that "pending—Build #90 is already in progress").
I also have Throttle Concurrent Builds plugin installed and it allows this job to run more than once at the same time..
I still getting the "pending" message.
Can anyone tell me how to achieve it?
By default, multiple builds of the same job cannot run at the same time — that's why it remains in the queue with the "already in progress" message.
You can, however, change this behaviour in a job's configuration by enabling the "Execute concurrent builds if necessary" checkbox.
Note that, if you have multiple build nodes, the two concurrent builds of the job will likely be spread out to run on different nodes, rather than occupying both executors of the same build node.
If you require that they both run on the same machine, you should likely use the "Restrict where this job can run" option to ensure all builds of that job run on the same node.
You also need to make sure the salve is set for running more than one job.
To set your note to run more than one job:
Manage Jenkins -> Manage Nodes -> Configure Node -> # of executors
Has to set to 2 or whatever jobs you want to execute simultaneously.
As previously mentioned, you need to check the "Execute concurrent builds" box in the job's config screen.
You will then get a "job\workspace" and a "job1\workspace" (or something similar.) Since both jobs need their own workspace, the second instance will need to create a new workspace folder structure.
Just something to keep in mind if you expected both jobs to be running in the same directory.
I have a Jenkins server with half a dozen builds. Each of these builds is composed of a parent job that triggers anywhere between 4 and 6 smaller jobs that run in parallel. I use the EC2 plugin to keep the number of active slaves in line with the number of queued builds. Or in other words, slaves are coming and going all the time. Each slave has 7 executors (parent job + max(4, 6)).
It is absolutely crucial that all jobs of a build are executed on the same machine. I also cannot allow any jobs from build A to execute on a machine that has jobs from build B running.
What I'm looking for is a way that prevents Jenkins from using any inactive executors of a node as long as any jobs from a previous build are still active on it.
I've spent the day experimenting with a combination of the Throttle Concurrent Builds Plugin and the NodeLabel Parameter Plugin. Unfortunately, there seems to be a bug somewhere that causes throttled builds to not contribute to the Load Statistics of a slave. This in turn means that throttled build will never trigger Jenkins to spin up additional slaves. Obviously this is totally unacceptable.
You can try and use "This build is parameterized"
and pass the $NODE_NAME as a parameter between the builds and then use it at the "Restrict where this project can be run"
I need to build and test on multiple configurations: linux, osx and
solaris. I have slave nodes labeled "linux", "osx" and "solaris". On
each configuration, I want to (a) build (b) run smoke tests
(c) if smoke tests pass, then run full tests, and perhaps more.
I thought that multi-configuration jobs might be the answer, so I setup a
multi-configuration build job and it starts concurrent builds on each
OS. The build job will trigger a downstream smoke-test build, which, in
turn, triggers the full-test job.
I've run into the following issues
If one of the configurations fails, the job as a whole fails, and
Jenkins will not fire any downstream jobs (e.g., if the solaris build
fails, Jenkins will not run smoke tests or full tests for osx and
linux).
The solaris build takes about twice as long as the others (on the
order of an hour), and I'd prefer the linux and osx smoke tests not
wait for the solaris build to finish.
Does that mean I'm left with hand-crafting three pipelines of jobs, and
putting them behind a "start-all" job (i.e., creating and hand-chaining
the following jobs)?
build-linux smoke-test-linux full-test-linux
build-osx smoke-test-osx full-test-osx
build-solaris smoke-test-solaris full-test-solaris
Did I miss something obvious?
As far as I know the answer is to create 3 matrix jobs, one for each system. They then would have 3 subjobs (build, smoke-test, fulltest) with the build-job as a touchstone.
Have you thought about combining the build, smoke-test and full tests into a single multi-configuration job? Other than being a little messy, this should work for you.
To answer your first issue: to trigger a downstream job regardless of result, use trigger parameterized build to run when complete (always trigger) and then check "build w/o parameters"
To answer your second issue: either use an all encompassing multi-configuration (matrix) job or use three separate job streams as you mentioned. UPDATE: you could run 3 sequential matrix jobs for each step (build, smoke-test, full tests) but it would mean that if any of the build steps failed then none of the smoke-tests would be run.