Using Jenkins or Hudson I would like to create a pipeline of builds with fork and join points, for example:
job A
/ \
job B job C
| |
job D |
\ /
job E
I would like to create arbitrary series-parallel graphs like this and leave Jenkins the scheduling freedom to execute B/D and C in parallel whenever a slave is available.
The Join Plugin immediately joins after B has executed. The Build Pipeline Plugin does not support fork/join points. Not sure if this is possible with the Throttle Concurrent Builds Plugin (or deprecated Locks & Latches Plugin); if so I could not figure out how. One solution could be to specify build dependencies with Apache Ivy and use the Ivy Plugin. However, my jobs are all Makefile C/C++/shell script jobs and I have no experience with Ivy to verify if this is possible.
What is the best way to specify parallel jobs and their dependencies in Jenkins?
There is a Build Flow plugin that meets this very need. It defines a DSL for specifying parallel jobs. Your example might be written like this:
build("job A")
parallel (
{
build("job B")
build("job D")
},
{
build("job C")
}
)
build("job E")
I just found it and it is exactly what I was looking for.
There is one solution that might work for you. It requires that all builds start with a single Job and end with a definite series of jobs at the end of each chain; in your diagram, "job A" would be the starting job, jobs C and D would be the terminating jobs.
Have Job A create a fingerprinted file. Job A can then start multiple chains of builds, B/D and C in this example. Also on Job A, add a promotion via the Promotions Plugin, whose criteria is the successful completion of the successive jobs - in this case, C and D. As part of the promotion, include a trigger of the final job, in your case Job E. This can be done with the Parameterized Trigger Plugin. Then, make sure that each of the jobs you list in the promotion criteria also fingerprint the same file and get the same fingerprint; I use the Copy Artifact Plugin to ensure I get the exact same file each time.
Related
I got a WS variable which is saving the Jenkins WORKSPACE variable.
During stages I always check if the WS == WORKSPACE Jenkins generic variable.
What I saw that during parallel runs ( when 2 different workspaces are created inside C:/jenkins/workspace#1 and C:/jenkins/workspace#2 the $WORKSPACE is swapping between those 2 parallels builds.
The problem is reproduced rarely , less than 10 percent of cases but I find it quite strange , for the picture above -> The first workspace its AVB_Aplicattions_BOSCH-3 its going trough 3 stages and in the 4th stages the $WORKSPACE variable its swapping with AVB_APLICATTIONS_BOSCH-4( the other parallel build).If I will look on the other build (AVB_APLICATTIONS_BOSCH-4 I will see the same problem-> the workspace is becoming AVB_APLICATTIONS_BOSCH-3).
During this builds I compile using cmake files, I'm afraid that the results will be not correct.
I was thinking to use the in build dir() function during each stage to be sure im on the same workspace .
Some one know a good approach for this case and also why its happening?
I don't want to deactivate parallel's builds.
The problem is that Jenkins variable scopes aren't really safe with parallel stages (see further sources / link). There are actually several questions and answers (and workaround) how to work with variables when using parallel stages. Long story short: don't use global variables but pay extra attention to define local variables with def to limit their scope:
Jenkins - matrix jobs - variables on different slaves overwrite each other?
Variables get overwritten in parallel stages
Parallel matrix and global variables without race condition?
Jenkins community links:
https://issues.jenkins.io/browse/JENKINS-55040
https://issues.jenkins.io/browse/JENKINS-54732
I am trying to run two separated Pipelines from one DataFlow Job similar to below question:
Parallel pipeline inside one Dataflow Job
If we run two separated Pipelines using Single DataFlow job using single p.run() like below:
(
p | 'Do one thing' >> beam.Create(List1)
)
(
p | 'Do second thing' >> beam.Create(List2)
)
result = p.run()
result.wait_until_finish()
I think it will launch two separate pipelines in single Dataflow job but will it create two Bundles and will it run on two different workers?
You are correct: These two sections of a graph will run under the same Dataflow job.
Dataflow paralelizes execution of work items dynamically. This means that depending on your pipeline, the amount of work that it has, etc., Dataflow may decide to schedule on the same worker or on separate workers.
Normally, if there's enough work on both parallel pipelines (~20s on each), then yes: They will run in parallel.
use case: I want to run the regression suite on multiple images(image selection is variable it can be 4 or 5).
I want to create 1 master job who will take the images name as an input, and This master job passes the image name one by one to downstream regression job. The number of images can be variable.
Master job
INPUT image: a,b,c .....
|
|
-------------------------------------------------
| | |
Regression job Regression REgression
Input image: a Image b Image c
Can anyone just tell me how can I perform this task in Jenkins?
To Solve this, I have used the pipeline and Active choice parameter plugins.
Here is the configuration:
Here is the problem, I am getting ThunderImage list as [p,1,p,2,p,3] instead of ['p1','p2','p3'] .
So,you want to select which regression Job starts depending on the selected input from your Master Job? For this you could use the Post build task and use it's regex functionality to check for the input Paramter inside the Build Log.
If you are using a Pipeline job you can use the solution of Christopher Orr from: Jenkins trigger build dependent on build parameters
We have multiple SCM jobs. Job A triggers Job B and Job C triggers Job D. While job C running if there is a checkin in Job A, after Job C is completed instead of triggering Job D , Job A is triggered and Job D is in queue. Once Job A is complete then Job D is triggered while Job B is in queue. Is this a bug ? would expect Job C to trigger Job D regardless of any SCM change in upstream job. How do you solve this problem?
Yes this is expected, assuming that you only have one executor. First some terminology:
Queue, this is the place where a build is before it's executed, usually because all executors are busy.
Triggered, means that a a build of a job is added to the queue.
Executor, Takes the first build in the queue and builds it.
Now let's do a timeline:
C is triggered, adds build C#1 to the queue and the build starts right away since there is nothing executing
A is triggered and adds build A1 to the queue, since the only executor is busy building C#1
Build C#1 is nearing its completion and at the end it triggers a build of D, build D#1. Build D#1 is put on the queue right after build A#1
Once C#1 finishes, the executor takes the next item in the queue, which happens to be A#1 (since it's been there the longest).
A#1 finishes as adds B#1 to the queue, right after D#1
D#1 is built since it is the first in the queue
Finally B#1 is built
So as you can se, the executor always takes the build that has been in the queue the longest. There are ways you can alter this priority, for example using the Priority Sorter Plugin, it allows you to set higher priority for certain jobs, in your case you should put higher priority on job B and D.
Jenkins newbie, but have other build-server experience.
I'm in progress of setting up a test job, where software on two nodes need to ping-pong with each other.
I have a pool of labeled nodes (lets call them A, running windows 7) to run the testing software and another pool of labeled nodes (lets call these B, running lubuntu 14.10).
The testing is performed through TCP/IP and needs various command line stimuli on both A and B nodes throughout the test. In the end I need to gather artifacts from both the A and B nodes.
I imagine the requirement and need to control multiple nodes simultaneous isn't so rare, but I'm really having a hard time locating info on this on the web.
Might there be a plugin for this that I've missed?
Below are my thoughts of what needs to be performer, should a single plugin not exist to help me out.
My preferred solution would be a single job, but then I need to find out how to perform following:
Check out from SVN to Node A.
Check out from SVN to Node B.
Execute Windows script on Node A.
Execute Linux script on Node B.
Collect artifact from Node A.
Collect artifact from Node B.
Alternative to all the even bullets above, might be to perform those actions using SSH from either the master or the A node to control the B Node, but that leaves the following questions:
How to select one B node out of the B node pool - and mark it in use?
How to use the Jenkins SSH/slave credentials?
A totally different alternative could be to set up two jobs, one for Node A's and one for Node B's. But then I need to find out how to perform the following:
Associate one Node A job with a Node B job, so they are both aware of the association.
Perform two-ways inter-communication, allowing the Node A job to wait for a signal from a Node B job and visa verse.
Eagerly looking forward to your answers!
In a similar scenario we use, we have two jobs (I'm not aware of a way to run a single job on two nodes).
Job A runs on node A and sets up a server and then triggers job B, which is set to run on node B (as a build step).
Job B will start its client app, which is configured to work with the server installed by A (an IP configuration in my case).
The job A (server) goes into a wait loop (using bash while) that checks if client running on B has connected (I use a flag file in a shared location).
Once connected, both jobs do a few more steps and complete. Each job will end with its own reporting.
I hope this helps.
In my case I've used Multi-configuration project with Slave axis. Then you can synchronize execution between nodes using your own code, or mix it with (system) Groovy script to communicate between different configurations.
ie.
def siblings = []
build.getParentBuild().getExactRuns().each { run ->
if (!build.is(run)) {
if (run.hasntStartedYet()) {
println "Build " + name + " hasn't started yet!"
} else {
if (!run.isBuilding()) {
println "Build " + name + " has already completed!"
break
}
executor = run.getExecutor()
if (executor) {
hostname = executor.getOwner().getHostName()
// In case hostname could not be found explicitly,
// then set it based on node's name.
if (!hostname) hostname = executor.getOwner().getName()
if (hostname) {
siblings.add(hostname)
}
}
}
}
}