use case: I want to run the regression suite on multiple images(image selection is variable it can be 4 or 5).
I want to create 1 master job who will take the images name as an input, and This master job passes the image name one by one to downstream regression job. The number of images can be variable.
Master job
INPUT image: a,b,c .....
|
|
-------------------------------------------------
| | |
Regression job Regression REgression
Input image: a Image b Image c
Can anyone just tell me how can I perform this task in Jenkins?
To Solve this, I have used the pipeline and Active choice parameter plugins.
Here is the configuration:
Here is the problem, I am getting ThunderImage list as [p,1,p,2,p,3] instead of ['p1','p2','p3'] .
So,you want to select which regression Job starts depending on the selected input from your Master Job? For this you could use the Post build task and use it's regex functionality to check for the input Paramter inside the Build Log.
If you are using a Pipeline job you can use the solution of Christopher Orr from: Jenkins trigger build dependent on build parameters
Related
I got a WS variable which is saving the Jenkins WORKSPACE variable.
During stages I always check if the WS == WORKSPACE Jenkins generic variable.
What I saw that during parallel runs ( when 2 different workspaces are created inside C:/jenkins/workspace#1 and C:/jenkins/workspace#2 the $WORKSPACE is swapping between those 2 parallels builds.
The problem is reproduced rarely , less than 10 percent of cases but I find it quite strange , for the picture above -> The first workspace its AVB_Aplicattions_BOSCH-3 its going trough 3 stages and in the 4th stages the $WORKSPACE variable its swapping with AVB_APLICATTIONS_BOSCH-4( the other parallel build).If I will look on the other build (AVB_APLICATTIONS_BOSCH-4 I will see the same problem-> the workspace is becoming AVB_APLICATTIONS_BOSCH-3).
During this builds I compile using cmake files, I'm afraid that the results will be not correct.
I was thinking to use the in build dir() function during each stage to be sure im on the same workspace .
Some one know a good approach for this case and also why its happening?
I don't want to deactivate parallel's builds.
The problem is that Jenkins variable scopes aren't really safe with parallel stages (see further sources / link). There are actually several questions and answers (and workaround) how to work with variables when using parallel stages. Long story short: don't use global variables but pay extra attention to define local variables with def to limit their scope:
Jenkins - matrix jobs - variables on different slaves overwrite each other?
Variables get overwritten in parallel stages
Parallel matrix and global variables without race condition?
Jenkins community links:
https://issues.jenkins.io/browse/JENKINS-55040
https://issues.jenkins.io/browse/JENKINS-54732
I am trying to run two separated Pipelines from one DataFlow Job similar to below question:
Parallel pipeline inside one Dataflow Job
If we run two separated Pipelines using Single DataFlow job using single p.run() like below:
(
p | 'Do one thing' >> beam.Create(List1)
)
(
p | 'Do second thing' >> beam.Create(List2)
)
result = p.run()
result.wait_until_finish()
I think it will launch two separate pipelines in single Dataflow job but will it create two Bundles and will it run on two different workers?
You are correct: These two sections of a graph will run under the same Dataflow job.
Dataflow paralelizes execution of work items dynamically. This means that depending on your pipeline, the amount of work that it has, etc., Dataflow may decide to schedule on the same worker or on separate workers.
Normally, if there's enough work on both parallel pipelines (~20s on each), then yes: They will run in parallel.
I want to run a job on different slaves, but in a predefined order.
i.e. I would like the job to run on the second machine only when all the executors of the first slave are busy.
I am trying to use "Scoring Load Balancer" plugin to allocate scores to different slaves,i.e. if I have 3 nodes, NodeA,NodeB and NodeC, having preferences of 9,5 and 1 respectively and no. of executors 10 on each node.
The nodes have a common label "WindowSlave".
Also, I have defined a job,"ProjectX", with project preference score of 10 and the label "WindowSlave" as the preferred label.
I had expected that if I run 100 concurrent builds for the "ProjectX" then the execution would happen in the order :
NodeA(10 builds) -> NodeB(10 builds) -> NodeC(10 builds) -> NodeA(10 builds) -> NodeB(10 builds)->... and so on.
From my observations it is still not clear if above scenario would always be achieved.
Also it happens that any random slave starts behaving as the main slave and co-ordinates with the other slave,such that all the build workspace are created on that particular slave.
What am i missing here??
I am running some RecommenderJob (org.apache.mahout.cf.taste.hadoop.item.RecommenderJob) based job from Mahout 0.7 and notice that there are options like startPhase and endPhase. I am guessing these are to run only portions of the pipeline assuming you have necessary input data from prior run(s). But I am having a hard time understanding what kinds of phases there are in RecommenderJob. I am in the middle of reading the source code but it looks like it will take a while. In the meantime I am wondering if anybody can shed light on how to use these options (startPhase in particular) with RecommenderJob class?
Here is what I found:
phase 0 is about PreparePreferenceMatrixJob and it has 3 hadoop jobs:
PreparePreferenceMatrixJob-ItemIDIndexMapper-Reducer
PreparePreferenceMatrixJob-ToItemPrefsMapper-Reducer
PreparePreferenceMatrixJob-ToItemVectorsMapper-Reducer
phase 1 is about RowSimilarityJob and it has 3 jobs:
RowSimilarityJob-VectorNormMapper-Reducer
RowSimilarityJob-CooccurrencesMapper-Reducer
RowSimilarityJob-UnsymmetrifyMapper-Reducer
phase 2 is about RecommenderJob and it has 3 jobs:
RecommenderJob-SimilarityMatrixRowWrapperMapper-Reducer
RecommenderJob-UserVectorSplitterMapper-Reducer
RecommenderJob-Mapper-Reducer
phase 3 is the last one and it has only one job:
RecommenderJob-PartialMultiplyMapper-Reducer
Also output from phase 1 here in RecommenderJob class is exactly the same as the output from phase 0 and 1 of ItemSimilarityJob (but the temp directory names are different).
Yes, that's correct. It's a fairly crude mechanism. Really it controls which of a series of MapReduce jobs are run. You have to read the code to know what they are, yes. They vary by job.
If I'd done it over again I would have just made it detect the presence of output to know to skip the jobs. (That's what I've done in my next-gen recommender project.)
Using Jenkins or Hudson I would like to create a pipeline of builds with fork and join points, for example:
job A
/ \
job B job C
| |
job D |
\ /
job E
I would like to create arbitrary series-parallel graphs like this and leave Jenkins the scheduling freedom to execute B/D and C in parallel whenever a slave is available.
The Join Plugin immediately joins after B has executed. The Build Pipeline Plugin does not support fork/join points. Not sure if this is possible with the Throttle Concurrent Builds Plugin (or deprecated Locks & Latches Plugin); if so I could not figure out how. One solution could be to specify build dependencies with Apache Ivy and use the Ivy Plugin. However, my jobs are all Makefile C/C++/shell script jobs and I have no experience with Ivy to verify if this is possible.
What is the best way to specify parallel jobs and their dependencies in Jenkins?
There is a Build Flow plugin that meets this very need. It defines a DSL for specifying parallel jobs. Your example might be written like this:
build("job A")
parallel (
{
build("job B")
build("job D")
},
{
build("job C")
}
)
build("job E")
I just found it and it is exactly what I was looking for.
There is one solution that might work for you. It requires that all builds start with a single Job and end with a definite series of jobs at the end of each chain; in your diagram, "job A" would be the starting job, jobs C and D would be the terminating jobs.
Have Job A create a fingerprinted file. Job A can then start multiple chains of builds, B/D and C in this example. Also on Job A, add a promotion via the Promotions Plugin, whose criteria is the successful completion of the successive jobs - in this case, C and D. As part of the promotion, include a trigger of the final job, in your case Job E. This can be done with the Parameterized Trigger Plugin. Then, make sure that each of the jobs you list in the promotion criteria also fingerprint the same file and get the same fingerprint; I use the Copy Artifact Plugin to ensure I get the exact same file each time.