I have a matrix job that run only on a certain virtual slave. It has two axis with two elements each. The job runs nightly, and each configuration takes 2-3 hours. Sometimes the results are needed the first things in the morning - and the configuration that is needed is still building.
Is it possible to build matrix jobs in certain order? I'd prefer a solution that is easily re-configurable.
It's possible using Priority Sorter Plugin, although it's still buggy and requires a lot of work: JENKINS-8597
Related
I have a scenario where I have N numbers of nodes and N number of tests, and these tests will be distributed to the nodes. My nodes have a label Windows.
Here's an example:
I have a pipeline job that will manage the distribution of the tests to the nodes. I set my pipeline job to run 10 tests on 10 VMs having the label Windows and it will run smoothly. However, one of my requirements is to concurrently run that pipeline job. The problem I might encounter If I have 10 tests on VMs 1-10 in the first run of the job, and run another job for 5 tests for VMs 11-15, given that I am using the Windows label, there might be a possibility that Jenkins will assign the test to VMs 1-10 but should run on VMs 11-15 or vice versa.
The solution I came up with is to dynamically change the label of the VMs from one of the jobs to a unique label that will only be used for that Job. Unfortunately, I still don't know how to do that.
Basically, I just want to logically group my nodes via label on demand in my pipeline script.
I searched all throughout the internet and yet I still wasn't able to find a solution that fits my needs.
Kindly help me with this. Also, I am open to using a different approach.
We are currently struggling with the following task. We need to run a windows application (single instance only working) 1000 times with different input parameters. One run of this application can take up to multiple hours. It feels like we have the same problem like any video rendering farm – each picture of a video should be calculated independently and parallel – but it is not rendering.
Currently we tried to execute it with Jenkins and Pipeline jobs. We used the parallel steps in pipeline and lets Jenkins queue and execute the application. We use the Jenkins Label Expression to lets Jenkins choose which job can be run on which node.
The limitation in Jenkins is currently with massive parallel jobs (https://issues.jenkins-ci.org/browse/JENKINS-47724). When the queue contains multiple hundred jobs adding new jobs took much longer – will become even worse by increasing queue. And main problem: Jenkins will start the execution of parallel pipeline part-jobs only after finishing adding all to the queue.
We already investigated ideas how to solve this problem:
Python Distributed: https://distributed.readthedocs.io/en/latest/
a. For single functions it looks great, but for the complete run like we have in Jenkins => Deploy and collect results looks complex
b. Client->Server bidirectional communication needed – no chance to bring it online through a NAT (VM Server)
BOINC: https://boinc.berkeley.edu/
a. for our understanding we had to extend the backend in a massive way to bring our jobs working => to configure the jobs in BOINC we had to write a lot of new automating code
b. currently we need a predeployed application which can differ between different inputs => no equivalent of Jenkins Label Expression
Any ideas how to solve it?
Thanks in advance
I'm trying to build a Jenkins declarative pipeline that will build on all agents in parallel.
How can I do this without disabling sandbox?
I have come across this page: https://jenkins.io/blog/2017/09/25/declarative-1/ but it seems repetitive, especially when padded out with my code as nearly all operations are performed almost the same on every node. Is there a way to do this and avoid repeating code?
I suggest that you follow the common pattern described in the referenced article.
By assigning labels identifying the node's operating system and allocating nodes based on these labels, you ensure that the job runs exactly once in each of the different build environments.
A severe drawback of your suggestion to build on all of the available agents (as said, I don't know anything how to actually do that)) would be in the case of one or multiple build agents being offline. So you don't run on Windows, because the server was just rebooting, but your build result is green as nothing failed? Not a good idea, isn't it?
Another benefit of the label-based approach is that you can easily add additional build agents to cope with increased number of builds, e.g., as your team grows. You don't want to build twice on Windows, when you add another build agent with Windows, right?
So I strongly recommend: Assign labels to your build agents and then specify, on which agents your job needs to run.
I am trying to create a jenkins job that will run some code on various servers to validate them.
I would like to be able to specify either an individual server or give a directive such as "evens" for servers 02, 04, 06... or "odds" for servers 01, 03, 05... and have the job run for either a single or many servers.
I'm searching for the cleanest way to do this, I've tried using a scheduler job that would handle the odds and evens cases but would prefer, if possible, not to have to split the single and many server cases into different jobs. I've also looked into using a matrix job that could be configured to run under different parameter set but haven't found any documentation to fully solve my problem.
Can anyone point me in the right direction?
I am not sure I fully understood you but I will try:
I understood that you want to activate the same job many times with different parameters.
your options are :
1. using master job that will activate all other jobs with different parameters ( like you said with matrix or even simpler )
2. you can do it in pipeline rather easily with node scopes and loops for using different parameters
3. you can use jenkins-cli and activate the same job with different parameters every activation.
I hope it helped you
We have different jobs running on our jenkins. Some jobs are heavy and taking a lot of CPU and RAM, some are not. So I would like to have some plugins to help me set the weight for those jobs, just like https://wiki.jenkins-ci.org/display/JENKINS/Heavy+Job+Plugin.
But since we are using Jenkins pipeline which is not supported by Heavy Job plugin (See https://issues.jenkins-ci.org/browse/JENKINS-41940). Is there any other equivalent for pipeline jobs just like that?
Not as flexible as a dynamic weight, but to avoid overload you can create several executors with a different label (such as many with label light and only a few with label heavy) and then use node to target these labels.
This is not a solution to the packing problem, only prevents too many jobs with a heavy class from running at the same time.