I want to run one job multiple times (each build with different parameters) on 2 executors.
I want to execute them based on their build priority value.
Unfortunately Priority Sorter plugin doesn't help in my case, it doesn't sort it correctly - my builds are being executed based on the timestamp they were added to the queue instead of priority.*
I believe this priority mechanism should be implemented before, on a queue level.
How to achieve that?
*-I tested it on the newest Jenkins version and the newest Priority Sorter plugin version
I think what you should try is Accelerated Build Now plugin. This plugin allows the Jenkins users to launch a build right there, even if the queue is long. It prioritises human launched jobs and brings them to the top.
The Priority Sorter I am using (v3.6.0 and Jenkins v.2.73.3) will not see a job unless you have enabled the Execute Concurrent builds if necessary.
So, allow parallel builds for that job and maybe decrease the number of cores to 1. See if that works. If not, you can try Throttle Concurrent Builds Plugin. This allows you to assign as many cores as you want in the specific job.
Here is a patch for Priority Queue plugin. For your case it could be this patch.
I do have a job which requires external ressources and therefore it should not executed twice or more often. I used Block Queued Job Plugin to block the job if of a list of jobs is currently running.
This creates sometimes a build queue with some jobs blocked by the plugin ... which is correct.
But now I do need to reorder the build queue to give a specific build a chance to be executed.
There is usually just the fifo principle in place but I do need to overwrite this in specific situations manually.
simple queue plugin ... can not deal with blocked jobs
priority sorter .... sees to be outdated and not working for such a simple thing ...
Currently I write down the parameter handed over per job delete all and afterwards rebuild with the new order and with the parameters which were manually written down.
This is quit bad and I do need a working solution. Maybe I missed the right plugin.
I'm working on a busy Jenkins server with many big matrix jobs being built all the time. Working with another job can be really annoying if you want instantaneous results... Is there an option to give a job temporary priority for just one build so it skips the queue and gets built on the next available executor?
You can try using Priority Sorter Plugin and add your jobs with higher priority than the rest
I am using throttle concurrent build to run job in parallel. But I am not able to run the job in parallel. Only single build is triggered.
In Job configuration : selected Throttle Concurrent Builds and specified Maximum Total (ex:4)and/or Maximum Per Node(Ex:2)
selected “Execute concurrent build if possible” option also
I have one Master(2 Executors) and one Agent(2 Executors) in Jenkins.
Kindly help me to resolve this problem.
From the Throttle Concurrent Builds Plugin
It should be noted that Jenkins, by default, never executes the same
Job in parallel, so you do not need to actually throttle anything if
you go with the default. However, there is the option Execute
concurrent builds if necessary, which allows for running the same Job
multiple time in parallel, and of course if you use the categories
below, you will also be able to restrict multiple Jobs.)
So you need to check the box, which I think might be in the advance settings
I need some advice writing a Job scheduler in Erlang which is able to distribute jobs ( external os processes) over a set of worker nodes. A job can last from a few milliseconds to a few hours. The "scheduler" should be a global registry where jobs come in, get sorted and then get assigned and executed on connected "worker nodes". Worker nodes should be able to register on the scheduler by telling how many jobs they are able to process in parallel (slots). Worker nodes should be able to join and leave at any time.
An Example:
Scheduler has 10 jobs waiting
Worker Node A connects and is able to process 3 jobs in parallel
Worker Node B connects and is able to process 1 job in parallel
Some time later, another worker node joins which is able to process 2 jobs in parallel
Questions:
I seriously spent some time thinking about the problem but I am still not sure which way to go. My current solution is to have a globally registered gen_server for the scheduler which holds the jobs in its state. Every worker node spawns N worker processes and registers them on the scheduler. The worker processes then pull a job from the scheduler (which is an infinite blocking call with {noreply, ...} if no jobs are currently availale).
Here are some questions:
Is it a good idea to assign every new job to an existing worker, knowing that I will have to re-assign the job to another worker at the time new workers connect? (I think this is how the Erlang SMP scheduler does things, but reassigning jobs seems like a big headache to me)
Should I start a process for every worker processing slot and where should this process live: on the scheduler node or on the worker node? Should the scheduler make rpc calls to the worker node or would it be better for the worker nodes to pull new jobs and then execute them on their own?
And finally: Is this problem already solved and where to find the code for it? :-)
I already tried RabbitMQ for job scheduling but custom job sorting and deployment adds a lot of complexity.
Any advice is highly welcome!
Having read your answer in the comments I'd still recommend to use pool(3):
Spawning 100k processes is not a big deal for Erlang because spawning a process is much cheaper than in other systems.
One process per job is a very good pattern in Erlang, start a new process run the job in the process keeping all the state in the process and terminate the process after the job is done.
Don't bother with worker processes that process a job and wait for a new one. This is the way to go if you are using OS-processes or threads because spawning is expensive but in Erlang this only adds unnecessary complexity.
The pool facility is useful as a low level building block, the only thing it misses your your functionality is the ability to start additional nodes automatically. What I would do is start with pool and a fixed set of nodes to get the basic functionality.
Then add some extra logic that watches the load on the nodes e.g. also like pool does it with statistics(run_queue). If you find that all nodes are over a certain load threshold just slave:start/2,3 a new node on a extra machine and use pool:attach/1to add it to your pool.
This won't rebalance old running jobs but new jobs will automatically be moved to the newly started node since its still idle.
With this you can have a fast pool controlled distribution of incoming jobs and a slower totally separate way of adding and removing nodes.
If you got all this working and still find out -- after some real world benchmarking please -- you need rebalancing of jobs you can always build something into the jobs main loops, after a message rebalance it can respawn itself using the pool master passing its current state as a argument.
Most important just go ahead and build something simple and working and optimize it later.
My solution to the problem:
"distributor" - gen_server,
"worker" - gen_server.
"distributor" starts "workers" using slave:start_link, each "worker" is started with max_processes parameter,
"distributor" behavior:
handle_call(submit,...)
* put job to the queue,
* cast itself check_queue
handle_cast(check_queue,...)
* gen_call all workers for load (current_processes / max_processes),
* find the least busy,
* if chosen worker load is < 1 gen_call(submit,...) worker
with next job if any, remove job from the queue,
"worker" behavior (trap_exit = true):
handle_call(report_load, ...)
* return current_process / max_process,
handle_call(submit, ...)
* spawn_link job,
handle_call({'EXIT', Pid, Reason}, ...)
* gen_cast distributor with check_queue
In fact it is more complex than that as I need to track running jobs, kill them if I need to, but it is easy to implement in such architecture.
This is not a dynamic set of nodes though, but you can start new node from the distributor whenever you need.
P.S. Looks similar to pool, but in my case I am submitting port processes, so I need to limit them and have better control of what is going where.