I am looking into Quartz.NET to process our daily jobs and it seems to handle our scenarios. We also have jobs that are added to a queue table in SqlServer and this table is polled for work every few seconds. How can this be handled using Quartz.NET?
Quartz.net will replace the queue/polling functionality. Quartz.net has its own queue (jobstore) which it polls and executes jobs from it.
What you will do is schedule your jobs in quartz (queue them) and then Quartz.Net will execute them based on the time they need to be executed (determined by the trigger). In Quartz.Net the job (does the work) is separate from the trigger (determines when I job should run).
Related
Till today I used Quartz.net for a single job or different jobs that worked independently.. now I need to run 2 different jobs based on some data I've. I've created 3 jobs for now and called them one after the other (am doing ugly things I won't write here to run them one after the other...)
I was wondering if I use a queue, shared between the jobs, (or even 3 background worker that works on different queue) can I have this working with Quartz.net?
I mean I have the main job run, the other works on their on queue, but the main job is not scheduled until the worker is finished?
I've just put on the main job the [DisallowConcurrentExecution] attribute
Thanks
I am looking for a way to be able run Active Job serially. Ideally, a long running Job 1 is scheduled to run at a certain time. A similarly running Job 2 is slated to run only after Job 1 completes. Job 3 then waits for Job 2 to run to completion before it starts and so on.
I have to admit that I am rather new to background jobs in Rails but I am already using Active Job with Sidekiq as the job runner for simple fire-and-forget tasks.
I like Active Job because it provides a simple enough interface to dive almost immediately into background jobs processing. I can use Sidekiq without having to define workers, for example.
For reference, I have achieved something similar but it was on .NET using the excellent Hangfire library which has continuations where you pass the ID of a parent job ensuring that the job will run only after the parent job has successfully completed.
It would be nice to have something as clean and simple as that using Sidekiq and Active Job but really any alternative ways to achieve the same thing are welcome. It doesn't have to be Sidekiq and Active Job.
The most straightforward way to to this is to call a third job from within a second job, and the second one from within a first job
How can I manage to execute job after the first job that has executed is done in sidekiq. For example:
I triggered the first job for this morning
GoodWorker.perform_async(params) #=> JID-eetc
while it is still in progress I've executed again a job in the same worker dynamically
GoodWorker.perform_ascyn(params) #=> JID-eetc2
and etc.
What's going on now is Sidekiq processing the jobs all of the time,
is there a way performing the job one at a time?
Short answer: no.
Long answer: You can use a mutex to guarantee that only one instance of a worker is executing at a time. If you're running on a cluster, you'll need to use Redis or some other medium to maintain the mutex. Otherwise, you might try putting these jobs in their own queue, and firing up a separate instance of Sidekiq that only monitors that queue, with a concurrency of one.
Can you not setup Sidekiq to only have one thread? Then only one job will be executed at a time.
We have designed our test jobs to some sort of "abstract" test jobs, that run according to a set of parameters. These jobs are triggered use "runner" jobs that simply trigger them with the correct parameters (mostly generated by matrix jobs).
When we run multiple "runners", that all they do is simply trigger the abstract jobs, they occupy much needed workers (especially when it is a matrix job, that creates multiple temporary "runner" jobs).
Is there a way to tell jenkins to not spend a worker on a job that only trigger other jobs, or trigger jobs within the same worker?
It depends on what you use to trigger the jobs
If you use Trigger/call builds on other projects action, it has an option to Block until the triggered projects finish their builds. If that is checked, the triggering parent job will remain running and waiting for the triggered job to finish (thus occupying at least 2 executors). However, if you keep that unchecked, it will launch the triggered job, and the triggering job will end soon after.
I want my builds to wait until triggered jobs are completed, for reporting purposes and such (I don't want that logic in the triggered jobs due to their abstract nature).
What I decided to do, since the triggering jobs are very lightweight, I restricted them all to the master. I allocated a large number of workers to the master, since they won't do much work and they will simply manage the triggering of other jobs.
Configuration
Quartz.net 2.0.1
JobStore: SqlServer
Statefull jobs are running inside a windows service.
There is also a console application that allow firing the same jobs inside service.
I want to initialize correctly quartz scheduler so that it will respect non-concurrent job execution while allowing to fire a certain job immediately from console.
As long as you mark your job with the DisallowConcurrentExecutionAttribute attribute, you can schedule your jobs from any source you want and the scheduler will make sure only one instance runs at a time.