I'm building a windows service which is performing a scheduled task which processes a queue of commands (from a legacy system) at a regular interval (once per min) using Quartz.net
If the task takes longer than 1 min, which would be unusual but possible under certain circumstances, I'd like it to simple ignore the triggers it missed firing.
However I can't seem to get this to happen. It does the processing and then quickly fires all the triggers it missed in quick succession. As I understand you can set a threshold for the misfires but I don't seem to be-able to get this working.
I'm using [DisallowConcurrentExecution()] on the Job to ensure only one instance of the job is running at any one time.
Below are a couple of snippets. First passing in some config info - is this how you set the misfire threshold?
NameValueCollection config = new NameValueCollection();
config.Add("quartz.jobStore.misfireThreshold", "600000");
schedFact = new StdSchedulerFactory(config);
Building a trigger with what I assume to be the correct setting of Ignore misfires:
var trigger = TriggerBuilder.Create()
.WithIdentity("trigger1", "group1")
.StartNow()
.WithSimpleSchedule( x => x.WithMisfireHandlingInstructionIgnoreMisfires()
.WithIntervalInSeconds(60)
.RepeatForever())
.Build();
Thoughts much appreciated.
Code for the Job:
Just playing with rough ideas at this point so just running in a console app and randomly delaying a job so it shoots over the interval which is just set at 10 seconds for this. All the backed up misfires fire in quick succession after a couple of delays.
[DisallowConcurrentExecution()]
public class SomeJob : IJob
{
public SomeJob() { }
public void Execute(IJobExecutionContext context)
{
Random rnd = new Random(DateTime.UtcNow.Second);
int delay = rnd.Next(2);
Console.WriteLine("Executing Job with delay of "+ delay + " at " + DateTime.UtcNow.ToString());
if (delay == 1)
{
System.Threading.Thread.Sleep(1000 * 25); // sleep for 25 seconds
}
}
}
Example console output:
Executing Job with delay of 1 at 21/05/2015 21:27:17
Executing Job with delay of 1 at 21/05/2015 21:27:42
Executing Job with delay of 0 at 21/05/2015 21:28:07 <-- stacked misfires
Executing Job with delay of 0 at 21/05/2015 21:28:07 <--
Executing Job with delay of 0 at 21/05/2015 21:28:07 <--
Executing Job with delay of 0 at 21/05/2015 21:28:07 <--
Executing Job with delay of 0 at 21/05/2015 21:28:16
Executing Job with delay of 0 at 21/05/2015 21:28:26
Executing Job with delay of 1 at 21/05/2015 21:28:36
Executing Job with delay of 0 at 21/05/2015 21:29:01
Executing Job with delay of 0 at 21/05/2015 21:29:01
Executing Job with delay of 1 at 21/05/2015 21:29:06
WithMisfireHandlingInstructionIgnoreMisfires() is the wrong method for what you want, it doesn't mean the job will not trigger misfires, it means it will fire all triggers that were missed as soon as possible and will then go back to ordinary schedule. Which is what you see happening.
What you need is WithMisfireHandlingInstructionNextWithRemainingCount(). This will signal the scheduler to ignore misfires and wait for the next scheduled time.
Misfiring strategies can be a bit confusing, for a concise explanation look here (it's for the java scheduler but it doesn't matter).
Even with the right strategy, you need to understand how the misfire threshold works- if you get it wrong your misfired trigger may still fire.
From the documentation: "A misfire is the time span by which a trigger must have missed its next-fire-time, in order for it to be considered "misfired" and thus have its misfire instruction applied".
You've set the value to 600000 milliseconds (and this value is for all triggers, unfortunately there is no threshold per trigger)- the misfire strategy will only be applied if your trigger fires 600000 milliseconds after the time it should have been fired. You can lower or increase it according to your requirement.
For more about the misfire threshold, read here.
Related
Below is a simplified case.
I have one node named comp01. And I have a Jenkins job named Compatibility.
Compatibility is scheduled as follows:
0 12 * * 1 %IntegrationNode=Software_1
0 17 * * 1 %IntegrationNode=Software_2
0 22 * * 1 %IntegrationNode=Software_3
0 2 * * 2 %IntegrationNode=Software_4
0 7 * * 2 %IntegrationNode=Software_5
The jobs start as scheduled. But sometimes, because of some verification failure, the previous job takes more than expected time. So, the next job starts before the completion of the previous job.
Is there a way available in Jenkins, in which the next scheduled job stays in a queue until previous job is complete? Or can we schedule based on previous job status?
We have tried limiting executors for this job, but when more than a couple of jobs are queued, then the expected behavior is not observed.
We have also tried by creating resource-groups and adding multiple nodes to it, but still, expected behavior is not observed when multiple jobs are in queue.
EDIT-1:
We can't use options { disableConcurrentBuilds() } since we start the job concurrently on different nodes. Here we are struggling to ensure that when a job is started on a node, then the other scheduled jobs for the same node should wait till the current job completes.
Have you tried setting the below option?
options { disableConcurrentBuilds() }
Update
AFAIK there is no OOB solution for your problem. But you can definitely implement something. Without seeing your actual Pipelines I can't give a concrete answer. But here ae some options.
Option 01
Use Lockable Resources and create a resource per Jenkins IntegrationNode and acquire it when running the Job, the next build will wait until the lock is released.
lock(resource: 'IntegrationNode1', skipIfLocked: false) {
echo "Run your logic"
}
Option 02
You can implement a waiting logic to check the status of the previous Build. Here is an sample Pipeline and possible Groovy code you can leverage.
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
echo "Waiting"
def jobName = "JobA"
def buildNum = "92"
waitUntil { !isPending(jobName, buildNum) }
echo "Actual Run"
}
}
}
}
}
def isPending(def JobName, def buildNumber) {
def buildA = Jenkins.instance.getItemByFullName(JobName).getBuild(buildNumber)
return buildA.isInProgress()
}
I know I can specify .WithMisfireHandlingInstructionDoNothing() when building the trigger but some of my jobs are triggered via the IScheduler.TriggerJob() method, so without any triggers.
I can detect and log misfires in the ITriggerListener listener but how can I stop Quartz from trying to fire the job again? If I understand correctly .VetoJobExecution is not usable since the job has to be triggered successfully anyway.
Any other ideas?
Edit: my implementation
JobDataMap jobData = new JobDataMap(data);
IJobDetail jobTemplate = await jobScheduler.GetJobDetail(jobKey);
var jobTrigger = TriggerBuilder.Create()
.ForJob(jobTemplate)
.UsingJobData(jobData)
.WithSimpleSchedule(s => s.WithRepeatCount(0).WithMisfireHandlingInstructionNextWithRemainingCount())
.StartNow()
.Build();
await jobScheduler.ScheduleJob(jobTrigger);
Well if you just want the TriggerJob behavior you can achieve that just by adding one simple trigger to scheduler that is going to trigger immediately and configure retry policy for that. So if you can change the call sites of TriggerJob to create a simple trigger instead (maybe an extension method that allows to define the policy), the Quartz source for TriggerJob is here.
I think I've got it.
You have to use the WithMisfireHandlingInstructionNextWithRemainingCount policy, and WithRepeatCount(0).
But the trick here is to set the right value in the IScheduler MisfireThreshold to the misfire to be considered as a real misfire. I mean, if your misfire threshold is set to 60 seconds (default) then any job not executed in less than 60 seconds from schedule, will NOT be considered as a misfire. And so, the misfire policy will not be used.
In my case, with 3 second duration jobs, I had to set the WithMisfireThreshold to 1 second or less.
This way, the job is not retried if "really misfired".
In Jenkins, I want to be able to delay the execution of a job, i.e. for 1 hour, after I click Build.
I want to set up the parameters, click Build and the job to stay in the Queue for 1 hour, without using an executor, and then start. I do not want to schedule a job periodically or sth like that. Just to force it to stay in the queue for a certain (maybe configurable) period of time.
Is there a way to do this?
Using sleep here is exactly the right thing to do. To workaround the fact that no new nodes are consumed and idle, just execute the sleep command on a special sleepnode. So every Job will use this sleepnode for sleeping, which means no additional nodes would get consumed.
node('sleeper') {
stage('Countdown 1hr') {
sleep 3600
}
}
node('buildnode1') {
stage('build') {
echo "What ever should be done here"
}
}
I have a grails application and a quartz job running on it. The job contains the below code similar to below .
class MyJob{
static triggers = {}
def printLog(msg){
String threadId = Thread.currentThread().getId()
String threadName = Thread.currentThread().getName()
log.info(threadId+" - "+threadName+" : "+msg)
}
def execute(context)
{
printLog("Before Sync");
synchronized(MyJob){
printLog("Inside Sync");
try{
printLog("Before sleep 20 minutes")
Thread.sleep(1200000)
printLog("After sleep")
}catch (Exception e){
log.error("Error while sleeping")
}
}
printLog("After Sync")
}
}
I have scheduled it to trigger a job every minute
I can see in the logs that one thread is getting the synchronized block and then the other jobs start piling up, waiting for the thread to finish, this is working as expected.
The issue here is the jobs stop after 10 minutes by that time it have created 10 Threads. Out of that one is sleeping for 20 minutes and other 9 are waiting for the 1st thread to release the lock. Why is no new jobs created ?
I saw in some answers I can fix the issue by modifying my triggers section like below
static triggers = {
simple repeatInterval: 100
}
I tried the above option and its still showing only 10 jobs.
From where its taking the default configuration of 10 ?
How can i modify the value to do infinitely ?
I am new to grails and quartz, so I have no idea what is happening.
I think the Grails plugin sets the threadCount to 10 in the bundled quartz.properties file, assuming you're using Grails 3 you can override in application.yml like this:
quartz:
threadPool:
threadCount: 25
Grails 2 - application.groovy
quartz {
props {
threadPool.threadCount = 100
}
}
In general, it's not a a good idea to lock the Job thread with sleeps
If you have a job running a long process you must to split it in several jobs in order to release the Thread as soon as posible
I have an application using Grails Quartz plugin. I need to have two jobs to have multiple instances running, but have separate limitation on number of threads to be used for each job. As far as I understand, I need separate Thread Pools, which is possible by having separate schedulers. However, I cannot figure out how to create multiple schedulers with Quartz plugin.
Assuming you want to use different triggers to start the job multiple times. this works for me.
class MyJob {
static triggers = {
cron name: 'trigger1', cronExpression: "0 30 12 ? * WED"
cron name: 'trigger2', cronExpression: "0 30 12 ? * SAT"
}
def execute() {
// execute task, do your thing here
println "Job executed"
}
}
Finally, about concurrent tasks.
This is from the plug-in page:
By default Jobs are executed in concurrent fashion, so new Job
execution can start even if previous execution of the same Job is
still running.
Quartz plugin 2.0.13
According to the official documentation :
Multiple triggers per job are allowed.
For instance,
class MyJob {
static triggers = {
simple name:'simpleTrigger', startDelay:10000, repeatInterval: 30000, repeatCount: 10
cron name:'cronTrigger', startDelay:10000, cronExpression: '0/6 * 15 * * ?'
custom name:'customTrigger', triggerClass:MyTriggerClass, myParam:myValue, myAnotherParam:myAnotherValue
}