I'm trying out Quartz.net, which would possibly solve an issue of mine. However I can not seem to find a way to start a job at a given time of day (07:30) to run a number of times (1000) with a interval in milliseconds (1).
I've tried a CronSchedule, but intervals can not be set. With SimpleSchedule, a start time can not be set and with DailyTimeIntervalSchedule I can not set interval in milliseconds. I've also tried to combine a setup with the varoius with the fluent api, to no avale.
Is what I try to achieve actually not possible in Quartz.net?
This might be what you are after:
IJobDetail theJobToRun = JobBuilder.Create<NoOpJob>().Build();
var trigger = TriggerBuilder.Create()
.StartAt(DateBuilder.DateOf(7, 30, 0))
.WithSimpleSchedule(x => x
.WithInterval(TimeSpan.FromMilliseconds(1))
.WithRepeatCount(999))
.ForJob(theJobToRun)
.Build();
Just as sgmoore noted, you might not get millisecond precision as your thread pool will be saturated with jobs and it all depends how much work they true. Quartz.NET infrastructure will also take its own time watching for fire times and instantiating jobs.
Related
I have a situation where I'm trying to count the number of files loaded into the system I am monitoring. I'm sending a "load time" metric to Datadog each time a file is loaded, and I need to send an alert whenever an expected file does not appear. To do this, I was going to count up the number of "load time" metrics sent to Datadog in a 24 hour period, then use anomaly detection to see whether it was less than the normal number expected. However, I'm having some trouble finding a way to consistently pull out this count for use in the alert.
I can't use the count_nonzero function, as some of my files are empty and have a load time of 0. I do know about .as_count() and count:metric{tags}, but I haven't found a way to include an evaluation interval with either of these. I've tried using .rollup(count, time) to count up the metrics sent, but this call seems to return variable results based on the rollup interval. For instance, if I compare intervals of 2000 and 4000 seconds, I would expect each 4000 second interval to count up about the sum of two 2000 second intervals over the same time period. This does not seem to be what happens at all - the counts for the smaller intervals seem to add up to much more than the count for the larger one. Additionally some rollup intervals display decimal numbers as counts, which does not make any sense to me if this function is doing what I thought it was.
Does anyone have any thoughts on how to accomplish this? I'd really appreciate any new ideas.
We got one or two CheckPoint Failure during processing data every day. The data volume is low, like under 10k, and our interval setting is '2 minutes'. (The reason for processing very slow is we need to sink the data to another API endpoint which take some time to process at the end of flink job, so the time is Streaming data + Sink to external API endpoint).
The root issue is:
Checkpoints time out after 10 mins, this caused by the data processing time longer than 10 mins, so the checkpoint time out. We might increase the parallelism to fast the processing, but if the data become bigger, we have to increase the parallelism again, so don't want to use this way.
Suggested solution:
I saw someone suggest to set the pause between old and new checkpoint, but I have some question here is, if I set the pause time there, will the new checkpoint missing the state in the pause time?
Aim:
How to avoid this issue and record the correct state that doesn't miss any data?
Failed checkpoint:
Completed checkpoint:
subtask didn't respond
Thanks
There are several related configuration variables you can set -- such as the checkpoint interval, the pause between checkpoints, and the number of concurrent checkpoints. No combination of these settings will result in data being skipped for checkpointing.
Setting an interval between checkpoints means that Flink won't initiate a new checkpoint until some time has passed since the completion (or failure) of the previous checkpoint -- but this has no effect on the timeout.
Sounds like you should extend the timeout, which you can do like this:
env.getCheckpointConfig().setCheckpointTimeout(n);
where n is measured in milliseconds. See the section of the Flink docs on enabling and configuring checkpointing for more details.
So I'm running a LUA script that executes every minute, this is controlled by software and I can't control the timing of the execution. I would like to check the time every day and trigger a function at a specific time. However, I would like to execute another script 5 minutes before that happens.
Right now I'm doing a string comparison using os.date() and parsing it to a string. It works, but the code isn't pretty and if the time changes, i have to manually change the time in two different variables, I'm pretty new to LUA so I've been having difficulty figuring out the best way to do this.
So the question is, how do I set a time variable, and compare that variable to os.date (or os.time) ?
There's no fancy way to do timers in pure Lua. What you can do to avoid os.date() strings, is to generate timestamps with preset dates, os.time accepts a key-value table to set a date:
timestamp = os.time({year=2019, month=1, day=n})
By iteratively increasing the n variable, you will receive a timestamp for every new n day after January 1st 2019. E.g.
os.date("%Y-%m-%d %H-%M-%S",os.time({year=2019,month=1,day=900})
--> 2021-06-18 12-00-00
If you can't save the day variable to keep track of (between application restarts), get the current "today" day and iterate from there:
os.date("%Y-%m-%d %H-%M-%S",
os.time({year=os.date("%Y"),month=os.date("%m"),day=os.date("%d")+n}
)
Using os.date with custom format and os.time makes your code independent of currently set date locale.
After you have determined the timestamp of the first task, offset the second actual task by five minutes secondTaskTimestamp = fistTaskTimestamp + 5*60 (or use os.time again). Your timer checker only should compare timestamps by now.
Now when you have to change the pre-configured time, you will only have to change the time-date of the first task, and the second task will be automatically offset.
Related: How do I execute a global function on a specific day at a specific time in Lua?
I have a quartz job schedule like so:
_scheduler.ScheduleJob(job,
TriggerBuilder.Create().ForJob(job)
.WithSimpleSchedule(x =>
x.WithIntervalInSeconds(requeryTime.ToInt32())
.RepeatForever()
.WithMisfireHandlingInstructionNowWithRemainingCount()
)
.StartNow()
.Build())
To test I have the schedule run every 10 seconds. If I pause the service and then resume, I can still see all the jobs being called one after another. I want Quartz to ignore (do not queue) the jobs and just resume the schedule.
e.g. If the job runs every 10 secs and I pause the service for 1 minute, I get 6 jobs run in quick succession.
How do I go about ensuring Quartz ignores those jobs?
Short answer is to change the misfireThreshold in the configuration:
<add key="quartz.jobStore.misfireThreshold" value="60000" />
The default is 60000 milliseconds (60 seconds).
org.quartz.jobStore.misfireThreshold
The the number of milliseconds the scheduler will 'tolerate' a trigger
to pass its next-fire-time by, before being considered "misfired". The
default value (if you don't make an entry of this property in your
configuration) is 60000 (60 seconds).
Since your trigger runs every 10 seconds you can reduce the threshold to 9 seconds and it should work as you expect.
If you want to learn more about the strategies involved you can read this article and this one as well.
A similar question has been asked recenlty and Circular Reference gave a really good answer.
I have some tests that are very slow and I want to set as timeout 15 minutes.
As test purpose I have this example:
[Test, Timeout(900000)]
public void Test1()
{
Thread.Sleep(900001);
}
The test after some time stops without errors.
What's the correct way to do it?
As you know, both Thread.Sleep and NUnit's TimeoutAttribute take time in milliseconds. Specifying times that are 1 millisecond off from each other isn't enough to guarantee a timeout due to thread scheduling and general timer accuracy. See this answer for a little more discussion about the accuracy of Thread.Sleep, and by extension NUnit's timeout thread's accuracy.
Try specifying a larger difference between the two numbers and I suspect you'll see it behaves as you'd expect. For instance, sleep for 900100 milliseconds and leave the timeout value as is. With a timeout of 15 minutes, you won't notice an extra tenth of a second waiting for it to timeout.