My objective is to have Quartz.NET execute a job at precisely 25Hz or every 40ms.
I'm using the following trigger:
ITrigger MyTrigger = TriggerBuilder.Create().WithIdentity("T1").ForJob("MyJob").WithSimpleSchedule(x => x.WithInterval(TimeSpan.FromMilliseconds(40)).RepeatForever()).Build();
and the following job:
[DisallowConcurrentExecution]
private class MyJob : Quartz.IJob
{
public void Execute(IJobExecutionContext context)
{
Idx++;
Console.WriteLine("Job {0} fired at {1}ms", Idx, MyStopWatch.ElapsedMilliseconds);
}
}
The problem is that the first 150 executions or so fire too quickly. For example the first 60 iterations all fire at either 20ms or 21ms on the stopwatch. Afterward they fire in bunches every 200ms until it becomes stable around 1000ms, and then starts firing every 40-42ms as intended.
How can I prevent Quartz from triggering a job if the previous job was fired within 40ms?
What is the source of this behavior?
You should not rely on Quartz for millisecond precision scheduling. Quartz has a lot of infrastructure (misfire checks, compensation etc) that will make it unsuitable for this kind of precision.
I'd argue that stable firing on bare virtual machine level could be a challenge too due to GC etc that might pause things a bit.
Quartz can do a lot for you but in this case I'd just go with custom Thread or Timer implementation if these are your requirements.
Related
Gradle does not work correctly in a docker environment, it is destined to use too much memory and be killed for using too much memory.
The memory manager gets its snapshots using the following class
https://github.com/gradle/gradle/blob/master/subprojects/process-services/src/main/java/org/gradle/process/internal/health/memory/MemInfoOsMemoryInfo.java
and in particular Gradle determines how much free memory is left by reading /proc/meminfo, which provides an inaccurate reading in a container.
Gradle only kills off Worker Daemons when a request comes in to make a new Worker Daemon with a larger min heap size then is available according to this reading.
Thus, Gradle will keep making workers until it uses up the alotted amount for the container and be killed.
Does anyone have a workaround for this? Don't really understand how this hasn't been a problem for more people. I suppose it only really becomes an issue if your worker daemons can't be reused and so new ones get created, which is the case for me as I have a large number of modules.
I have a temporary workaround wherein I give every jvm spawned a huge -Xms and so it always triggers the min heap size > available and so always removes prior worker daemons, but this is not satisfactory.
-- edit
To preempt some things, --max-workers does not affect the number of Worker Daemons allowed to exist, it merely affects the number which are allowed to be active. Even with --max-workers = 1, it is allowed to have arbitrary many idle Worker Daemons.
Edit - Ignore the below, it somewhat works but I have since patched Gradle by overwriting the MemInfoOsMemoryInfo class and it works a lot better. Will provide a link to the MR onto Gradle soon.
Found a reasonable work around, we listen for the os memory updates, and every time a task is done we request more memory than is determined to be free, ensuring a daemon is stopped.
import org.gradle.process.internal.health.memory.OsMemoryStatus
import org.gradle.process.internal.health.memory.OsMemoryStatusListener
import org.gradle.process.internal.health.memory.MemoryManagertask
task expireWorkers {
doFirst {
long freeMemory = 0
def memoryManager = services.get(MemoryManager.class)
gradle.addListener(new TaskExecutionListener() {
void beforeExecute(Task task) {
}
void afterExecute(Task task, TaskState state) {
println "Freeing up memory"
memoryManager.requestFreeMemory(freeMemory * 2)
}
})
memoryManager.addListener(new OsMemoryStatusListener() {
void onOsMemoryStatus(OsMemoryStatus osMemoryStatus) {
freeMemory = osMemoryStatus.freePhysicalMemory
}
})
}
}
Our pipeline buffers events and does an external fetch (for enrichment) every 500 events. When a timer is fired, these events are then processed when a timer fires. Of course, when you have e.g. 503 events, there will be 3 events that were not enriched.
From experiments we learned that #FinishBundle is always called before the timer. It even seems the result of the bundle in committed before the timer executed (checkpointing?). If we could access the state from #FinishBundle and perform an enrichment on these last events, they would be part of the committed bundle.
I believe this would solve our exactly-once problem: currently the timer also needs to fetch and will do this again upon re-execution. When we could adjust the state in the #FinishBundle the fetch is no longer needed and when the timer re-executes it will start from the state.
Apparently, it is not possible to access the state from the #FinishBundle function, as the following gives errors:
#FinishBundle
public void finishBundle(FinishBundleContext c,
#StateId("buffer") BagState<AwesomeEvent> bufferState) {
LOG.info("--- FINISHBUNDLE CALLED ---");
// TODO: ENRICHMENT
LOG.info("--- FINISHBUNDLE DONE ---");
}
Am I doing something wrong or is there another way of accessing the state from this function?
Also, am I making the correct assessment about the timer behavior?
I am using the Serverless Framework to consume messages from SQS. Some of the messages sent to the queue do not get consumed. They go straight to the in-flight SQS status and from there to my dead letter queue. When I look at my log of the consumer, I can see that it consumed and successfully processed 9/10 messages. One is always not consumed and ends up in the dead letter queue. I am setting reservedConcurrency to 1 so that only one consumer can run at a time. The function consumer timeout is set to 30 seconds. This is the consumer code:
module.exports.mySQSConsumer = async (event, context) => {
context.callbackWaitsForEmptyEventLoop = false;
console.log(event.Records);
await new Promise((res, rej) => {
setTimeout(() => {
res();
}, 100);
});
console.log('DONE');
return true;
}
Consumer function configuration follow:
functions:
mySQSConsumer:
handler: handler.mySQSConsumer
timeout: 30 # seconds
reservedConcurrency: 1
events:
- sqs:
arn: arn:aws:sqs:us-east-1:xyz:my-test-queue
batchSize: 1
enabled: true
If I remove the await function, it will process all messages. If I increase the timeout to 200ms, even more messages will go to straight to the in-flight status and from there to the dead letter queue. This code is very simple. Any ideas why it's skipping some messages? The messages that don't get consumed don't even show up in the log using the first console.log() statement. They seem entirely ignored.
I figured out the problem. The SQS queue Lambda function event triggering works differently than I thought. The messages get pushed into the Lambda function, not pulled by it. I think this could be engineered better by AWS, but it's what it is.
The issue was the Default Visibility Timeout set to 30 seconds together with Reserved Concurrency set to 1. When the SQS queue gets filled up quickly with thousands of records, AWS starts pushing the messages to the Lambda function at a rate that is faster than the rate at which the single function instance can process them. AWS "assumes" that it can simply spin up more instances of the Lambda to keep up with the backpressure. However, the concurrency limit doesn't let it spin up more instances - the Lambda function is throttled. As a result, the function starts returning failure to the AWS backend for some messages, which will, consequently, hide the failed messages for 30 seconds (the default setting) and put them back into the queue after this period for reprocessing. Since there are so many records to process by the single instance, 30 seconds later, the Lambda function is still busy and can't process those messages again. So the situation repeats itself and the messages go back to invisibility for 30 seconds. This repeats total 3 times. After the third attempt, the messages go to the dead letter queue (we configured our SQS queue that way).
To resolve this issue, we increased the Default Visibility Timeout to 5 minutes. That's enough time for the Lambda function to process through most of the messages in the queue while the failed ones wait in invisibility. After 5 minutes, they get pushed back into the queue and since the Lambda function is no longer busy, it will process most of them. Some of them have to go to invisibility twice before being successfully processed.
So the remedy to this problem is either increasing the Default Invisibility Timeout like we did or increasing the number of failures necessary before a message goes to the dead letter queue.
I hope this helps someone.
I'm developing a Java EE 7 application on wildfly 8.2 and need to run a periodic background task. I inject an executor service and schedule a task, this part is working fine:
#Resource
private ManagedScheduledExecutorService executorService;
...
executorService.scheduleWithFixedDelay(() -> {
try {
// do some stuff
} catch (Throwable t) {
log.error("Error", t);
}
}, 0, 1, TimeUnit.MINUTES);
Now the (actually nice) feature is that upon redeploy the scheduled task is saved and therefore is still scheduled in the new deployment.
But how can I detect if the task is already scheduled to avoid scheduling it multiple times?
I tried to use a ScheduledFutureand cancel the task on #PreDestroy
and #PrePassivate
reloadTreeFuture = executorService.scheduleWithFixedDelay(() -> {
...
#PreDestroy
#PrePassivate
protected void shutdown() {
reloadTreeFuture.cancel(true);
}
This is working fine as long as the corresponding task is not executing at the very moment the cancel is fired. Since the task is long running and running frequently the chance of hitting it in the middle of an execution is somewhat high.
If the cancel is fired while the task is still executing the cancel seems to do nothing. It immediatly returns and the method ScheduledFuture.isDone() also returns true but from the logs I can see the task is still executing in the background until it hits a point where it needs an injected Bean which is not available due to the undeployment process. The process then ends with org.jboss.msc.service.ServiceNotFoundException - but is still scheduled.
reloadTreeFuture.cancel(true);
while (!reloadTreeFuture.isDone()) {
Thread.sleep(200); // I know this is bad - it's just for testing
}
So basic question: how can I make sure the task is not scheduled twice (or even more)?
You could write down the id of each task and check them before executing. Of course this may be not a best solution, but it works.
What is the right approach when writing celery tasks that communicate with service that have a rate limits and sometimes is missing (not responding) for a long time of period?
Do I have to use task retry? What if service is missing too much time? Is there a way to store this tasks for later execution after a long time period?
What if this is a subtask in a long task?
First, i suggest you to set a socket timeout for avoid long waiting of a response.
Than you can catch the socket TimeOutException and in this particoular case to a retry with big amount of time, like 15 minutes.
Anyway normally I use an incrementalRetry with a percentage of increment, this will increase the time every time the task retry, this is useful when you write task that depends on external services that can be available for long time.
You can set on task an high number of retry like 50 and than set the standard retry time by using the var
#20 seconds
self.default_retry_delay = 20
after you can implement a method like this for your task
def incrementalRetry(self, exc, perc = 20, args = None):
"""By default the retry delay is increased by 20 percent"""
if args:
self.request.args = args
delay = self.default_retry_delay
if self.request.kwargs.has_key('retry_deleay'):
delay = self.request.kwargs['retry_deleay']
retry_delay = delay+round((delay*perc)/100,2)
#print "delay"+str(retry_delay)
self.retry(self.request.args,
self.request.kwargs.update({'retry_deleay':retry_delay}),
exc=exc,countdown=retry_delay, max_retries=self.max_retries)
What if this is a subtask in a long task?
If you don't need the result you can launch it in async mode by using task.delay(args=[])
A nice feature is also the task group that allow you to launch different tasks and after all are finished you can to something else in you work flow.