We are trying out AWS SWF, our decider. The activity is very simple, just make a decision based on the event, then call the API to do something. However, after running the decider and activity workers for a while, a few hours a day, we found the time it takes for a workflow to go from DecisionTaskStarted to DecisionTaskCompleted or ActivityTaskStarted to ActivityTaskCompleted can take up to 5 minutes plus. We were also seeing activity taking 30 minutes plus or more to go from started to completed when the activity execution (actual business logic) only takes 100ms to finish.
Where should I start looking into this problem or is this an expected behavior? The load of the system is pretty light.
Related
(Cross-posted on Electrical Engineering Stack Exchange)
I'm using FreeRTOS in an application which requires the processor to sleep in low power mode for a long time (as long as 12 hours), then wake up in response to an interrupt request. I'm having a problem with the amount of time taken for FreeRTOS to wake up.
Before going to sleep, I disable the scheduler through a call to vTaskSuspendAll().
On waking, I calculate the amount of time that the processor has been asleep, and update FreeRTOS via a call to vTaskJumpTime(). I then make a call to xTaskResumeAll() to restart the scheduler.
The issue I have is that xTaskResumeAll() makes a call to xTaskIncrementTick() once for each tick that has been missed while the processor was asleep (recorded through vTaskJumpTime()). It takes about 12 seconds to wake after an hour asleep (3,600,000 calls to xTaskIncrementTick()).
Much as I'm tempted to modify the FreeRTOS xTaskIncrementTick() function, so that it can jump a number of ticks in one call, experience says that I would be smart to look for a standard way first.
Has anyone found another way to implement this behavior which doesn't result in the long wake-up delay?
This is implemented on an Microchip/Atmel SAM4L (Cortex-M4). I use the AST sub system to record the time that the processor is asleep. The sleep is implemented through BPM sub system bpm_sleep() in RETENTION mode.
FreeRTOS's built in low power support uses vTaskStepTick() to jump the tick count forward in one go. It can do that because it won't calculate a wake time (the time at which it leaves low power mode) past the time it knows a task must unblock because a timeout expired. If you remain in sleep mode past the time a task should have unblocked because its timeout expired you have an error anyway. The best place to get FreeRTOS support is it's dedicated support forum.
We need to process a lot of data in the background for iOS and came across BGProcessingTaskRequest which is intended for tasks which may take minutes. However in practice, the task is always killed after exactly 295 seconds on my iOS 14.8 device.
Is this the most amount of background processing time we can expect on iOS or are there any other ways to increase the background execution time? If not, is it possible to chain the requests by scheduling the task again in it's own handler?
As you know, they only promise that it “can run for minutes”. One would have hoped that setting the task’s requiresExternalPower, might increase the allotted time as the battery concerns are eliminated, but in my tests, the allotted time is still limited to roughly 5 minutes (tested in iOS 15).
As you suggest, you can schedule another processing request when time expires. In my experiments, though, this subsequent request took even longer before it started and expired even more quickly.
In short, you can chain requests as you suggest, but you are at the mercy of the OS.
I have an activity function which will:
read a bunch of results from a process elsewhere in my pipeline, via azure blob storage
perform some processing on each result, using the fan-out, fan-in pattern
Combine the original results with the newly processed data, and re-upload to blob storage
When I have a large number of results (~10,000) - I'm experiencing large delays between finishing my processing, and the upload activity actually triggering.
I'm seeing the processing take ~3-4 minutes, then my "PersistResults" activity is scheduled - then 10 minutes later, the "PersistResults" activity actually runs, and takes about 20 seconds to run.
My guess is that the large payload on the "Persist" activity function is slowing it down a lot - though I've read nothing about limiting that, and the documentation certainly implies that any actual work I do (e.g. storing results) should be done in the activities to keep the orchestrator deterministic.
Actually uploading the results to storage appears to be fairly quick, given that when my activity does run, it only takes 20 seconds.
The final payload (all combined results) is roughly 50MB uncompressed - I notice that within the durable functions storage account it uses a compressed version for the activity input.
Is this significant delay expected when using durable functions like this?
Is there anything I've likely done to cause such a delay?
Is there anything I can do to prevent it?
I am analysing an app's slow performance using iOS Instruments. To load a login page it takes around 25 seconds. In Instruments, the timer shows 25 seconds to load the page. But when I sum the running times of the call tree, It is just around 4 seconds only. I want to know where the slowness is occuring. Is there anyway to force instruments to show all the time taken in call tree?
Note: I tried Xamarin profiler also. It shows maximum time taken by any call as 1E-06 ms. Is there any way to know the time taken by the whole method?
Have you considered using the Stopwatch class? It is supported in Project Core Libraries and can be used in a high-resolution mode for higher accuracy. It would allow you to time the execution of a particular method (which sounds like what you are attempting to accomplish). You can find Microsoft documentation and examples here.
I have an MVC 4 app which is performing a slow request roughly every 45 seconds. It's normally 200ms and all requests will be 200ms for about 45 seconds before there is one for about 4500ms. Then back to 200ms.
I have no caching set up.
I am however using a PrincipalContext to query AD.
I was wondering if anyone had any idea before I start setting up some profiling on the calls.
quick:
slow:
Thanks Pinch, but the problem was Active Domain and PrincipalContext. See the solution here, I turned off NetBIOS over TCP/IP and everything is running much better now.
Though I was displaying menu items based on the roles a user has, perhaps I'll cache these somewhere so I don't have to query AD every request.