I have an app that's crashing due to too many "thread wakeups". For example:
45004 wakeups over the last 220 seconds (205 wakeups per
second average), exceeding limit of 150 wakeups per second over 300 seconds
This is difficult to debug because I know of no direct way to measure thread wakeups. The closest I've found is an Instruments template called System Trace that will show you number of blocked thread events. Presumably, this is closely related since a blocked thread means that that thread will sleep and then wake up when it becomes unblocked.
The weird thing about this is that the number of blocked threads is in the 10,000's range per second when the app is running normally and doesn't crash. My assumption is that a blocked, sleeping thread only counts towards your "wakeups" limit in certain circumstances - e.g. I would expect that a thread that is locked due to a mutex lock counts, whereas the OS simply transitioning to other threads in normal operation doesn't.
It would be amazing to me if Instruments had a Thread Wakeups template. The only documentation I can find is here - https://developer.apple.com/library/content/technotes/tn2151/_index.html:
The exception subtype WAKEUPS indicates that threads in the process are being woken up too many times per second, which forces the CPU to wake up very often and consumes battery life.
Typically, this is caused by thread-to-thread communication (generally using peformSelector:onThread: or dispatch_async) that is unwittingly happening far more often than it should be. Because the sort of communication that triggers this exception is happening so frequently, there will usually be multiple background threads with very similar Backtraces - indicating where the communication is originating.
Here’s some Objective-C code based on Ivan’s answer you can copy + paste somewhere into your project (e.g. your applicationDidFinishLaunching: method) to log the number of wakeups per second (works on Mac and iOS):
#include <mach/task.h>
#include <mach/mach.h>
...
__block NSUInteger lastWakeups = 0;
[NSTimer scheduledTimerWithTimeInterval:1.0 repeats:YES block:^(NSTimer * _Nonnull timer) {
struct task_power_info info = {0};
mach_msg_type_number_t count = TASK_POWER_INFO_COUNT;
kern_return_t ret = task_info(current_task(), TASK_POWER_INFO, (task_info_t)&info, &count);
if (ret == KERN_SUCCESS) {
NSUInteger wakeups = info.task_interrupt_wakeups + info.task_timer_wakeups_bin_1 + info.task_timer_wakeups_bin_2;
NSLog(#"WAKEUPS: %lu per second", (unsigned long)(wakeups - lastWakeups));
lastWakeups = wakeups;
} else {
NSLog(#"Error: unable to get CPU wakeups (%d)", ret);
}
}];
Please, take a look here https://developer.apple.com/forums/thread/124180 There is a description of a code of getting wakeup count in your app, not only in the instrument. May help you:
#include <mach/task.h>
#include <mach/mach.h>
BOOL GetSystemWakeup(NSInteger *interrupt_wakeup, NSInteger *timer_wakeup) {
struct task_power_info info = {0};
mach_msg_type_number_t count = TASK_POWER_INFO_COUNT;
kern_return_t ret = task_info(current_task(), TASK_POWER_INFO, (task_info_t)&info, &count);
if (ret == KERN_SUCCESS) {
if (interrupt_wakeup) {
*interrupt_wakeup = info.task_interrupt_wakeups;
}
if (timer_wakeup) {
*timer_wakeup = info.task_timer_wakeups_bin_1 + info.task_timer_wakeups_bin_2;
}
return true;
}
else {
if (interrupt_wakeup) {
*interrupt_wakeup = 0;
}
if (timer_wakeup) {
*timer_wakeup = 0;
}
return false;
}
}
Also there you can find some reasons why wakeups occur too much times.
The "Energy Efficiency Guide for Mac Apps" at the end mentions a command-line utility called timerfires that can be used to see what is causing wakeups.
However, the utility seems to be outdated on macOS 12 Monterey, as I was getting errors like the following when I first tried to run it:
probe description fbt::thread_dispatch:entry does not match any probes
I had to copy the utility and edit it to remove all the DTrace methods that are no longer available to get the tool working.
Once that was done, the tool will show each timer invocation, which is very helpful to track down timers in order to reduce wakeups.
Related
How can I avoid the Soft WDT reset Error in this loop.
The Error consistently occurs when reaching the number 3190.
unsigned long TimeFrame = 10000;
void setup() {
Serial.begin(9600);
}
void loop() {
unsigned long StartTime = millis();
while (millis() - StartTime <= TimeFrame){
Serial.println(millis() - StartTime);
}
}
I could count 4 times to 2500 but would this be the correct approach to this error?
Thanks for the explanation. I added a delay(10) to the code and it works.
void setup() {
Serial.begin(9600);
}
void loop() {
unsigned long StartTime = millis();
while (millis() - StartTime <= TimeFrame){
Serial.println(millis() - StartTime);
delay(10);
}
}
WDT is the "watchdog timer". Watchdog timers are used to get control back when something goes wrong in a system - say, an infinite loop or some other unexpected condition. When the underlying system gets control back it resets these timers so that they start counting up from zero again. When they hit their maximum value they trigger a hardware reset on the chip.
Your code measures the duration of the ESP8266 software watchdog timer - in this case 3.19 seconds.
The ESP8266 has both hardware and software watchdog timers. loop() isn't intended to run indefinitely - it's intended to do a small amount of work and then return. When it returns, the ESP8266 SDK gets to reset the watchdog timers.
Both the delay() and yield() functions give the SDK a chance to say "things are okay" and reset the timers. If you need to have long running code in loop() you should call one of those occasionally to give the rest of the system a chance to run.
Keeping loop() brief isn't just about the watchdog timer, either. It also gives the network stack a chance to run and do processing that it needs to do.
You should always design your program so that loop() does a small batch of repetitive processing and then returns. It should never contain an infinite loop or long loop of code.
For instance, suppose you need to do something every 20 seconds. This is the wrong way to do it:
void loop() {
unsigned long start = millis();
while(millis() - start < 20*1000) ;
do_something();
}
That breaks the way the software is designed to work - with loop() executing briefly. It doesn't allow any other software to run while it's waiting. The watchdog timer will fire and reset your CPU.
This is better:
void loop() {
delay(20*1000);
do_something();
}
because delay() lets the underlying system get control back, reset the watchdog timer and also do network-related processing.
In my opinion, this is best:
static unsigned long start_time;
void setup() {
start_time = millis();
}
void loop() {
if(millis() - start_time > 20*1000) {
do_something();
start_time = millis();
}
}
because it does the least possible work inside loop(), only when it's time to do the work.
I just got a memory leak in my code after I updated my iPad to iOS9, which worked fine on iOS8 and iOS7.
I have an anonymous thread created by the following code:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
[self threadWork];
});
And the thread does a pair of malloc/free call like this:
- (void)threadWork {
// Create a serial queue.
dispatch_queue_t mySerialQueue = dispatch_queue_create("myQueue", NULL);
while (1) {
// Do a simple malloc.
int *foo = (int *)malloc(1024);
// Do free in serial queue.
dispatch_async(mySerialQueue, ^{
free(foo);
});
[NSThread sleepForTimeInterval:1.0 / 60.0];
}
}
This routing will keep the memory usage increasing and finally crashes device on iOS 9. The problem also happened on new/delete in Objective-C++.
I found some other way to do this without memory leak:
Use main queue or global queue to instead the serial queue.
Create concurrent queue instead the serial queue.
Use [NSThread detachNewThreadWithSelector:toTarget:withObject:] to create the thread instead GCD.
I don't understand why this simple routing causes this problem.
I've searched this on google but found nothing.
How can I do this with keeping serial queue and GCD anonymous thread?
Update:
I tried to put NSLog commands in my code to figure out when will the malloc/free be called. The result shows that both of them are called immediately and come in pair. I also tried to slow the thread down to once per second, but the problem still here.
The test code of thread:
- (void)threadWork {
uint64_t mallocCount = 0;
__block uint64_t freeCount = 0;
dispatch_queue_t mySerialQueue = dispatch_queue_create("MyQueue", NULL);
while (1) {
void *test = malloc(1024);
NSLog(#"malloc %llu", ++mallocCount);
dispatch_async(mySerialQueue, ^{
free(test);
NSLog(#"free %llu", ++freeCount);
});
[NSThread sleepForTimeInterval:1.0];
}
}
The console result:
...
2015-10-23 09:51:33.876 OS9MemoryTest[759:153135] malloc 220
2015-10-23 09:51:33.876 OS9MemoryTest[759:153133] free 220
2015-10-23 09:51:34.877 OS9MemoryTest[759:153135] malloc 221
2015-10-23 09:51:34.878 OS9MemoryTest[759:153133] free 221
2015-10-23 09:51:35.883 OS9MemoryTest[759:153135] malloc 222
2015-10-23 09:51:35.883 OS9MemoryTest[759:153133] free 222
I think I've found a better way to do this without leak problem rather than using dispatch_sync.
The point seems to be the setting of Quality of Service (QoS) class of serial queue.
Doing free in a queue which have QOS_CLASS_UNSPECIFIED QoS class causes this problem.
In my question, I free memory in a serial queue which was created by the following call :
dispatch_queue_t mySerialQueue = dispatch_queue_create("MyQueue", NULL);
Its QoS setting is QOS_CLASS_UNSPECIFIED which causes this problem.
If create a serial queue with dispatch_queue_attr_t object, which have QoS setting excepted QOS_CLASS_UNSPECIFIED, the code runs perfectly without leaking:
- (void)threadWork {
// Create a serial queue with QoS class.
dispatch_queue_attr_t attr = dispatch_queue_attr_make_with_qos_class(DISPATCH_QUEUE_SERIAL, QOS_CLASS_DEFAULT, 0);
dispatch_queue_t mySerialQueue = dispatch_queue_create("myQueue", attr);
while (1) {
// Do a simple malloc.
int *foo = (int *)malloc(1024);
// Do free in serial queue.
dispatch_async(mySerialQueue, ^{
free(foo);
});
[NSThread sleepForTimeInterval:1.0 / 60.0];
}
}
I still don't understand why this problem would happened on iOS9,
but setting the QoS seems to make things work.
I need to write simple watchdog which could detect deadlocks in iOS.
I already implemented simple one using GCD dispatch source working as a timer which increments a counter (dispatching on main thread).
This works quite nice for detecting if main thread did stuck but I would like to improve it by being able to detect deadlock on any thread.
My idea is for periodically loop throughout all threads and compare thread backtraces between checks, but I am not sure it it is good approach:
PSEUDO-CODE:
const task_t this_task = mach_task_self();
const thread_t this_thread = mach_thread_self();
kern_return_t kr;
thread_act_array_t threads;
mach_msg_type_number_t thread_count;
kr = task_threads(this_task, &threads, &thread_count);
for (mach_msg_type_number_t i = 0; i < thread_count; i++) {
// 0. get thread:
thread_t thread = threads[i];
// 1. do not analyze this thread (it is an watchdog thread):
if (this_thread == thread)continue;
// 2. suspend thread for a moment:
if((kr = thread_suspend(thread)) != KERN_SUCCESS)continue;
// 3. get backtrace:
int backtraceLength = ksbt_backtraceThread(thread, (uintptr_t*)backtrace, sizeof(backtrace));
// 4. todo - compare.... (HOW TO???)
Anyone can help if comparing backtraces is good approach to detect if thread did stuck or is there better POSIX method?
Does iOS have any kind of very low level condition lock that does not include locking?
I am looking for a way to signal an awaiting thread from within the Core Audio render thread, without the usage of locks. I was wondering if something low level as a Mach system call might exist.
Right now I have a Core Audio thread that uses a non-blocking thread safe message queue to send messages to another thread. The other thread then pulls every 100ms to see if messages are available in the queue.
But this is very rudimentary and the timing is awful. I could use condition locks, but that involves locking, and I would like to keep any kind of locking out of the rendering thread.
What I am looking for is having the message queue thread wait until the Core Audio render thread signals it. Just like pthread conditions, but without locking and without immediate context switching? I would like the Core Audio thread to complete before the message queue thread is woken up.
Updated
A dispatch_semaphore_t works well and is more efficient than a mach semaphore_t. The original code looks like this using a dispatch semaphore:
#include <dispatch/dispatch.h>
// Declare mSemaphore somewhere it is available to multiple threads
dispatch_semaphore_t mSemaphore;
// Create the semaphore
mSemaphore = dispatch_semaphore_create(0);
// Handle error if(nullptr == mSemaphore)
// ===== RENDER THREAD
// An event happens in the render thread- set a flag and signal whoever is waiting
/*long result =*/ dispatch_semaphore_signal(mSemaphore);
// ===== OTHER THREAD
// Check the flags and act on the state change
// Wait for a signal for 2 seconds
/*long result =*/ dispatch_semaphore_wait(mSemaphore, dispatch_time(dispatch_time_now(), 2 * NSEC_PER_SEC));
// Clean up when finished
dispatch_release(mSemaphore);
Original answer:
You can use a mach semaphore_t for this purpose. I've written a C++ class that encapsulates the functionality: https://github.com/sbooth/SFBAudioEngine/blob/master/Semaphore.cpp
Whether or not you end up using my wrapper or rolling your own the code will look roughly like:
#include <mach/mach.h>
#include <mach/task.h>
// Declare mSemaphore somewhere it is available to multiple threads
semaphore_t mSemaphore;
// Create the semaphore
kern_return_t result = semaphore_create(mach_task_self(), &mSemaphore, SYNC_POLICY_FIFO, 0);
// Handle error if(result != KERN_SUCCESS)
// ===== RENDER THREAD
// An event happens in the render thread- set a flag and signal whoever is waiting
kern_return_t result = semaphore_signal(mSemaphore);
// Handle error if(result != KERN_SUCCESS)
// ===== OTHER THREAD
// Check the flags and act on the state change
// Wait for a signal for 2 seconds
mach_timespec_t duration = {
.tv_sec = 2,
.tv_nsec = 0
};
kern_return_t result = semaphore_timedwait(mSemaphore, duration);
// Timed out
if(KERN_OPERATION_TIMED_OUT != result)
;
// Handle error if(result != KERN_SUCCESS)
// Clean up when finished
kern_return_t result = semaphore_destroy(mach_task_self(), mSemaphore);
// Handle error if(result != KERN_SUCCESS)
I'm seeing an issue wherein CFReadStreamRead, as part of a streamed file upload, never returns.
This seems to happen only on iOS7 — and far more often when debugging against a physical device than in the simulator — or at least, it's far more evident there.
We have an HTTP (or HTTPS, the problem occurs either way, with a locally-hosted or remote server) POST of a file, via straight-line, blocking (non-event-driven) CFNetwork calls. It's a necessity of the C code calling this handler; there's no provision for callbacks.
That's well and good, the network calls are happening in background threads and/or via async dispatch.
The network code in question boils down to (removing error handling for brevity):
CFReadStreamRef upload = CFReadStreamCreateWithFile(
kCFAllocatorDefault, upload_file_url);
CFRelease(upload_file_url);
CFReadStreamOpen(upload);
CFReadStreamRef myReadStream = CFReadStreamCreateForStreamedHTTPRequest(
kCFAllocatorDefault, myRequest, upload);
CFReadStreamOpen(myReadStream);
CFIndex numBytesRead = CFReadStreamRead(myReadStream, buf, sizeof(buf));
// etc.
On its own, this code wants to hang immediately under iOS7. If I add a loop with some calls to usleep before it (checking CFReadStreamHasBytesAvailable along the way), it will almost always succeed. Every few hundred tries, it will still fail, never returning. Again, the main thread is unaffected.
I'd hoped the GM would clear up this behavior, but it's still present.
Adding a runloop/callback method to watch for bytes-available events has no effect - when the call hangs, no events are seen, either.
Any suggestions as to why this is happening, or how it can be prevented? Anyone else seeing different CFReadStream behavior under iOS 7?
I've try such nasty workaround and it works for me, problem is that I'm requesting delta values from server, so if something goes wrong I'm just fetching new delta value with, in general case it will not work (in logs I see that timeout kicks in sometimes). At least this prevents form permanent thread blocking and gives a chance to handle somehow this problem:
NSInteger readStreamReadWorkaround(CFReadStreamRef readStrem, UInt8 *buffer, CFIndex bufferLength) {
static dispatch_once_t onceToken;
static BOOL isProblematicOs = YES;
dispatch_once(&onceToken, ^{
isProblematicOs = [[UIDevice currentDevice].systemName compare: #"7.0" options: NSNumericSearch]!=NSOrderedAscending;
});
NSInteger readBytesCount = -2;
if (isProblematicOs) {
CFStreamStatus sStatus = CFReadStreamGetStatus(readStrem);
NSDate *date = [NSDate date];
while (YES) {
if(CFReadStreamHasBytesAvailable(readStrem)) {
readBytesCount = CFReadStreamRead(readStrem, buffer, bufferLength);
break;
}
sStatus = CFReadStreamGetStatus(readStrem);
if (sStatus!=kCFStreamStatusOpen && sStatus !=kCFStreamStatusAtEnd
|| [date timeIntervalSinceNow]<-15.0) {
break;
}
usleep(50000);
}
} else {
readBytesCount = CFReadStreamRead(readStrem, buffer, sizeof(buffer));
}
return readBytesCount;
}
I don't like this solution but so far I don't see an alternative.