I need to write simple watchdog which could detect deadlocks in iOS.
I already implemented simple one using GCD dispatch source working as a timer which increments a counter (dispatching on main thread).
This works quite nice for detecting if main thread did stuck but I would like to improve it by being able to detect deadlock on any thread.
My idea is for periodically loop throughout all threads and compare thread backtraces between checks, but I am not sure it it is good approach:
PSEUDO-CODE:
const task_t this_task = mach_task_self();
const thread_t this_thread = mach_thread_self();
kern_return_t kr;
thread_act_array_t threads;
mach_msg_type_number_t thread_count;
kr = task_threads(this_task, &threads, &thread_count);
for (mach_msg_type_number_t i = 0; i < thread_count; i++) {
// 0. get thread:
thread_t thread = threads[i];
// 1. do not analyze this thread (it is an watchdog thread):
if (this_thread == thread)continue;
// 2. suspend thread for a moment:
if((kr = thread_suspend(thread)) != KERN_SUCCESS)continue;
// 3. get backtrace:
int backtraceLength = ksbt_backtraceThread(thread, (uintptr_t*)backtrace, sizeof(backtrace));
// 4. todo - compare.... (HOW TO???)
Anyone can help if comparing backtraces is good approach to detect if thread did stuck or is there better POSIX method?
Related
I have an app that's crashing due to too many "thread wakeups". For example:
45004 wakeups over the last 220 seconds (205 wakeups per
second average), exceeding limit of 150 wakeups per second over 300 seconds
This is difficult to debug because I know of no direct way to measure thread wakeups. The closest I've found is an Instruments template called System Trace that will show you number of blocked thread events. Presumably, this is closely related since a blocked thread means that that thread will sleep and then wake up when it becomes unblocked.
The weird thing about this is that the number of blocked threads is in the 10,000's range per second when the app is running normally and doesn't crash. My assumption is that a blocked, sleeping thread only counts towards your "wakeups" limit in certain circumstances - e.g. I would expect that a thread that is locked due to a mutex lock counts, whereas the OS simply transitioning to other threads in normal operation doesn't.
It would be amazing to me if Instruments had a Thread Wakeups template. The only documentation I can find is here - https://developer.apple.com/library/content/technotes/tn2151/_index.html:
The exception subtype WAKEUPS indicates that threads in the process are being woken up too many times per second, which forces the CPU to wake up very often and consumes battery life.
Typically, this is caused by thread-to-thread communication (generally using peformSelector:onThread: or dispatch_async) that is unwittingly happening far more often than it should be. Because the sort of communication that triggers this exception is happening so frequently, there will usually be multiple background threads with very similar Backtraces - indicating where the communication is originating.
Here’s some Objective-C code based on Ivan’s answer you can copy + paste somewhere into your project (e.g. your applicationDidFinishLaunching: method) to log the number of wakeups per second (works on Mac and iOS):
#include <mach/task.h>
#include <mach/mach.h>
...
__block NSUInteger lastWakeups = 0;
[NSTimer scheduledTimerWithTimeInterval:1.0 repeats:YES block:^(NSTimer * _Nonnull timer) {
struct task_power_info info = {0};
mach_msg_type_number_t count = TASK_POWER_INFO_COUNT;
kern_return_t ret = task_info(current_task(), TASK_POWER_INFO, (task_info_t)&info, &count);
if (ret == KERN_SUCCESS) {
NSUInteger wakeups = info.task_interrupt_wakeups + info.task_timer_wakeups_bin_1 + info.task_timer_wakeups_bin_2;
NSLog(#"WAKEUPS: %lu per second", (unsigned long)(wakeups - lastWakeups));
lastWakeups = wakeups;
} else {
NSLog(#"Error: unable to get CPU wakeups (%d)", ret);
}
}];
Please, take a look here https://developer.apple.com/forums/thread/124180 There is a description of a code of getting wakeup count in your app, not only in the instrument. May help you:
#include <mach/task.h>
#include <mach/mach.h>
BOOL GetSystemWakeup(NSInteger *interrupt_wakeup, NSInteger *timer_wakeup) {
struct task_power_info info = {0};
mach_msg_type_number_t count = TASK_POWER_INFO_COUNT;
kern_return_t ret = task_info(current_task(), TASK_POWER_INFO, (task_info_t)&info, &count);
if (ret == KERN_SUCCESS) {
if (interrupt_wakeup) {
*interrupt_wakeup = info.task_interrupt_wakeups;
}
if (timer_wakeup) {
*timer_wakeup = info.task_timer_wakeups_bin_1 + info.task_timer_wakeups_bin_2;
}
return true;
}
else {
if (interrupt_wakeup) {
*interrupt_wakeup = 0;
}
if (timer_wakeup) {
*timer_wakeup = 0;
}
return false;
}
}
Also there you can find some reasons why wakeups occur too much times.
The "Energy Efficiency Guide for Mac Apps" at the end mentions a command-line utility called timerfires that can be used to see what is causing wakeups.
However, the utility seems to be outdated on macOS 12 Monterey, as I was getting errors like the following when I first tried to run it:
probe description fbt::thread_dispatch:entry does not match any probes
I had to copy the utility and edit it to remove all the DTrace methods that are no longer available to get the tool working.
Once that was done, the tool will show each timer invocation, which is very helpful to track down timers in order to reduce wakeups.
So I'm putting 10 tasks on a concurrent queue using dispatch_async. They do not block the next task, and gets processed in order. My UI is responsive.
for (int i = 0; i < 10; i++) {
dispatch_async(concurrencyQueue, ^() {
NSLog(#"..calling insertion method to insert record %d", i);
dispatch_sync(serialQueue, ^() {
//this is to simulate writing to database
NSLog(#"----------START %d---------", i);
[NSThread sleepForTimeInterval:1.0f];
NSLog(#"--------FINISHED %d--------", i);
});
});
}
Within each task, we simulate a write to database with a "1 sec sleep" on a serial Queue via dispatch_sync.
I always thought dispatch_sync blocks everyone, and syncs its tasks because that's how it behaves when I use it individually. However, in this situation, it does not block the main thread. Instead, it runs beautifully in the background like I want it.
Is it because whatever thread is associated with the queue is being affected?
For example, the main thread is executing the concurrent queue via dispatch_async and that's why it is not blocked.
The dispatch_sync only syncs and blocks against the background thread that's working on the concurrent queue. Hence, the dispatch_sync is associated with the background thread, thus never affecting my UI main thread.
Is my thinking correct?
thank you!
You never block the main thread because your code is running on either the threads of the concurrencyQueue or the thread of the serialQueue. None of those are the main thread.
All of the calls to sleep happen one by one on the thread of the serialQueue. So it is the thread of the serialQueue that is blocked.
However, since you dispatch to the serialQueue using dispatch_sync, you are also blocking each thread of the concurrent queue. This would be better pictured if you add another NSLog after the call to dispatch_sync.
for (int i = 0; i < 10; i++) {
dispatch_async(concurrencyQueue, ^() {
NSLog(#"..calling insertion method to insert record %d", i);
dispatch_sync(serialQueue, ^() {
//this is to simulate writing to database
NSLog(#"----------START %d---------", i);
[NSThread sleepForTimeInterval:1.0f];
NSLog(#"--------FINISHED %d--------", i);
});
NSLog(#"..called insertion method to insert record %d", i);
});
}
That 2nd NSLog after the dispatch_sync will show you better how the dispatch_sync is affecting the calls to dispatch_async.
Yes, you are right. dispatch_sync() blocks only the thread the queue is running on.
I have a loop which I run and in each iteration I have a block that I want run on an NSOperationQueue. The underlying queue is serial. This loop could add hundreds of potentially long running block tasks. When I set m_opQueue.suspended = YES the blocks will still keep executing.
I'm well aware that a single block cannot stop right in the middle, but I expected that pausing the NSOperationQueue would simply not execute the next operation until suspended was false.
Can anyone explain whether I'm wrong or how I achieve what I want?
dispatch_queue_t index_queue = dispatch_queue_create("someQueue", DISPATCH_QUEUE_SERIAL);
m_OpQueue = [[NSOperationQueue alloc] init];
m_OpQueue.underlyingQueue = index_queue;
for ( NSUInteger i = 0; i < total; i++ ) {
void (^block)(void) = ^void() {
// Do stuff.
NSLog(#"processing complete.");
};
// Effectively adds a NSBlockOperation.
[m_OpQueue addOperationWithBlock:block];
}
This curious behavior you describe (where previously enqueued operations will continue to start even after the queue has been suspended), is caused by how you created the serial queue.
Generally, you create a serial operation queue by setting maxConcurrentOperationCount:
m_OpQueue.maxConcurrentOperationCount = 1;
If you do that (no need to set underlyingQueue), you see the expected behavior.
I just got a memory leak in my code after I updated my iPad to iOS9, which worked fine on iOS8 and iOS7.
I have an anonymous thread created by the following code:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
[self threadWork];
});
And the thread does a pair of malloc/free call like this:
- (void)threadWork {
// Create a serial queue.
dispatch_queue_t mySerialQueue = dispatch_queue_create("myQueue", NULL);
while (1) {
// Do a simple malloc.
int *foo = (int *)malloc(1024);
// Do free in serial queue.
dispatch_async(mySerialQueue, ^{
free(foo);
});
[NSThread sleepForTimeInterval:1.0 / 60.0];
}
}
This routing will keep the memory usage increasing and finally crashes device on iOS 9. The problem also happened on new/delete in Objective-C++.
I found some other way to do this without memory leak:
Use main queue or global queue to instead the serial queue.
Create concurrent queue instead the serial queue.
Use [NSThread detachNewThreadWithSelector:toTarget:withObject:] to create the thread instead GCD.
I don't understand why this simple routing causes this problem.
I've searched this on google but found nothing.
How can I do this with keeping serial queue and GCD anonymous thread?
Update:
I tried to put NSLog commands in my code to figure out when will the malloc/free be called. The result shows that both of them are called immediately and come in pair. I also tried to slow the thread down to once per second, but the problem still here.
The test code of thread:
- (void)threadWork {
uint64_t mallocCount = 0;
__block uint64_t freeCount = 0;
dispatch_queue_t mySerialQueue = dispatch_queue_create("MyQueue", NULL);
while (1) {
void *test = malloc(1024);
NSLog(#"malloc %llu", ++mallocCount);
dispatch_async(mySerialQueue, ^{
free(test);
NSLog(#"free %llu", ++freeCount);
});
[NSThread sleepForTimeInterval:1.0];
}
}
The console result:
...
2015-10-23 09:51:33.876 OS9MemoryTest[759:153135] malloc 220
2015-10-23 09:51:33.876 OS9MemoryTest[759:153133] free 220
2015-10-23 09:51:34.877 OS9MemoryTest[759:153135] malloc 221
2015-10-23 09:51:34.878 OS9MemoryTest[759:153133] free 221
2015-10-23 09:51:35.883 OS9MemoryTest[759:153135] malloc 222
2015-10-23 09:51:35.883 OS9MemoryTest[759:153133] free 222
I think I've found a better way to do this without leak problem rather than using dispatch_sync.
The point seems to be the setting of Quality of Service (QoS) class of serial queue.
Doing free in a queue which have QOS_CLASS_UNSPECIFIED QoS class causes this problem.
In my question, I free memory in a serial queue which was created by the following call :
dispatch_queue_t mySerialQueue = dispatch_queue_create("MyQueue", NULL);
Its QoS setting is QOS_CLASS_UNSPECIFIED which causes this problem.
If create a serial queue with dispatch_queue_attr_t object, which have QoS setting excepted QOS_CLASS_UNSPECIFIED, the code runs perfectly without leaking:
- (void)threadWork {
// Create a serial queue with QoS class.
dispatch_queue_attr_t attr = dispatch_queue_attr_make_with_qos_class(DISPATCH_QUEUE_SERIAL, QOS_CLASS_DEFAULT, 0);
dispatch_queue_t mySerialQueue = dispatch_queue_create("myQueue", attr);
while (1) {
// Do a simple malloc.
int *foo = (int *)malloc(1024);
// Do free in serial queue.
dispatch_async(mySerialQueue, ^{
free(foo);
});
[NSThread sleepForTimeInterval:1.0 / 60.0];
}
}
I still don't understand why this problem would happened on iOS9,
but setting the QoS seems to make things work.
Does iOS have any kind of very low level condition lock that does not include locking?
I am looking for a way to signal an awaiting thread from within the Core Audio render thread, without the usage of locks. I was wondering if something low level as a Mach system call might exist.
Right now I have a Core Audio thread that uses a non-blocking thread safe message queue to send messages to another thread. The other thread then pulls every 100ms to see if messages are available in the queue.
But this is very rudimentary and the timing is awful. I could use condition locks, but that involves locking, and I would like to keep any kind of locking out of the rendering thread.
What I am looking for is having the message queue thread wait until the Core Audio render thread signals it. Just like pthread conditions, but without locking and without immediate context switching? I would like the Core Audio thread to complete before the message queue thread is woken up.
Updated
A dispatch_semaphore_t works well and is more efficient than a mach semaphore_t. The original code looks like this using a dispatch semaphore:
#include <dispatch/dispatch.h>
// Declare mSemaphore somewhere it is available to multiple threads
dispatch_semaphore_t mSemaphore;
// Create the semaphore
mSemaphore = dispatch_semaphore_create(0);
// Handle error if(nullptr == mSemaphore)
// ===== RENDER THREAD
// An event happens in the render thread- set a flag and signal whoever is waiting
/*long result =*/ dispatch_semaphore_signal(mSemaphore);
// ===== OTHER THREAD
// Check the flags and act on the state change
// Wait for a signal for 2 seconds
/*long result =*/ dispatch_semaphore_wait(mSemaphore, dispatch_time(dispatch_time_now(), 2 * NSEC_PER_SEC));
// Clean up when finished
dispatch_release(mSemaphore);
Original answer:
You can use a mach semaphore_t for this purpose. I've written a C++ class that encapsulates the functionality: https://github.com/sbooth/SFBAudioEngine/blob/master/Semaphore.cpp
Whether or not you end up using my wrapper or rolling your own the code will look roughly like:
#include <mach/mach.h>
#include <mach/task.h>
// Declare mSemaphore somewhere it is available to multiple threads
semaphore_t mSemaphore;
// Create the semaphore
kern_return_t result = semaphore_create(mach_task_self(), &mSemaphore, SYNC_POLICY_FIFO, 0);
// Handle error if(result != KERN_SUCCESS)
// ===== RENDER THREAD
// An event happens in the render thread- set a flag and signal whoever is waiting
kern_return_t result = semaphore_signal(mSemaphore);
// Handle error if(result != KERN_SUCCESS)
// ===== OTHER THREAD
// Check the flags and act on the state change
// Wait for a signal for 2 seconds
mach_timespec_t duration = {
.tv_sec = 2,
.tv_nsec = 0
};
kern_return_t result = semaphore_timedwait(mSemaphore, duration);
// Timed out
if(KERN_OPERATION_TIMED_OUT != result)
;
// Handle error if(result != KERN_SUCCESS)
// Clean up when finished
kern_return_t result = semaphore_destroy(mach_task_self(), mSemaphore);
// Handle error if(result != KERN_SUCCESS)