I've created OOM crashes by growing an infinitely large NSArray of NSStrings, and I've even tried calling exit(0) just to make it look like an OOM. While these things to have worked to terminate the app unexpectedly, I don't see any OOMs reported on Crashlytics and it doesn't call the delegate callback, crashlyticsDidDetectReportForLastExecution:, on the next run of the app. I'm running the app on a real device that is not connected to a simulator, and any other kind of crash/error it reports fine. Does anyone have any idea what the issue might be?
Mike from Fabric here. We chatted over Fabric support as well, but to clarify for all. OOMs are detected on the server, but are processed in batch at the end of a current UTC day and will be visible in your Fabric Crashlytics' dashboard starting the next UTC day after our processing completes.
Related
Recently I had an issue in my iOS app that caused the app to freeze. The UI was unresponsive and the user had to kill and restart the app to make it work again.
In Google's Play Console I see such issues as ANRs ('Application not responding' errors).
But in Xcode I don't get any reports unfortunately. Crashlytics also didn't show any reports.
Is it possible to somehow get some kind of report with a stacktrace similar to the ANRs in Google's Play Console?
Please note:
I know what caused the app to freeze in this specific issue and also how to debug and fix it. What I
want is an error report in case the issue wasn't found during QA but it is
occurring in a user session.
The freezing version was not published to App Store, only to
TestFlight.
The reason for unresponsive UIs could be due to a number of reasons:
One reason could be because of high load (running out of RAM, Swap Space). For detecting these scenarios (without crashing behavior) you need Application Performance Monitoring, for example with Firebase. The advantage with Firebase is you don't have to prompt for consent, so you get more data to detect hangs. The Xcode Organizer can provide Hang Rate metrics though if you have enough users who have consented to share their data (new in Xcode 11).
Another reason could be due to logical errors in code. For example, a user triggers an unexpected app flow that was missed during manual testing. In these instances, it can be harder to narrow down but could be trackable with manual user testing beforehand e.g TestFlight. You can use In-App Feedback tools (users send you emails) to gather context about the issue or Instabug for screenshots and advanced feedback.
The other reason could be because of deadlocks. If you use DispatchGroup for instance or DispatchSemaphores to queue network requests and the requests time out then that could also lead to suspended UI frames. I would look carefully at where you leave the dispatch group as well - all possible branches should have a dispatch_group_leave statement to avoid hanging on dispatch_group_wait.
You can use the Thread State Trace Instrument to see whether system calls are blocking your main thread to debug this issue. The classic iOS dev advice is to only perform UI updates on the main thread and move all other logic to a background thread. This may not solve the root cause but does prevent hangs. The same logic applies for Core Data fetches, inserts, deletes, and updates. Move Core Data operations to background threads in network request callbacks as well.
If you get crash reports, it's simple to diagnose an OOM error in a Crashlytics crash report for example (comes up in Insights). Heap corruption can also be inferred from crash reports, and while the metadata may not be the same for the same behavior, it could point you to a screen or action when you try and reproduce the bug.
Watchdog Unresponsive App crashes can be intercepted to come up in crash reports if the user doesn't kill the app beforehand, but you mentioned the user killed the app so this may not be available.
There is no direct equivalent ANR error reporting for iOS apps, given that Hang Rates are only aggregated if enough sessions have been detected. You could do screen recording of app sessions (ServiceNow), but I don't advise it because it has privacy implications.
Crashlytics is unable to send abnormal terminations fired by the watchdog. I've researched this for Jetsam terminations. You will see them in the console logs. see this link for the watchdog crash iOS generates: Watchdog Unresponsive App. If there's a way to monitor the main thread for queued operations executing for too long, you could fire a fatal error ie fire a fatal before the watchdog terminates the app.
Do some research on MetricKit. It's new to iOS 13. There's a class called MXHangDiagnostic. You can use it in your app to monitor for hangs and report on them as you see fit.
It is not really a question but I wanted to write it down here so other people facing the problem could find the solution.
Since the release of iOS 13, some of our users were contacting us telling that our app was resetting each time they go into background for a medium amount of time (approximatively 30sec).
Of course, absolutely no clue of the behavior, nothing on Crashlytics etc. We were able to reproduce on a QA device but unable to reproduce on a debugger attached device.
So obviously we used the Console to monitor what's going on and we noticed that a weird message was being sent after approx. 25sec after our app was put into background:
default 10:14:01.658579+0100 XXX RunningBoardServices Received process assertions expiration warning!
default 10:14:01.659214+0100 XXX RunningBoardServices Notifying client of imminent expiration of assertion
default 10:14:01.659568+0100 XXX RunningBoardServices Expiration notification complete
If you were fast enough to relaunch the app, everything just worked fine, but if you left the app inactive for another 5sec (approx.) and then relaunched it, then the app was resetting (appDidFinishLaunching...).
...
...
We dug deeper and deeper without finding anything... until we decided to update some of our external libraries through Cocoapods, and guess what? we found our guilty guy: GoogleDataTransport from the Firebase suite!
We were using the 1.1.3 version and updating to the 3.0.1 fixed the issue. I guess they were doing some "bad" stuff in the background...
That's it guys. I hope it will help some others!
Cheers.
EDIT:
From this page (Firebase release notes) https://firebase.google.com/support/release-notes/ios#6.11.0, you can read:
Fixed race condition that prevented upload from completing while app was in the background.
Version 6.10.0 - October 8, 2019
From what I understand, maybe Firebase was not notifying iOS that the upload they were doing in the background was completed. It could explain why iOS 13 kills an app in this case.
🤷‍♂️
EDIT 2:
Apple apparently released an iOS update (13.2.2) which is supposed to solve these background issues: https://twitter.com/engadget/status/1192512171252551682?s=12.
Fixes an issue that could cause apps to quit unexpectedly when running in the background
We tested it using an old app which does not include our previous fix and unfortunately the problem is still present...
The crash was due to a background task leak. In my application, some SDKs don't use the background task correctly. Because it was killed by the watchdog, debugging will not appear, you can print beginBackgroundTask and endBackgroundTask during debugging. If there is a mismatch, a leak occurs.
I wrote a category to hook background task related methods to avoid leaking being killed by watchdog：https://github.com/ruanjx/MPIBackgroundTaskProtection
More information about background task：https://medium.com/swlh/handling-background-tasks-in-ios-13-67f717d94b3d
Starting from iOS 12 the CLLocationManager doesn't run endless anymore in the background. The app get's terminated without a crashlog at random times. This was working fine before iOS 12.
To demonstrate this problem I've created an sample app here
The demo app just launches an CLLocationManager and keep this running in the background. While running on the background we keep track of it by logging it. The problem is that the app get terminated by iOS. The demo app is created to demonstrate this problem.
Steps to reproduce
Run the app on the device
Grand access to locationmanager
Put the app to the background
Wait for 1-48hours
Result:
The app is terminated
App is terminated without any reason after random time.
Expected result:
The app is still running.
How it should work
This is confirmed by an Apple engineer:
Once the CLLocationManager updates are started in the foreground and you did all the work to have it running in the background, the location updates should run endless in the background until:
app is force quit
device is rebooted
app stops location updates
app releases the CLLocationManager object
app crashes
iOS terminates the app due to memory shortage,
the locationManager object is released, overreleased, or overwritten. You should make sure that your view controller is not being instantiated, which then resets the locationController class. If that happens when the app is in the background, you will restart updates, which will cause the app to be eventually suspended. You should make sure the locationController is a singleton.
app is crashing. Check to see if there are crash logs on the device you are testing
iOS is terminating the app due to memory shortage. In this case, you will find JetsamEvent logs on the device that shows your app being terminated. You can check the timestamps and locate the one that is around the time your app stopped working.
Updated answer:
Apple fixed this bug in iOS 12.2 beta 2 (16E5191d)
Original analyse and bug detection:
Together with Apple Developer Technical Support we have analyzed this issue with Sysdiagnose files. Following these guidelines you can install profiles to have more logging on your device. I don't know exactly how these logs work and where to find this issue, but Apple did this for me and came with this first analyse:
On the suspension event you observed on 2018/10/22 01:01:12:587, this
is what I see (about a minute after your last activity logging)
[CllocationManag:2725] Terminating with description:
{
owner = ;
target = rw.sp.flitsmeister.frameworks.CllocationManagerBackgroundTest; }
This is basically saing that your app was terminated, because the
system needed drive space, and killed a bunch of apps so it can delete
their /tmp and /Library/Caches directories. I have seen this process
to be a little more aggressive in iOS 12, but seeing you are on a
256GB device, and have ~179 GB free after the cleanup, I am finding it
hard to believe this was justified.
After sending some more sysdiagnose and reproduction cases Apple did his best to analyse and ended with the following conclusion:
Unfortunately I don’t bring good news.
It turned out that, currently in iOS 12 there is a new mechanism that
will terminate long running background apps periodically as the system
needs to free resources. At this time, this process is a bit too
aggressive, and I am working with the relevant teams to get this to
behave better.
So, at this point, I would like you to file a bug report. Explain the
symptoms. And make sure you upload the sysdiagnose files along with
the bug report. (I already sent yours in, but it doesn’t hurt to have
the new ones as well). And let me know the bug number please.
So this means that currently in iOS 12 you app won't run endless on the background. I've filled in a bug report, the number is 45581276 and will try to keep this thread updated.
I tried clearing out the /tmp and /Library/Caches directories when the app goes in the background without any change in behavior. I also tried the same periodically when the app processing background locations without any change as well.
All question you can see in title.
I don't know why it crashes when run without Xcode.
How can diagnose this?
How can get logs?
Does startup take longer than 30 seconds sometimes? If you are starting up your app and you are connected to the debugger springboard will let you app take any amount of time to start up. If you are not connected to a debugger, springboard will kill your app after 30 seconds.
If you have physical access to the device you can pull the logs directly from the device via the Xcode organiser when you next plug it in.
Another option, if you don't have physical access to the device would be to add a library to retrieve crash logs for you, such as Crashlytics which will upload the crash log to a server the next time the application is launched.
Generally in C it means you haven't initialised some memory, somewhere. Thus you'll sometimes have the correct conditions for startup, and other times not. A common case is expecting zero as the starting case for other calculations: if you haven't zeroed the block of memory in question, your process and others have (between this run and the last) written and rewritten that memory so that it will sometimes be zero (thus a successful startup) and other times not.
In your case, you could probably use callgrind to check where the crash is occurring.
Currently, I am using Crittercism for crash reporting and making sure that I add dSYMs to get symbolicated crash reports.
But it is not helping with some of my crashes which are segmentation faults (SIGSEGV and SIGBUS). They occur randomly and I haven't been able to reproduce them on device and on simulator. I have tried to find a pattern by trying my app on different ios devices with different network connections (3G, Wifi, Edge) but with no success.
What can be my next step?
Not much to go on, but here are a few places to look:
If you have multiple threads, check to make sure they are behaving properly. Make sure you synchronize properly if multiple threads could be accessing the same objects.
Check your NSNotifications - could one be posted when you are not expecting it?
I have found that the hardest bugs to find are those that are caused by asynchronous events - either in other threads or due to external events that you might be monitoring.
Not being able to reproduce the bug in your development environment will make it very hard to find.