iOS: Symbolicating crashlogs with exception reasons - ios

Soo, with TestFlight's end it has become even more pressing to be able to fully understand iOS crashlogs.
There are numerous questions here at SO about this topic but in my experience none of the solutions provided enough insight into the crash itself.
There still seems to be quite a confusion about this in the iOS world.
For the sake of completion here are the methods and steps I found and took.
(Disclamer: I had access to everyting: the .app file, the .dSYM and .crash files)
1| symbolicatecrash - command line utility
define the developer directory: export DEVELOPER_DIR="/Applications/Xcode.app/Contents/Developer”
install the utility from its original directory: sudo cp /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKitBase.framework/Versions/A/Resources/symbolicatecrash /usr/bin
put the dSYM file, the .app file and the .crash file in the same folder
symbolicatecrash “FILENAME.crash" "FILENAME.app" "FILENAME.app.dSYM"
2| atos - command line utility
open crash file, get the memory address one line at a time from the stack trace, like so: 8 NAME_OF_MY_APP 0x000ad031 0xa7000 + 24625
and use atos to symbolicate that one line: atos -arch armv7 -o NAME_OF_MY_APP.app/NAME_OF_MY_APP 0x000ad031 0xa7000 + 24625
3| gdb
there is a great post from Jerry Krinock for a third option... but it uses gdb and gdb is no longer available in OS X Mavericks: http://www.cocoabuilder.com/archive/xcode/312725-can-symbolicate-crash-reports.html
Now either I'm dumb, stupid, or both but for me, the problem with all these methods is that they inconsistent.
In my experience, symbolicating with symbolicatecrash will give us the method names for all the stack trace, but no line numbers and little to no info about the exception thrown.
Atos is a bit more descriptive, but still no exception description and you have to manually do this for all the lines you want symbolicated.
Also, atos sometimes points to method calls that are not in output of symbolicatecrash...
When I uploaded a build to TestFlight along with the .dSYM they could give me precise method names, line numbers and exception description that I just cannot find in these symbolications.
Is the exception description even in the crashlog?
Do we have to implement an unhandled exception handler within the app and send the reports "home" (ie to a web server?) to have access to what exception was thrown, where?

symbolicatecrash - command line utility
You are calling the script wrong, it should be:
symbolicatecrash “FILENAME.crash" "NAME_OF_MY_APP.app.dSYM"
atos - command line utility
This call is also wrong:
atos -arch armv7 -l LOAD_ADDRESS_OF_THE_APP -o NAME_OF_MY_APP.app.dSYM 0x000ad031
Where LOAD_ADDRESS_OF_THE_APP is the first address shown in the Binary Images section for your app. See also: iOS crash reports: atos not working as expected
Now regarding the additional questions/remarks:
Line number not appearing using symbolicatecrash:
This is because you passed along the app binary, which mostly has all symbols stripped (due to size) and even if the symbols would not be stripped it would never show line numbers. Always use the dSYM for symbolication if you have it.
Exception description missing:
In most cases, Apple's iOS crash reports do not provide the Application Specific Information block in the reports which would contain that information. If you only have Apples crash reports, there is nothing you can do.
Some additional information:
Symbolication system calls
You will need the symbols of the specific CPU architecture and iOS version the crash report was created with. So you would need at least an armv7 device and an arm64 device with the specific iOS version and having those at least once connected to Xcode that it could import the symbols. Otherwise there will be symbols left unsymbolicated in the report.
Let Xcode symbolicate it
If your system is setup correctly, all you need to do is drag the crash report into the top Devices Logs entry in the Xcode organizer and it will call symbolicatecrash for you and find the correct dSYM via Spotlight. (Requires the dSYMs folder to be indexed by Spotlight)
Alternatives to collect crash reports
There are multiple alternatives out there to collect crash reports. Open Source solutions, free services and paid services that provide SDKs to collect the crash reports in your app and send them to a server, symbolicate and group them automatically. (Disclaimer: I am the developer of QuincyKit Open Source project and Co-Founder of HockeyApp.net service)

Related

Empty dSYM folder in Xcode archive

I've Googled this extensively, and either the situation someone's experiencing involves a different Xcode version (and therefore different build options), or a presence of a dSYM file.
So, here's the situation. I received a crash report through Xcode. It's just memory addresses. Trying to open it in project gives me the same memory addresses. Thought about manual symbolication but for that I need my dSYM file corresponding to the archive I built. But when I 'Show package contents' for the archive I built in finder, its dSYM folder is empty.
My Xcode settings at the time of archiving were:
DEBUG_INFORMATION_FORMAT: DWARF with dSYM File
STRIP_INSTALLED_PRODUCT: Yes Switching to No makes no difference.
DEPLOYMENT_POSTPROCESSING: No Switching to Yes makes no difference.
Also, I selected 'Include symbols' when uploading the archive to iTunes Connect.
The process for deployment with Apple is confusing-as-hell enough, without having to worry that when a crash does happen, the resport is in fact readable!
So my questions are:
1) Why was my archive missing a dSYM file?
2) If the dSYM file would've been generated, where could it be?
3) If I really do not have a dSYM file, can I still somehow get human-readable symbol names? I've got the original archive I uploaded and access to source code for that build.
3a) If I Product>Archive again (and assuming this time a dSYM does get generated), can I use this dSYM file instead? Or will it have a different UUID, causing it to be incompatible with the crash log cause...well...Apple?
Xcode version: 6.4
Here's what my crash log looks like in Xcode Organizer:
Thanks.
EDIT:
I upgraded to Xcode 7.3 before trying this but it may also work for version 6.
The solution to question (1) is to set the following in project build settings:
GCC_GENERATE_DEBUGGING_SYMBOLS
On XCode 8.3.2, select Yes on Generate Debug Symbols
For question 1), I also don't know. It may be a bug of Xcode. You can archive
the same code again, then generate a same dSYM file.
For question 2), you can search 'dSYM' in the folder '~/Library', because 'dSYM' file output in there generally. If not found, try searching it in the entire disk.
For question 3), you must have the system library symbol file that the crash log listed in 'Binary Images' section. You can find it in '~/Library/Developer/Xcode/iOS DeviceSupport'. If not found, you can connect an iPhone with the same OS version showed in the crash log to Xcode. After Xcode finished processing it, the system library symbol file of the iPhone can be copied to the folder. Then, you can re-symbolicate the crash log.
For question 3a), For the same app code, different Archive may be have a different dSYM file(UUID). If you use it to symbolicate the crash log, the symbolicated crash line is very close to the real crash line, so this can also help you to infer the real cause of crash.
From your screenshot, there only one line from your app code not be symbolicated. Now, you can generate the dSYM file of your app through using the app code which causes the crash to archive again. After you generate the dSYM file, use command line dwarfdump -u yourApp.app.dSYM to get UUID of it,Then check the uuid if contained in the first line of 'Binary images''. If NO, you can modify the UUID in the first line of 'Binary images'' to same as the new UUID which got from dwarfdump -u XXX.dSYM, Note the cpu architecture. Finish this, you can re-symbolicate the crash log in Xcode, or use command line symbolicatecrash crashreport.crash yourApp.app.dSYM. Note, you must guarantee the version of your app code same as version in the crash log, if not, the result is unbelievable.

How does crash reporting tools on iOS desymbolicate the crash reports on a release build?

On iOS, the Debug symbols are stripped from the release binaries for security reasons. So how does a crash reporting tool like Fabric,Hockey etc "desymbolicate" and show a nice stack trace of the crash point from a release build???
Do they capture/trace the crashes on their own, rather than relying on OS generated traces?
The following applies to OS X, iOS, tvOS and also watchOS:
If binaries themselves are containing symbols, they can only provide class name and method names. But never filenames or line numbers. So their benefit is limited but the increase in binary size is big, ~30-50% bigger (estimations, can be larger or smaller depending on the app)
If your build settings are setup properly (default settings = DEBUG_INFORMATION_FORMAT set to DWARF with dSYM File), then ever single time you do build you'll also get a dSYM package with a dwarf file that contains everything needed to symbolicate and also getting filenames and line numbers.
So how do iTunes Connect, Xcode, Fabric, HockeyApp and others actually do symbolication?
They are all using the dwarf files in the dSYM package. They take the memory address from a stack frame, find the corresponding binary image in the Binary Images section of the crash report by matching the address ranges, take the UUID of the binary image, find the dSYM package which contains the matching UUID for the matching CPU architecture and then run a tool like atos against it to get the (demangled) symbols.
And how do they get the stack traces?
The OS generates crash reports out of process which get used by iTunes Connect and Xcode.
All 3rd party systems need code added to the app which sets up crash handlers for signal based crashes and unhandled exceptions and at crash time try to safely fetch all information needed in a crash report. This code runs in the apps process and therefor may not do a lot of things (e.g. it may not allocate memory at crash time) to safely collect that data and not destroy any users data or cause a device deadlock. 3rd party systems can not get the system generated crash reports, because those are located outside of the apps sandboxes.

How to symbolicate crash log Xcode?

Xcode 5 organizer had a view which would list all the crash logs. and we could drag drop crash logs here. But since Xcode 6, I know they have moved devices out of organize and have a new window for the same. But I do not find a place where I view the crash logs which i drag-dropped in Xcode 5 after upping to Xcode 6. Anybody knows the answer ?
Writing this answer as much for the community as for myself.
If there ever are problems symbolicating a crash report, one can overcome them as follows:
Create a separate folder, copy Foo.app and Foo.app.dSYM from the corresponding .xcarchive into the folder. Also copy the .crash report into the folder.
Open the crash report in TextEdit or elsewhere, go to the Binary Images: section, and copy the first address there (e.g. 0xd7000).
cd into the folder. Now you can run the following command:
xcrun atos -o Foo.app/Foo -arch arm64 -l 0xd7000 0x0033f9bb
This will symbolicate the symbol at address 0x0033f9bb. Please make sure to pick the correct value for the -arch option (can be obtaned from the first line in the Binary Images: section, or figured out from the Hardware Model: in the crash report and the app's supported archs).
You can also copy the necessary addresses (e.g. a thread call stack) from the crash report directly into a text file (in TextEdit, hold Option and select the necessary text block, or copy and cut), to get something like this:
0x000f12fb
0x002726b7
0x0026d415
0x001f933b
0x001f86d3
Now you can save this into a text file, e.g. addr.txt, and run the following command:
xcrun atos -o Foo.app/Foo -arch arm64 -l 0xd7000 -f addr.txt
This will give a nice symbolication for all the addresses at once.
P.S.
Before doing the above, it's worth checking that everything is set up correctly (as atos will happily report something for basically any supplied address).
To do the checking, open the crash report, and go to the end of the call stack for Thread 0. The first line from the end to list your app (usually the second one), e.g.:
34 Foo 0x0033f9bb 0xd7000 + 2525627
should be the main() call. Symbolicating the address (0x0033f9bb in this case) as described above should confirm that this is indeed main() and not some random method or function.
If the address is not that of main(), check your load address (-l option) and arch (-arch option).
P.P.S.
If the above doesn't work due to bitcode, download the dSYM for your build from iTunes Connect, extract the executable binary from the dSYM (Finder > Show Package Contents), copy it into the directory, and use it (i.e. Foo) as the argument to atos, instead of the Foo.app/Foo.
You can refer this one too, I have written step by step procedure of Manual Crash Re-Symbolication.
Crash Re-Symbolication
STEP 1
Move all the above files (MyApp.app, MyApp-dSYM.dSYM and MyApp-Crash-log.crash) into a Folder with a convenient name wherever you can go using Terminal easily.
For me, Desktop is the most easily reachable place ;)
So, I have moved these three files into a folder MyApp at Desktop.
STEP 2
Now its turn of Finder, Go to the path from following whichever is applicable for your XCODE version.
Use this command to find the symbolicatecrash script file,
find /Applications/Xcode.app -name symbolicatecrash
Xcode 7.3 and newer (Xcode 8, ..., Xcode 14, ...): /Applications/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash
STEP 3
Add the found symbolicatecrash script file's directory to $PATH env variable like this: sudo vim /etc/paths.d/Xcode-symbolicatecrash and paste the script file's directory and save the file. When opening a new terminal, you can call symbolicatecrash at any folder as commands located in /usr/bin.
Or
Copy symbolicatecrash file from this location, and paste it to the Desktop/MyApp
(Wait… Don’t blindly follow me, I am pasting sybolicatecrash file in folder MyApp, one that you created in step one at your favorite location, having three files.)
STEP 4
Open Terminal, and CD to the MyApp Folder.
cd Desktop/MyApp — Press Enter
export DEVELOPER_DIR=$(xcode-select --print-path)
 — Press Enter
./symbolicatecrash -v MyApp-Crash-log.crash MyApp.dSYM
 — Press Enter
That’s it! Symbolicated logs are on your terminal…
Now simply, find out the Error and resolve it ;)
Ok I realised that you can do this:
In Xcode > Window > Devices, select a connected iPhone/iPad/etc top left.
View Device Logs
All Logs
You probably have a lot of logs there, and to make it easier to find your imported log later, you could just go ahead and delete all logs at this point... unless they mean money to you. Or unless you know the exact point of time the crash happened - it should be written in the file anyway... I'm lazy so I just delete all old logs (this actually took a while).
3a. Make sure the log file has the extension .crash (rather than .txt or .ips)
Just drag and drop your file into that list. It worked for me.
For me the .crash file was enough. Without .dSYM file and .app file.
I ran these two commands on the mac where I build the archive and it worked:
export DEVELOPER_DIR="/Applications/Xcode.app/Contents/Developer"
/Applications/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash /yourPath/crash1.crash > /yourPath/crash1_symbolicated.crash
There is an easier way using Xcode (without using command line tools and looking up addresses one at a time)
Take any .xcarchive file. If you have one from before you can use that. If you don't have one, create one by running the Product > Archive from Xcode.
Right click on the .xcarchive file and select 'Show Package Contents'
Copy the dsym file (of the version of the app that crashed) to the dSYMs folder
Copy the .app file (of the version of the app that crashed) to the Products > Applications folder
Edit the Info.plist and edit the CFBundleShortVersionString and CFBundleVersion under the ApplicationProperties dictionary. This will help you identify the archive later
Double click the .xcarchive to import it to Xcode. It should open Organizer.
Go back to the crash log (in Devices window in Xcode)
Drag your .crash file there (if not already present)
The entire crash log should now be symbolicated. If not, then right click and select 'Re-symbolicate crash log'
Xcode 11.2.1, December 2019
Apple gives you crash log in .txt format , which is unsymbolicated
**
With the device connected
**
Download ".txt" file , change extension to ".crash"
Open devices and simulators from window tab in Xcode
select device and select device logs
drag and drop .crash file to the device log window
We will be able to see symbolicated crash logs over there
Please see the link for more details on Symbolicating Crash logs
Follow these steps in Xcode 10 to symbolicate a crash log from an app build on the same machine:
Inside Organizer, locate the archive where the app is based on.
Click on the Download Debug Symbols button. Nothing will appear in your Downloads folder, but that's OK.
Connect the build machine to an iOS device.
Select the device in Devices and Simulators.
Click on the View Devices Logs button.
Drag-and-drop the crash file to the left panel. The file must end with a .crash extension, otherwise the drag fails.
Switch to the All Logs tab.
Select the added crash file.
The file should automatically symbolicate, otherwise use the right-click context menu item Re-Symbolicate Log.
You need access to the .dSYM package (folder) that contains a DWARF file, and you should open the .crash file with an editor.
Looking at the backtrace section, you should see something like this:
...
13 TheElements 0x0000000100f62ca0 0x100f5c000 + 27808
14 UIKitCore 0x00000001843e3044 -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 356 (UIApplication.m:2328)
...
Binary Images:
0x100f5c000 - 0x101673fff TheElements arm64 ...
...
Note the long address in the stacktrace section, the 3rd column (0x0000000100f62ca0)
Note the short address in the 4th column (0x100f5c000)
Note the architecture in the Binary Images section (arm64)
Execute the following:
$ atos -arch <arch> -o TheElements.app.dSYM/Contents/Resources/DWARF/TheElements -l <short_address> <long_address>
You should get a result like this:
-[AtomicElementViewController myTransitionDidStop:finished:context:]
Authoritative source: https://developer.apple.com/library/content/technotes/tn2151/_index.html#//apple_ref/doc/uid/DTS40008184-CH1-SYMBOLICATE_WITH_ATOS
Note: if for any reason you don't have access to the .dSYM file, you can recreate the .xcarchive using Xcode>Product>Archive, but make sure you are building the exact same code. Then you can extract the symbols from inside the .xcarchive package.
Make sure that your Xcode application name doesn't contain any spaces. This was the reason it didn't work for me. So /Applications/Xcode.app works, while /Applications/Xcode 6.1.1.app doesn't work.
From Apple's docs:
Symbolicating Crash Reports With Xcode
Xcode will automatically attempt to symbolicate all crash reports that it encounters. All you need to do for symbolication is to add the crash report to the Xcode Organizer.
Connect an iOS device to your Mac
Choose "Devices" from the "Window" menu
Under the "DEVICES" section in the left column, choose a device
Click the "View Device Logs" button under the "Device Information" section on the right hand panel
Drag your crash report onto the left column of the presented panel
Xcode will automatically symbolicate the crash report and display the results
To symbolicate a crash report, Xcode needs to be able to locate the following:
The crashing application's binary and dSYM file.
The binaries and dSYM files for all custom frameworks that the application links against. For frameworks that were built from source with the application, their dSYM files are copied into the archive alongside the application's dSYM file. For frameworks that were built by a third-party, you will need to ask the author for the dSYM file.
Symbols for the OS that the that application was running on when it crashed. These symbols contain debug information for the frameworks included in a specific OS release (e.g, iOS 9.3.3). OS symbols are architecture specific - a release of iOS for 64-bit devices won't include armv7 symbols. Xcode will automatically copy OS symbols from each device that you connect to your Mac.
If any of these are missing Xcode may not be able to symbolicate the crash report, or may only partially symbolicate the crash report.
The easiest process to symbolicate crash logs:
preserve the xcarchive file from the organizer during IPA building process for future use.
When the crash occurs, collect the crash logs from affected device. The extension should be .crash. If the crash log is in .ips format, just rename it to .crash.
Double click the xcarchive from the stored path to make it appear in organizer(if not present already).
open in xcode window->devices and simulators -> view device logs -> all logs -> drag and drop the .crash file.
Wait for 5secs. Bang! the application calls in stack trace will be symbolicated!
You may still see a lot of symbols though! those are internal library and framework calls.
This is the easiest one, tried and tested!
I was struggling to have the crash report symbolicated through atos but I was reluctant as the process seems cumbersome, But I found the crash report in the Xcode-> Window -> Organizer->Crashes(in left-side menu) Xcode will automatically download the crash logs and will symbolicate automatically, From there you can easily find the reason of the crash.

Unable to symbolicate crash log

I am facing some problem in symbolicating my crash log.
I followed all the steps as answered by naveenshan in this thread.
But once I execute the line
atos -arch armv7 -o 'app name.app'/'app name' 0x0003b508,
it just throws me back the address 0x0003b508 back on the console.
I am not able to figure out why its not symbolicating my crash log.
I even tried symbolicating by drag and drop my crash log into device logs in my xcode. But even that doesnt seem to work.
You have to have the exact dSYM that was generated along the build that is installed on the device and generates the crash report.
Please follow the steps explained in this answer (even though the original problems are not the same, the steps help identify if you have the correct dSYM): Symbolicate Crash Log when App Name Contains Apostrophe and Space (Deployed App)
Executing the line with atos you posted, is most likely to return nothing useful, since the address given as a parameter does not have to exist in your app. You have to use a proper address taken from the crash report and calculated properly as detailed here: iOS crash reports: atos not working as expected
Best approach is the follow the steps written in the first linked post.

Only crashing on Testflight

We're using TestFlight to send out pilots of our app.
There's a part of the app that is crashing, and we had a lot of trouble reproducing the crash. The code their is fairly simple.
It turns out that the users who got the App via TestFlight get the crash, while if you build the app and install it using the IDE it doesn't crash!
Anyone have ideas about what might be causing this?
Any ideas for workarounds? We don't want to stop using TestFlight.
Make sure to build your app in Release Mode not in Debug. The app may only crash when in Release.
The first thing I'd try is to map the crash stack trace to function names in your application. This may yield useful insight into the nature of the crash:
As soon as a crash is reported request the crash log. This can be obtained through Xcode's organizer or if that's not an option it can be screen-captured from the iPhone's Settings -> General -> About -> Diagnostics & Usage -> Diagnostic & Usage Data. Scroll to the app name or the section LatestCrash-AppName.plist.
Although you can in theory symbolicate a crash, I find the procedure described below a foolproof way to get symbols from the stack. Convert all stack addresses for the crashing thread into method names.
Optionally request the iDevice syslog. This may include assertion failure messages which are also invaluable. Note that this should be done as quickly as possible as the syslog only holds so many entries before they get dropped. You can use the Organizer or the cmd line idevicesyslog to obtain this.
Manual symbolication:
This will work as long as your builds have debug information.
Obtain the _exact_same_ .ipa that crashed. If you didn't save it you can download it from the device by using iFunBox or the cmd line ideviceinstaller utility.
Unzip the .ipa
Run the following command on the executable file (Payload/AppName.app/AppName):
otool -tv AppName.app | c++filt > listing.asm
Wait while the previous step completes (may take a while). The generated listing.asm file will be several megabytes long.
Using an editor that can handle large files search listing.asm for the addresses listed in the stack trace. Note that the addresses may be a few bytes off (usually pointing 3 or so bytes ahead). Also, addresses that aren't found in listing.asm indicate addresses in iOS libraries. Ignore those for now.
Of course, if you're able to symbolicate, you can skip this procedure.
Good luck debugging!
We had a similar problem. The issue with us was static libraries. When we built the app from scratch and went to testflight, it was crashing but from the IDE it wasn't. The crash was because the static libraries did not get included when doing a build, but was getting included if I connected the iPad direct and used XCode to install.
A simple test will prove this:-
1.) Instead of building from IDE, create a .app file and then load it via iTunes and check if you are getting the crash.
We worked around this by creating the .iPA manually, that is creating the .app then making a Payload folder and putting the .app in it along with the info.plist.
Then things began to work in Testflight as well.

Resources