Preface: I am not an iOS developer and relevant documentation is scarce, so I may be missing something that should be obvious.
My team is building a Firebase app for iOS in Swift and we're getting crash reports in the console with obfuscated stack traces. The provided instructions to upload symbol files don't seem to be correct - the command as written returns an error about missing service account, and after we figured that one out it gave more warnings that we didn't decypher - but we managed to configure automatic upload and then rebuilt the latest release.
Now I see that a handful of UUIDs (presumably the phones/sims that we've run that rebuilt version on?) have uploads associated with them, but none of them match the reports we've received. What I don't understand is how we're supposed to generate symbol files for the rest of the UUIDs. Am I correct in understanding that a UUID refers to a specific device, and any given symbol file will match only a single build and UUID? We don't have access to the devices that are reporting these crashes; that's why we're relying on the console instead of debugging them in Xcode. If you only get the correct symbols for the devices that you physically connect to Xcode when you make the build, what's the point of getting the stack traces online?
Related
I am using Bitrise as a CI/CD on my iOS app.
In the workflow there is a step to release the app via Fabric.
When a crash happens the only think I can see in the Dashboard is many __hidden steps on the stack.
I guess that this is because of some missing symbols but Fabric doesn't says that any symbol is missing. I tried to add them manually any way but I am still seeing the __hidden crashes.
Any ideas of how can I fix this issue?
Here is an example of what I get.
Check out Apple's document describing hidden symbols. Look for the "Translating 'hidden' symbol names back to their original names" section.
The issue here isn't that Crashlytics is missing symbol information. It's that the symbol information they have literally tells them your symbols are named __hidden#xyz. This has resulted in obfuscation you are seeing here. As the Apple doc describes, it is possible to de-obfuscate these symbols, but I'm not sure if Crashlytics supports this. They have a little bit of info here.
My understanding is this happens when you opt to not share symbol information with Apple during the app submission process. If I'm right, once you do this, it is impossible to change until you release a new version of your app.
Update:
I've discovered an option in dsymutil that makes it possible to deobfuscate a dSYM in this situation.
--symbol-map=*bcsymbolmap*
This will modify the dSYM in place with deobfuscated symbols. Uploading that dSYM to Crashlytics should result in expected symbolication. However, due to caching, you will probably need to reach out to them first to make sure this change takes effect.
We are trying to address a critical crash in an iOS application that's using Fabric / Crashlytics. We don't have the contact details of the person that was last working on the application and uploaded the latest version to the App Store.
In the project dashboard I noticed the message "Found XXX unsymbolicated crashes from missing dSYMs in 1 version in the last 24 hours". Screenshot: https://i.imgur.com/YT9gggJ.jpg
I did the only sensible thing I could think of: I went to the App Store Connect dashboard. I downloaded the dSYM zip for the build in question as per the official instructions from Fabric: https://docs.fabric.io/apple/_images/download-dsym.png
I then went to the dSYM tool and uploaded the zip directly. Turns out only two of the four files required were in the zip (I also examined it myself): https://i.imgur.com/JqxZcaD.jpg
So... I'm in a black box situation here...
I'm not an iOS developer
I don't have access to the computer used to build the project or the person that produced the build
I'm assisting an iOS developer that has just joined the team
I have access to the project repository
I have access to the App Store Connect profile
I have admin access in Fabric
My questions are:
Why were the other two dSYM files not in the zip I downloaded form the App Store?
Why are some crashes associated with one dSYM and the rest to another? Is there any sort of categorization of the crashes that we currently have access to?
Can we do something to gain access to all the production crash reports without publishing a new app version to the App Store? We're trying to avoid this scenario ATM.
EDIT #1:
So this person that published the build was not exactly following best practices. I wanted to explore the possibility of them having the dSYM files committed to the server repo.
This is their gitignore:
.DS_Store
xcuserdata
<PROJECT NAME>.xcodeproj/project.xcworkspace/xcshareddata/
build/
Build/
I guess the build folder pretty much rules out this possibility. I also searched the file structure for files containing the text "com_apple_xcode_dsym_uuids ==" with no success...
EDIT #2:
I'm appending an exceprt of the answer Fabric support gave me:
If your application is using frameworks, the product folder will have
a separate dSYM file generated for each framework built. Eventually
all of them are needed if we want to cover our bases and be able to
symbolicate a crash in every possible location in our app. A dSYM file
generated while building a specific version of the application can
only be used to symbolicate crashes from that specific version only.
dSYM files are identified by a Unique ID (UUID), which changes every
time we modify and rebuild our code, and that ID is what is used to
match a symbol file to a specific crash. A dSYM may be associated with
more than one UUID, as it may contain debug information for more than
one architecture.
It looks to me that you may need to upload a new build, which is not too bad after all if it includes any bug fixes. Regarding your questions:
Why were the other two dSYM files not in the zip I downloaded form the App Store?
The option to download dSYMs from App Store connect is available only if the app was distributed using Bitcode. Bitcode is intermediary representation of the source which App Store uses to produce the final optimised machine code targeting specific architecture/device. When Bitcode is selected all linked frameworks/libs should also be delivered using bitcode, so it looks weird to have only some dSYMs. Although unlikely, is it possible that the missing dSYMs are from another build?
Why are some crashes associated with one dSYM and the rest to another? Is there any sort of categorization of the crashes that we currently have access to?
Each target/framework/lib generates its own dSYMs, so your app probably depends on one or more frameworks and some of the crashes originate from that frameworks.
Can we do something to gain access to all the production crash reports without publishing a new app version to the App Store?
I can't think of another solution so far.
Question #1:
I am using Crashlytics for one of my commercial iOS Apps. Their service seems very helpful. I wonder if there is any Legal issue of using their services for commercial apps? Is there any limitations for using Crashlytics?
Question #2:
I need to upload .dsym file in their server to get crashlogs symbolicated. Is there any privacy issue. Is it possible to get my code (or any info) back by reverse engineering when they have .dsym file.
Question #3
Sometime some crashes are not showing in the crashlytics dashboard. Is it necessary to be online when crash happens to get crashlog in dashboard? Don't Crashlytics keep logs trace when offline and send logs back to the server when device become online? Any idea how they work?
Mike from Fabric here, but I'm not a lawyer.
1) Fabric and Crashlytics is used in many commercial apps, including our own. There aren't limitations to using Fabric. You can find the Fabric terms of service here.
2) The dSYM is what let's us or any other crash reporter symbolicate the crash report as it contains the symbols that map back to your app's source. None of your source code is uploaded. From Apple's own documentation:
As the compiler translates your source code into machine code, it also
generates debug symbols which map each machine instruction in the
compiled binary back to the line of source code from which it
originated. Depending on the Debug Information Format
(DEBUG_INFORMATION_FORMAT) build setting, these debug symbols are
stored inside the binary or in a companion Debug Symbol (dSYM) file.
The Debug Symbol file and application binary are tied together on a
per-build-basis by the build UUID. A new UUID is generated for each
build of your application and uniquely identifies that build. Even if
a functionally-identical executable is rebuilt from the same source
code, with the same compiler settings, it will have a different build
UUID.
3) Crashes are caught regardless if the app is connected to a network or not. However, crashes are only sent on relaunch of the app and would then be processed.
One thing to note is that if you're testing in the Simulator or with your device connected to Xcode will cause Xcode's debugger to capture the crash instead of us.
Further, if the dSYM hasn't been uploaded, then we're unable to process the crash report and we'll alert you in the Crashlytics dashboard of the missing dSYMs so that you can upload them.
We're using TestFlight to send out pilots of our app.
There's a part of the app that is crashing, and we had a lot of trouble reproducing the crash. The code their is fairly simple.
It turns out that the users who got the App via TestFlight get the crash, while if you build the app and install it using the IDE it doesn't crash!
Anyone have ideas about what might be causing this?
Any ideas for workarounds? We don't want to stop using TestFlight.
Make sure to build your app in Release Mode not in Debug. The app may only crash when in Release.
The first thing I'd try is to map the crash stack trace to function names in your application. This may yield useful insight into the nature of the crash:
As soon as a crash is reported request the crash log. This can be obtained through Xcode's organizer or if that's not an option it can be screen-captured from the iPhone's Settings -> General -> About -> Diagnostics & Usage -> Diagnostic & Usage Data. Scroll to the app name or the section LatestCrash-AppName.plist.
Although you can in theory symbolicate a crash, I find the procedure described below a foolproof way to get symbols from the stack. Convert all stack addresses for the crashing thread into method names.
Optionally request the iDevice syslog. This may include assertion failure messages which are also invaluable. Note that this should be done as quickly as possible as the syslog only holds so many entries before they get dropped. You can use the Organizer or the cmd line idevicesyslog to obtain this.
Manual symbolication:
This will work as long as your builds have debug information.
Obtain the _exact_same_ .ipa that crashed. If you didn't save it you can download it from the device by using iFunBox or the cmd line ideviceinstaller utility.
Unzip the .ipa
Run the following command on the executable file (Payload/AppName.app/AppName):
otool -tv AppName.app | c++filt > listing.asm
Wait while the previous step completes (may take a while). The generated listing.asm file will be several megabytes long.
Using an editor that can handle large files search listing.asm for the addresses listed in the stack trace. Note that the addresses may be a few bytes off (usually pointing 3 or so bytes ahead). Also, addresses that aren't found in listing.asm indicate addresses in iOS libraries. Ignore those for now.
Of course, if you're able to symbolicate, you can skip this procedure.
Good luck debugging!
We had a similar problem. The issue with us was static libraries. When we built the app from scratch and went to testflight, it was crashing but from the IDE it wasn't. The crash was because the static libraries did not get included when doing a build, but was getting included if I connected the iPad direct and used XCode to install.
A simple test will prove this:-
1.) Instead of building from IDE, create a .app file and then load it via iTunes and check if you are getting the crash.
We worked around this by creating the .iPA manually, that is creating the .app then making a Payload folder and putting the .app in it along with the info.plist.
Then things began to work in Testflight as well.
There is a build setting that says "Symbols Hidden By Default" that I have set to NO on my Debug build, but I'm still not getting symbolized crashes on my devices. Is this just not possible on ios ?. I know that on mac it is, but maybe there is another setting I'm missing ?
Thank you.
Symbolication will only work, if you did not start another build process after deploying a build onto your device. Each build will generate a unique combination of app binary and dSYM file with the same UUIDs. This ID is set in the crash report and used to find the correct dSYM to resolve the symbols in the crash report.
If you build again, these files get replaced and because of that get a new UUID also. So the symbolication process can not find the corresponding files and will not resolve the app specific symbols. The iOS symbols should work fine though.