Dealing with missing dSYM files in a black box situation - ios

We are trying to address a critical crash in an iOS application that's using Fabric / Crashlytics. We don't have the contact details of the person that was last working on the application and uploaded the latest version to the App Store.
In the project dashboard I noticed the message "Found XXX unsymbolicated crashes from missing dSYMs in 1 version in the last 24 hours". Screenshot: https://i.imgur.com/YT9gggJ.jpg
I did the only sensible thing I could think of: I went to the App Store Connect dashboard. I downloaded the dSYM zip for the build in question as per the official instructions from Fabric: https://docs.fabric.io/apple/_images/download-dsym.png
I then went to the dSYM tool and uploaded the zip directly. Turns out only two of the four files required were in the zip (I also examined it myself): https://i.imgur.com/JqxZcaD.jpg
So... I'm in a black box situation here...
I'm not an iOS developer
I don't have access to the computer used to build the project or the person that produced the build
I'm assisting an iOS developer that has just joined the team
I have access to the project repository
I have access to the App Store Connect profile
I have admin access in Fabric
My questions are:
Why were the other two dSYM files not in the zip I downloaded form the App Store?
Why are some crashes associated with one dSYM and the rest to another? Is there any sort of categorization of the crashes that we currently have access to?
Can we do something to gain access to all the production crash reports without publishing a new app version to the App Store? We're trying to avoid this scenario ATM.
EDIT #1:
So this person that published the build was not exactly following best practices. I wanted to explore the possibility of them having the dSYM files committed to the server repo.
This is their gitignore:
.DS_Store
xcuserdata
<PROJECT NAME>.xcodeproj/project.xcworkspace/xcshareddata/
build/
Build/
I guess the build folder pretty much rules out this possibility. I also searched the file structure for files containing the text "com_apple_xcode_dsym_uuids ==" with no success...
EDIT #2:
I'm appending an exceprt of the answer Fabric support gave me:
If your application is using frameworks, the product folder will have
a separate dSYM file generated for each framework built. Eventually
all of them are needed if we want to cover our bases and be able to
symbolicate a crash in every possible location in our app. A dSYM file
generated while building a specific version of the application can
only be used to symbolicate crashes from that specific version only.
dSYM files are identified by a Unique ID (UUID), which changes every
time we modify and rebuild our code, and that ID is what is used to
match a symbol file to a specific crash. A dSYM may be associated with
more than one UUID, as it may contain debug information for more than
one architecture.

It looks to me that you may need to upload a new build, which is not too bad after all if it includes any bug fixes. Regarding your questions:
Why were the other two dSYM files not in the zip I downloaded form the App Store?
The option to download dSYMs from App Store connect is available only if the app was distributed using Bitcode. Bitcode is intermediary representation of the source which App Store uses to produce the final optimised machine code targeting specific architecture/device. When Bitcode is selected all linked frameworks/libs should also be delivered using bitcode, so it looks weird to have only some dSYMs. Although unlikely, is it possible that the missing dSYMs are from another build?
Why are some crashes associated with one dSYM and the rest to another? Is there any sort of categorization of the crashes that we currently have access to?
Each target/framework/lib generates its own dSYMs, so your app probably depends on one or more frameworks and some of the crashes originate from that frameworks.
Can we do something to gain access to all the production crash reports without publishing a new app version to the App Store?
I can't think of another solution so far.

Related

dSYM files not available on iTunes/App Store Connect (bitcode enabled)

Our application has bitcode enabled, so we can't use dSYM files generated on our machines/CI but rather we should use dSYM generated during bitcode compilation. The problem here is that we don't have those files available to download on App Store Connect (no link is presented).
That's really weird. We suspected that maybe our 3rd party tool which we upgraded recently and that is used to obfuscate our binaries started to work incorrectly but all builds get "Validated" status and we can publish them in App Store without any issue.
I suppose that bitcode process runs correctly as we get smaller binaries available in App Store, but dSYM is somehow not produced.
I found a thread on Apple Developer Forums but nothing really uncovers the root of this issue: DSYMs No Longer Offered for Download?

Crashlytics - Do we need to pay for commercial app analysis

Question #1:
I am using Crashlytics for one of my commercial iOS Apps. Their service seems very helpful. I wonder if there is any Legal issue of using their services for commercial apps? Is there any limitations for using Crashlytics?
Question #2:
I need to upload .dsym file in their server to get crashlogs symbolicated. Is there any privacy issue. Is it possible to get my code (or any info) back by reverse engineering when they have .dsym file.
Question #3
Sometime some crashes are not showing in the crashlytics dashboard. Is it necessary to be online when crash happens to get crashlog in dashboard? Don't Crashlytics keep logs trace when offline and send logs back to the server when device become online? Any idea how they work?
Mike from Fabric here, but I'm not a lawyer.
1) Fabric and Crashlytics is used in many commercial apps, including our own. There aren't limitations to using Fabric. You can find the Fabric terms of service here.
2) The dSYM is what let's us or any other crash reporter symbolicate the crash report as it contains the symbols that map back to your app's source. None of your source code is uploaded. From Apple's own documentation:
As the compiler translates your source code into machine code, it also
generates debug symbols which map each machine instruction in the
compiled binary back to the line of source code from which it
originated. Depending on the Debug Information Format
(DEBUG_INFORMATION_FORMAT) build setting, these debug symbols are
stored inside the binary or in a companion Debug Symbol (dSYM) file.
The Debug Symbol file and application binary are tied together on a
per-build-basis by the build UUID. A new UUID is generated for each
build of your application and uniquely identifies that build. Even if
a functionally-identical executable is rebuilt from the same source
code, with the same compiler settings, it will have a different build
UUID.
3) Crashes are caught regardless if the app is connected to a network or not. However, crashes are only sent on relaunch of the app and would then be processed.
One thing to note is that if you're testing in the Simulator or with your device connected to Xcode will cause Xcode's debugger to capture the crash instead of us.
Further, if the dSYM hasn't been uploaded, then we're unable to process the crash report and we'll alert you in the Crashlytics dashboard of the missing dSYMs so that you can upload them.

FirebaseCrash symbol files and UUIDs

Preface: I am not an iOS developer and relevant documentation is scarce, so I may be missing something that should be obvious.
My team is building a Firebase app for iOS in Swift and we're getting crash reports in the console with obfuscated stack traces. The provided instructions to upload symbol files don't seem to be correct - the command as written returns an error about missing service account, and after we figured that one out it gave more warnings that we didn't decypher - but we managed to configure automatic upload and then rebuilt the latest release.
Now I see that a handful of UUIDs (presumably the phones/sims that we've run that rebuilt version on?) have uploads associated with them, but none of them match the reports we've received. What I don't understand is how we're supposed to generate symbol files for the rest of the UUIDs. Am I correct in understanding that a UUID refers to a specific device, and any given symbol file will match only a single build and UUID? We don't have access to the devices that are reporting these crashes; that's why we're relying on the console instead of debugging them in Xcode. If you only get the correct symbols for the devices that you physically connect to Xcode when you make the build, what's the point of getting the stack traces online?

Download dSYM fails "Missing App Version"

When I try to download the dSYM from the organizer, it gives me this error :
Missing App Version
An app record for “co.**” was found but a matching version for “1.0.3” build “10” was not.
I have enabled Bitcode. It's a swift app with Swift and Objective-C pods. Xcode Version 7.2 (7C68)
When I right click on the archive and look at it's package contents in Finder, I see a dSYMs folder. I zipped it and uplaoded it to Crashlytics, but it still shows dSYMs missing for all my builds. It even has App.dsym with some dsyms with UUIDs and others for my pods.
Is this an XCode bug?
I see somene has posted a similar question for tvOS on the apple developer forum but there are no answers there. I am not allowed to ask a question there for some reason :p
I haven't added a build to iTunesConnect yet, is that necessary to download dSYMs? Where does XCode download the dSYM from? Could it have something to do with the fact that my bundleId is of the form co.someString and not co.someString.someOtherString? I guess not, I'm just looking for anything.
Is disabling bitcode the only way to be able to get the dSYM for me?
Sounds like your dSYM file doesn't match the application binary, even if app version number and build number would be correct. Every time you build (archive) app, you get new and different dSYM file, too.
You cannot use just any dSYM file from any archived project, even if you made no code changes between archives. You must get the exactly correct archive and dSYM from within it.
Here's related info from HockeyApp, similar service to Crashlytics. Even if they are diff services, symbolicating crash reports is similar problem.

Invalid Signature - Code object is not signed at all

Im able to submit my app through Xcode 6.3.2 perfectly fine. Validation and analyzing pass perfectly. Once it successfully submits to the app store though I get an email from Apple:
"Dear developer,
We have discovered one or more issues with your recent delivery for "App". To process your delivery, the following issues must be corrected:
Invalid Signature - Code object is not signed at all. Make sure you have signed your application with a distribution certificate, not an ad hoc certificate or a development certificate. Verify that the code signing settings in Xcode are correct at the target level (which override any values at the project level). Additionally, make sure the bundle you are uploading was built using a Release target in Xcode, not a Simulator target. If you are certain your code signing settings are correct, choose "Clean All" in Xcode, delete the "build" directory in the Finder, and rebuild your release target. For more information, please consult https://developer.apple.com/library/ios/documentation/Security/Conceptual/CodeSigningGuide/Introduction/Introduction.html
Once these issues have been corrected, you can then redeliver the corrected binary."
I have tried redownloading the distribution cert, regenerating the distribution provisioning profile, added "--deep" to the code signing "Other Code Signing Flags." I even checked the bundle name etc, everthing is alpha numeric. I was able to submit fine on May 22nd, now on June 3rd everything breaks.
Doesnt make any sense, any help would be appreciated!
UPDATE & SOLUTION:
While I don't have a good explanation of why this suddenly has happened within the last week, I finally found a solution this morning.
I started with a new project and submitted to the app store with nothing but the identifier and correct version and build numbers, which processed fine. After that I started piecing in any assets that wasnt my own code until I got the "Invalid Binary" email. I narrowed it down to the Hockey App SDK (embedded framework) which was causing the issue and not even being used anymore so I removed it from the project (problem solved). The disturbing part is that nothing fails on my end during validation or submission and according to github this directory and content hasn't changed in a year, which leads me to believe something changed server side at Apple.
I did see a lot of posts via google saying that frameworks needed signed etc and when using Xcode 6 and iOS 8 it seems to be the standard which is why I assumed it might be something along these lines.
Im not sure how helpful this is as I was building for iOS and this article is in reference to Mac, but HockeyApp explains in order to distribute to the app store you need to sign the framework with your own identity here:
http://support.hockeyapp.net/kb/client-integration-ios-mac-os-x/hockeyapp-for-mac-os-x
If anyone has anymore technical notes on this or why this suddenly changed Id love to understand this better.
I've checked a variety of places and there seem to be several things that are now being rejected by iTunes Connect. The solution is typically to remove the offending resource from the Target -> Build Phases -> Copy Bundle Resources (as #azizus mentions). Unfortunately Apple doesn't tell you what file causes this issue with your builds so you have to go hunt for yourself. Here are some items that I've found that will do it:
Shell scripts (Look for .sh files, though they could have a different
extension)
Also, look out for files that are listed as executable, when they
shouldn't be. Those might be a good place to look for shell scripts
that you might have missed.
Frameworks (Framework bundles, even .a or .o files - you
don't need them as they will get compiled into the executable binary)
DocSets (I don't know why, but I found that the HockeyApp SDK
includes a DocSet bundle which was the cause in my experience)
Sometimes this might also happen due to some weird entitlements
issue. The entitlements you have may not match up with the App in the
provisioning portal.
Look out for invalid characters in your app name or file names (like
wildcard characters)
This is a pretty broad list, something I did to help in the search is build an archive and then show the contents of the .app in the archive using finder, sorting by file type. The strange thing is that these files actually exist in the _CodeSignature/CodeResources file.
My own theory on why this is happening is that Apple made some changes (or is making some changes) because of Extensions and WatchKit apps. Essentially, you are including a couple of binaries in the packaged IPA (phone app, extension, watch app). They probably want to make sure you're not including something else that could potentially be executed. Unfortunately, the error message is too vague (really it's incorrect) for most.
This took me 3 days to debug.
In the end it was due to an external framework I created (lets call it X) that I was importing via carthage. X had its own dependencies that it was importing via carthage as well. In order to link these frameworks it had a path in the build settings called Framework Search Paths set to the location of the frameworks. For some reason it was this flag in this framework that was causing the problem specified in the questions. I eventually imported X's dependencies with Git submodules so that I didn't have to set the Framework Search Paths flag. I the exported the framework and manually added it to my project I was submitting to the AppStore. Then it worked.
I can reproduce this when I 'create folder references' for my resources folder as opposed to 'create groups' when adding in.
I contacted HockeyApp and they suggested not to add the SDK to app bundle. So I navigated to Target -> Build Phases -> Copy Bundle Resources and removed HockeySDKResources.bundle from there. iTunes Connect accepted my binary.
In my case it was a info.plist duplicated that was not used. (it wasn't easy find out the problem). I removed almost all the files of my project until remove this one and.. it worked
Clearing the value for Code Sign Resource Rules Path in each target resolved the issue.

Resources