What is the easiest way to produce a Mach-O object file that does not have the SUBSECTIONS_VIA_SYMBOLS flag set, such that the linker (with -dead_strip) will not later try to cut the text section into pieces and guess which pieces are used?
I can use either a command-line option to llvm/gcc (4.2.1) that will prevent it from emitting .subsections_via_symbols in the first place, or a command-line tool that will remove the flag from an existing object file.
(Writing such a tool myself based on the Mach-O spec is an option, but if possible I'd rather not reinvent the wheel that hard).
Platform: iOS, cross-compiling from OSX with XCode 4.5.
Background: We're supplying a static library that other companies build into apps. When our library encounters a problem it produces a crash report with a stack trace and certain other key information that (if we're lucky) we get to analyze later. Typically the apps as deployed have been stripped of debug information so interpreting stack traces is a problem. If we were making the app ourselves we would just save the DWARF debug data from before stripping and use that to decode the addresses in the incoming crash reports. But we can't depend on the app makers supplying us with such data from their linking steps.
What we're doing instead is to let the crash report include the run-time address of selected function; from that we can deduce the offset between addresses in our linker map and addresses in the crash report. We're linking our entire library incrementally into a single .o before we stuff it into an .a; since it does only one big thing there wouldn't be much to save from removing unused functionality from it when the app is eventually linked. Unfortunately there's a few small pieces of code in the library that are sometimes not used (alternative API entry points for the main functionality, small helper functions for interpreting our error codes and the like), and if the app developer links with -dead_strip, it disturbs the address reconstruction of crash reports that the relative offsets in the final app differ from the linker map from our incremental link operation.
We can't realistically ask all app developers to disable dead-code stripping in their build process, so it seems a better way forward if we could mark our .o as "not dead-strippable" and have the eventual app linking respect that.
I solved it.
The output of an incremental link operation only has MH_SUBSECTIONS_VIA_SYMBOLS set if all the input objects have it set. And an object file produced from assembler input only has it set if there's an explicit directive set. So one can remove the flag by linking with an empty assembler input:
echo > empty.s
$(CC) $(CFLAGS) input.o empty.s -nostdlib -Wl,r -o output.o
Related
Some iOS 9 devices in the wild seem to crash with the error message that I receive from the very basic crash reporting in Xcode only
dyld: malformed mach-o: load commands size (16464) > 16384
Unfortunately that's all the info I get. I can't even debug or reproduce locally.
Can anyone hint me into the right direction here?
It occurs after updating my Cocoapods, so I guess there's one of them (or their dependency) that misbehaves.
After some investigation of my mach-O binary, I saw that the sizeofcmds is really 16464.
If I understand correctly, there seems to be a load command size limit of 16384, can anyone confirm this?
Does that mean I should remove dylibs and everything should be fine?
At WWDC18 I went to an Apple engineer who is working on dyld. Here’s what he had to say:
The Dyld code is downloadable from https://opensource.apple.com (the one specific to us can be found inside macOS 10.12)
For iOS 9 the maximum size of load commands is indeed 16k aka 1 memory page (There’s no way around it! This is imposed by the OS itself. For customer service telling people to update to iOS 10 (all devices that run iOS 9 can except for iPhone 4S) would be viable.)
Since iOS 10 the maximum size of commands is 32k
Majority of the size of the load commands is determined by strings (paths) of the frameworks (use command otool -L to see them
Possible solutions:
Use less libraries (that was our goto solution thus far, but we will change to umbrella libraries (see below))
Shortening names (might screw up header lookup of cocoa pods, maybe use head maps to fix that inside the Xcode build process → maybe more (high-level) info in WWDC18 session “Behind the scenes of the Xcode Build Process”)
Try to build static archives for libraries (should not have dynamic resources otherwise make copy phases and figure out where resources are)
Build frameworks that re-export other frameworks (umbrella frameworks). Use -reexport-l as a linker flag (not done often) → gonna make some runtime overhead when starting the app, also uses a bit more memory (man ld → for info on re-exports)
The engineer recommended to file a bugreport via bugreport.apple.com, because in the future even hitting the 32k limit is possible.
I found a solution that will (at least temporarily) work for me - but I still encourage everyone to provide a real solution and more detailed insights.
Anyway:
Extract your binary
Xcode Archive
Export as IPA
Rename YourApp.ipa to YourApp.zip and extract
Navigate to the subfolder payload to find your YourApp.app
Right click & Show Package Contents of your YourApp.app file
Copy your binary YourApp (no file extension) to a different location
Investigate your mach-o binary
Run otool -f on your binary
Note the align for both architectures are listed which, for me, says 2^14 (16384). This seems to be the threshold for the size of load commands.
Run otool -l on your binary
You'll see that the different architectures and their load commands are listed - as well as their sizeofcmds (size of commands).
Now the funny thing:
For arm64, the sizeofcmds (16464) was larger than the align (16384), while it wasn't for armv7.
Now I haven't found enough documentation on this, but I assume that align symbolizes a threshold that should not be reached by the load command size. And also that it adjusts automatically (since we are definitely not having that many frameworks in our app, there have to be apps that have more).
So I guess the error came from this unlikely case, that the sizeofcmds was different in between the architectures AND that one of them was actually valid (so that the align was not automatically adjusted).
Please correct me if I'm wrong, I am just assuming here and I really really want to understand why this happens.
Solve the issue
Remove frameworks until you are under the sizeofcmds for both architectures.
I know this is not scalable, we were lucky (and stupid) that we still had one barely used framework in there that we could easily remove.
Fortunately this only seems to be an issue on iOS9 and will therefore loose relevance over the next months, nevertheless, we should find out if I'm right
Investigation ideas
My assumption that the align is automatically adjusted could be investigated by just putting in more and more frameworks to see if it actually does.
If so, adding frameworks would also solve the original issue - not nice, but at least slightly more scalable.
Sidenote
I don't feel like I shed enough light on the origins of this issue and I had a lot of assumptions.
While my solution works, I really hope you feel encouraged to investigate this as well and give a better answer.
So here's the problem:
The Mach-O header size is expected to be 16k (optimized for the platform's pagesize). In the reference by rachit it's basically the same thing but the limit is 32K. Both are correct in that this is hard limit of dyld, the loader.
The total size of load commands exceeds this max size. Removing frameworks and libraries works for you because that removes LC_LOAD_DYLIB commands (And, there is no reason why you'd need so many frameworks anyway). Instead of removing frameworks, build your app from the ground up starting with the core frameworks, and adding so long as you get linker errors.
btw, 'Align' has nothing to do with this - Alignment refers to the fat (universal) architecture slices, and doesn't have anything to do with the Mach-O.
I was able to resolve this for my team after reviewing the result of otool -l. Turns out we had the same directory included in our framework search paths 5x causing our dylibs to be added as rpaths 5x.
I'd like to restrict how much access to resources jailbreak tweaks receive. Things like network/keychain/location access... Is it possible to manually add a entitlements plist per tweak?
Many thanks.
Tweak is a dylib - it will be loaded in a process. That process may have entitlements and those entitlements will be used for the tweak. That's it. Tweak doesn't have it's own entitlements.
As for your question. Because of what I said before you can't restrict just a tweak - your restrictions will be applied to the whole process that is being tweaked. You can't do anything about that. That's how tweaks work - they are dylibs dynamically loaded into process address space. After that the tweak becomes a part of the process. So any restrictions will be applied to the whole process which includes the tweak, application code and any other dylib/framework application is linked to.
So if you want to develop an application which will help a user put restrictions on tweaks, I don't think you can do such a thing. What you can do is to analyze which application are being tweaked, what entitlements does they have, what frameworks and dylibs are used by a tweak (mainly the private ones). And from that user can either enable or disable that tweak. You can even analyze import section and string literals of the tweak to determine exactly which APIs does it use.
Update
Could you explain to me how a native process communicates with a
tweak, before being loaded within the process space?
It doesn't. Before injecting tweak is a separate dylib that is not linked to any binary. CydiaSubstrate does all the injecting. The main part of the CydiaSubstrate is a special loader dylib. It's linked dynamically to launchd process on device start, to the process which is the first process in the iOS that starts all other processes. When a new process is spawned CydiaSubstrate loader dylib checks all tweak filters to see which ones it needs to inject into the process and injects them. After that tweak is loaded into process address space (becomes a part of the process) and tweak's constructor is called where usually all the hooks are being setup.
Could you explain to me as to how this is accomplished?
Suppose you have an array of objc class names, C/C++ functions, frameworks and dylibs as strings usage of which you would like to detect. There's easy solution. You can open tweak's file and just search through it for any matches. As tweaks are usually not very large it shouldn't take much time. And there's more difficult solution. Use dyld or any other API to parse mach-o sections to find imported symbols and string literals and then search through them for any matches.
I'm not sure if this answers your question from a user side, but if you are making a tweak, you can add XXX_CODESIGN_FLAGS = -Sentitlements.xml to your Makefile to add the entitlements described in entitlements.xml.
I am trying to get some a medium-to-large sized code base that is, frankly, well written with a high degree of portability.
I decided to package it as a loadable bundle (plugin) and piggy-backed off of one of the template app projects and followed some tutorials about adding a target for loadable bundles within an app.
Also, this loadable bundle depends on a custom framework which I built for iOS and added it as a dependent for the loadable bundle. ie. The plugin links to a framework wrapper for a static lib.
The custom framework built successfully. Granted I have not yet verified that it works. The idea is to test the integrated functionality.
My build settings are largely defaults with the exception of some preprocessor defines.
Because I don't really understand the code base yet, I am literally adding one file-at-a-time to the plugin target and building cleanly every 3-4 files added.
The build completes successfully but with many, many warnings as follows, with paths to intermediate build results...etc.:
"file was built for unsupported file format with a series of hex characters () which is not the architecture being linked (armv7s)". When I converted the hex chars to ascii it just showed "#1 /Users/my-username/? ".
When I do a 'file' on any .o in the intermediate build results, I get "ASCII c program text, with very long lines"
What am I doing wrong? What does that mean?
Thank you so much for your time.
The short answer is this:
If you get this message, then your project settings are messed up.
If you are linking your app against custom frameworks, make sure they are built as fat binaries
You will need to know very clearly the meanings of active architecture and how it is used and whether or not you want to only build the active architecture for your app, or all of the possible architectures.
If you are, like me, inheriting a slew of portable code that depended heavily on gcc and its extensions, expect to make changes around builtin* attributes and to make heavy use of __clang to make available macros that used to be defined through the GNUC et al.
Also, you will need to use the -E for clang to debug/understand the preprocessing and the file inclusion. That said, don't forget to take it out because effectively what will happen is that your .o will just contain text and the build may succeed, but the linker will give you the odd message subject of this question.
Finally, do understand that Xcode, like any piece of complex software, is buggy. Sometimes, it will keep settings that you get rid off. In my case, I included custom frameworks which I built after placing them in a local dir. Then I deleted them from the project and opted to trash when prompted. The build kept failing because the linker for some reason was looking for the local directory. You would have to edit the *.pbxproj and manually remove them.
I am building an iPhone application that I need to be position-independent. I am linking against third-party libraries to which I do not have source code.
If such a library has not been compiled with -fPIC and therefore is not position-independent, can I still link against it to produce a valid PIE binary?
Will the dynamic loader handle any text relocations that occur due to code from this library.
If not, what are my possible options to resolve this situation and still produce a PIE binary?
As far as I know, whether a source file was built with -fPIC or not, is not important unless you plan to create a dynamic library out of its object file. Actually most static libraries aren't built in a position independent manner. Not that that this would be forbidden, but it will result in a slightly bigger and slower binary later on (as the compiler/linker will not be able to perform certain types of optimizations).
And a binary runs in an address space of its own, it doesn't have to be position independent at all. That's the reason why dynamic libraries must be position independent as they are shared among different binaries that all require different memory positions for themselves, so it must be possible for the dynamic linker to move the library elsewhere in the process space of a binary at runtime.
Or to say it in even simpler words: Just link against that static library and everything will be fine.
I am uploading my iOS application on iTunes. I am using MonoTouch for compiling my LibGdx Game for iOS. In Android it is hardly 7-8mb. But When I upload on iTunes AppStore then its goes to 78 mb. I dont know why ? Please Let me know.
I have also received this error from Apple.
Dear developer,
We have discovered one or more issues with your recent delivery for "Run Panda Run: Racing". To process your delivery, the following issues must be corrected:
Invalid Executable Size - The executable size of 72037504 bytes exceeds the maximum allowed size of 60 MB.
It's hard to give a definite answer without more details. There's a lot of things that can affect the size of the applications. Let's start with the basic.
What you should check:
First, ensure that your application is not being build with Don't link. That will create very big applications since you'll be AOT'ing nearly the full .NET framework that Xamarin.iOS ships;
Second, make sure you're building for a single architecture (ARMv7). FAT binaries (e.g. ARMv7 and ARMv7s) are build two times and needs twice the space;
Third make sure you have not enabled the Debug build (it's possible to do so in Release build, it's a checkbox). That will create larger binaries to support debugging;
Fourth make sure you're using the LLVM compiler. It takes more time to compile but it generates better (and smaller) code;
Those initial checks are pretty easy to do and are the most common reasons for getting very large binaries.
To understand where the size come from you need to know how the application are built.
The main difference between the Android and iOS version is that JIT'ing (just-in-time compilation) is not allowed on iOS (Apple's rules).
That means the code must be AOT'ed (ahead-of-time compilation) and that process creates much larger executables (because IL is way more compact than native code);
If your code is generic-heavy then the final binary can get quite large since it will need to natively compile every generic possibilities (many cases can be shared, but value-types cannot).
What you can do to reduce size:
First try to reduce your managed code size. The easy way to do this is the enable the linker on every assemblies, i.e. Link all assemblies in your project options.
Many people think it's not worth linking their own code - because they know it will be needed at runtime (so the linker cannot remove it) or else they would not have written that code.
That's half true. The linker might not be able to remove most of your application code but if you're using 3rd party assemblies they are not likely 100% used. The linker can remove that extra code (and also remove, from the SDK, all the code that is kept to support that unneeded code). The more shared code you have the more the linker can help you.
Less IL code means faster AOT time, which translate to faster builds and smaller final binaires (including the executable size).
Note: there's a lot of documents and blog entries on how you can control the linker to skip some assemblies, some types or method from being processed/removed.
Second try to reduce your native size. If you're building native libraries then have a second look at them as they will be statically (not dynamically) linked to your final binary (another rule for iOS applications). Some of them might not be worth (feature wise) their weight in your final binary (and there might be lighter alternatives).
Debugging should not be enabled, as it will make the build unnecessarily large.
For more information refer : https://learn.microsoft.com/en-us/xamarin/ios/deploy-test/app-distribution/app-store-distribution/publishing-to-the-app-store?tabs=windows
I had the same problem but in my case I had minimum os version set to 8 in the info.plist causing a larger .ipa file. I changed this to version 10 and was able to pass the size requirements. Even 10 is a bit generous