I did a search on this but couldn't find a match, possibly due to several meanings of the phrase "load testing"... but what we are trying to do is to make sure our App (which is a medical device that MUST display updated data in a timely fashion) will process data at a required throughput, even if the iPhone gets busy -- e.g., if there are a bunch of apps open that require CPU attention.
The best test approach we can come up with is basically that... open a bunch of apps, and perhaps try to load the CPU by sending it a bunch of emails with large attachments.
But obviously that is just kind of a stab at the risk. Are there any better techniques out there, or tools? Thanks for any help.
From the perspective of your app development, you need to focus on fixing:
"What happens when data doesn't send?"
"What happens when the app gets closed by iOS for whatever reason?"
There's a million ways to create scenarios that will cause bad cases to happen. Knowing you are covered from general failure is the safest thing to rely on.
From the perspective of validating an existing app:
In past scenarios, when I haven't had control over the development of the app, I've tested these scenarios through manual exploratory testing on physical devices.
Nowadays the iPhone Simulator now provides lots of utilities so that you may not need to do this all manually. Scroll to the bottom for these answers.
Testing when the device doesn't reliably transfer data
Nontechnical way: Go on 3G and Airplane mode and toggle connectivity frequently as you use the app.
More technial way: set your phone up to proxy through your computer. Throttle the connection. Charles does this.
As you throttle the connection, use the app to do any transactions that talk to the backend.
Do they get sent correctly once you reconnect to Wi-Fi or a more stable Wireless connection?
Testing when the CPU is overloaded
Get the oldest phone that supports your highest target OS (e.g. iPhone 4S on iOS 9) and open a few games in the background.
Use your app for about 30 minutes aggressively.
By aggressively I mean, pretend to be an impatient user that taps 5 times on every button and everywhere else on the screen until you start to notice slowness.
Exploit the areas that seem to cause the slowness.
Hot spots are: animations, videos, things that cause data to go to the backend, tableviews, collection views, etc.
See what happens to the app when it crashes.
Did you get logged out?
Was any key data malformed or lost?
Using the simulator for these things
A note about using the simulator.
The simulator is a convenient way to simulate memory warnings and you can throttle your connection with Charles, or force a reboot. But the CPU will not be throttled in the same way as if you tested on a physical device.
That aside, see the below screenshots for what options are available on the simulator that might help you.
Hope this helps!
Related
I am facing a very weird issue: my iPhone is restarting.
This issue occurs in a particular scenario only.
Step 1: I have a sync process in which I'm loading data for the whole app. So I'm basically doing a heavy API call by uploading 4-5 camera captured images and syncing the app data;
Step 2: After syncing, I'm pressing the iPhone home button to make the app go in background;
Step 3: I'm locking the iPhone screen(by using side button);
After a few seconds I'm seeing the apple logo and the phone seems to restart. This is not replicating when the app is connected in debug mode. I checked the debug navigator app is using only 125 MB of memory, disk and network values is 0%. Energy Impact is showing high, I'm not sure this is due to high energy impact. I'm facing this issue only on iOS 12.4.
The fact that the phone (or possibly just springboard) restarts, and not just your application means this is Apple's bug. You're not supposed to be able to crash iOS even if you try.
Finding a likely cause will be hard since the system is not behaving the way it's supposed to. The device's logs may have more information from things other than your app. This may be a system API breaking due to any number of actions from your application.
Often with this kind of thing the next OS version will fix it, but if that's not the case or it's important to track down I would try removing ways you're interaction with the system (background APIs, notifications, etc.) and see if anything fixes the issue.
If you find the problem, you may even be doing things the "correct" way according to the documentation and have to find a workaround. If you have the time you can submit a bug to Apple so the underlying issue has a better chance of being fixed.
It seems when your app is in the background and phone locked, Automatic Reference Counting (ARC) closes some connection or deallocates a resource that makes the iPhone restart. Are you closing all connections and removing all references once the upload is complete?
Phones do not spontaneously restart just because of an app’s actions. You’re having an issue with your phone, not with a program. You need to repair or replace the phone.
My app has a bug, which I am unable to replicate when it is run when plugged into the computer.
I see it three times a day and it is quite annoying.
Is there any way to capture the state of the app (when not connected to the computer) and see what's going on? Kind of like what XCode does when you use a breakpoint, but somehow send the data to it (or some order tool that I am unaware of) wirelessly.
UPDATE:
The app does not crash. It's just that certain UITableViewCells don't work as they should at certain times.
Try using Crashlytics its awesome for crashes. You should also use TestFlight which will give you live sessions/logs of your app. These both combined will tell you many things.
This is a strange one, I have a need to create an iOS app that runs in the background on an iOS device, but can not be visible on the Home screen of the device. The app may need to show up in Settings to configure a few options, but it mostly needs to run behind the scenes.
I do not need to publish this app on the app store, it is strictly an enterprise app for my company.
Does anyone know how this kind of behavior might be achieved? Configuration profiles? API's? etc?
Edit: Jailbreaking the device is not really an option for us. We have to keep the devices as they are.
Backgrounding
There are several methods to get permanent background execution:
a) Silent audio which is mentioned by JRG-Developer
b) Usage of beginBackgroundTaskWithExpirationHandler + turning location manager on/off (it will reset remaining time to 600 seconds)
Making app invisible
You can use SBAppTags in Info.plist (take a look at this: How to remove app icon programatically from jail broken iPhone app?)
It will make your icon not visible on Home screen.
I've no idea how you can tackle the not be visible on the home screen criteria, but there are certain services that are allowed to be run in the background continuously, notably:
Background Audio (even silent audio)
While this is indeed very hacky, short of jailbreaking the devices, this may be your best bet.
While it's unlikely / very difficult to get past the review process, in the event for some reason you do need to (attempt) to publish this app to the App Store, some apps are even available on the App Store which take advantage of this hack, such as PasteBot.
You should definitely read their write-up here about getting their app's Cut-and-Paste functionality to work while in the background:
http://tapbots.com/blog/pastebot/pastebot-music-in-background
Another out-of-the-box idea, why not use Android devices instead, which do support multitasking?
Two methods:
Jailbreak. A jailbroken iPhone is essentially a Mac and you can use all Mac UNIX programming tricks - spawn a daemon (you can even ask launchd to do that), or something. You can set up enteprise-wise Cydia source. Refer to saurik's website for a walkthrough and set up an experimental server using Ubuntu (which used apt too)
A non-jailbreak way may be possible, but it will depend on what your app is doing. You can try use enterprise-wise push notifications.
Given that this is impossible to do on non jailbroken phones, I'd suggest the following:
develop a directory app, or a phone list app, something that is specific to your company.
have it connect to a web service once a day to register the device ID (so you know the user has not deleted the app)
tell users they need to keep the app open at all times (and if a user's device does not register one day, send them an email asking them to launch the app).
Now you can send back whatever you want with the app. If they kill it, you'll know the next day when you have a script look at the logs.
If you think this is a terrible idea, it is, but its the best you are going to get on iOS right now.
EDIT: you could send a notification every day - say in the AM - to make sure the user opened the app if it wasn't running.
From my experince, this kind of behaviour cannot be achieved on iOS. When an app enters into background, it is active only for a small particular amount of time. After that, it goes into hybernate state. It stops working. So the behaviour you said can not be achieved without abusing iOS workflow. Thats why some call iOS's multitasking as not true multitasking. Only jailbreaking might help your case. That was for one case. The second one where the app cannot be seen on home screen is again, impossible. It has to be on springboard to run.
I am developing a Blackberry Java application and I need an update count in the app such that the user will know how many notifications they have on the phone. The number of notifications show up on the home screen beside the icon. The way I am keeping track of the icon variable "iconCount" between different entry points is through a singleton. I need to have this count be present even after the user turns off their device on and off. So if there are 7 updates available for them 7 will show up until they check the application. It will not reset to 0 when the device is turned off.
As a test to see what would happen if I set the count to 7 and turned off the device through the simulator; I held down the power button until it stated "turning device off, press any key to abort" then the screen went black. I waited a few seconds and pressed the hangup button again and the simulator powered up to show 7 new notifications.
This is a bit strange to me, I though you would need persistent store to accomplish this? Was I turning off the device properly, or does a singleton in fact hold the variable even after the device is shut off?
Turning the device off in that manner does not really turn it off, that is why your singleton retains its value. If you want the value to persist accross reboots or battery pulls (which really turns it off) then you will have to put it in the persistent store.
A static member (as the singleton) will be unique inside a given application. But because BlackBerry Java implementation is so "particular" (to say the least), for each alternate entry point, you'll have a different "application instance", so several instances of your singleton will be created.
If you want to share an object at runtime between several applications, you have to publish it in the RuntimeStore. If you also need persistence, then use PersistentStore, or plain files, or sqlite databases in newer OSes.
And now about the simulator: don't trust the simulator. Once you close your app your variables will vanish. Even if you have a background process, when you turn off the device it won't exist any more. Test on a real device.
Remember: BlackBerry is not Java. You'll see many violations of the Java Specification (like for instance, the fact that the Persistable interface is not inherited) so get used to it.
There is nothing inherent in the Singleton design pattern that causes any sort of persistence. That said, it is possible that the BlackBerry JVM or your code is doing something special. Without a code sample, it is impossible to know.
Otherwise, when the singleton is unloaded from memory (and not put on disk), the data will evaporate.
The likely case is that the BlackBerry OS is essentially "hibernating" when it turns off and restores the application memory state on boot.
Sorry for the vague title, but not quite sure how to summarize this one.
The facts are:
I have a game that's been approved by Apple and is on the App Store.
It is a universal app. It uses textures designed for 320x480 on small screens, and uses larger textures (roughly four times as large) on retina and iPad screens.
While developing it, I would sometimes see low-memory warnings in the console log, but after reading about these it seemed like they were often somewhat spurious/unimportant, and in any event I was not having crashes, and my testers on a variety of devices (iPod Touch 2nd gen, iPod Touch 4th gen, iPhone3, iPhone4, iPad1, iPad2) were not seeing crashes.
When I started distributing the app to a wider set of beta testers through TestFlightApp, I got reports of some people seeing crashes as the app was loading, or very early after the user had chosen a level from the main menu and the app was loading the level textures. We discovered that if these users just restarted their devices, they didn't have problems any more. Since this was the first time we had seen problems like this, we attributed it to something TestFlightApp was doing, some funny state it was leaving the device in after the install (we talked to TestFlightApp about this and they had never heard of such a thing).
As stated, Apple approved the app and it's on the App Store. Soon after it went live, we got reports from some iPad1 users that it was crashing for them on app load, or soon thereafter, same kind of thing as we saw with certain TestFlightApp users. And again similar to the TestFlightApp users, these customers reported that restarting would often fix the problem. But it wasn't as nice because the problem tended to appear again. One of these users sent me several LowMemory...log files that she got off her PC after synching her iPad. There were about 10 such files, and none of them listed my program in the Processes list. Instead it showed other programs marked as either (active) or (jettisoned), and the "Largest Process" could be anything from MobileSafari to Kobo, but again my own app was never listed. So, I didn't understand that, but the bottom line seems to be that, for this user at least, something is pushing the memory over some limit where my app won't run well.
I have since gone back and talked to one of the TestFlightApp beta testers, and it turns out that he does indeed sometimes get the app crash again, so it wasn't just some residue from TestFlightApp. However, for him the crash is much less frequent than it is for this customer.
Other iPad1 testers of the game have never had any trouble. They report that they play the game for hours each day, use their iPad with several other apps in between, and rarely power it down. Similarly, I never had a crash with my iPod Touch 4th gen, which is similar to the iPad1 in the at it has a hi-res screen but only 256k RAM.
So, it's very mysterious to me what could be so different about these particular users' iPads. It's mysterious that the game works after the device is restarted, but then after some apps have been run the game (sometimes) has trouble loading. My understanding was that if my game demands memory, the OS will auto-close whatever other apps are running, as necessary, to effectively bring the amount of memory back up to the amount that's available on a freshly restarted device. My only conclusion is that after running some apps the device is left in a state where less memory is available because the OS cannot reclaim certain memory blocks or shut down certain apps.
Unfortunately I don't have one of these "misbehaving" devices to develop with. All I can think to do is try to reduce the memory needs of my app by a certain amount, and send it to one of these users who is having trouble and see if it fixes things. That seems like a potentially inefficient approach, however.
Anybody have a better idea?
Sounds like the memory spike during texture loading is what's causing the app to be terminated on some devices. It may well use less memory after everything's loaded than it does right near the end of initial loading. This could be explained by things being pushed to virtual memory, whereas direct texture loading could be bombarding the RAM with way too many allocations. My suggestions would be to:
Be more aggressive with destroying temporary data structures during loading (release a temporary structure the instant all of its useful values have been read/extracted by other things)
For autoreleased objects, keep an NSAutoreleasePool around at all times; you may even want to drain and realloc a pool several times over the course of one method if you use an exceedingly high number of autoreleased objects.
This may sound silly.. intentionally slow down your loading process. If you get rid of parallelized loads (loading multiple objects at once) or possibly insert a manual time delay in your loading thread/methods, this may give the OS more time to push things to virtual memory and thus Watchdog will not detect the app as being a RAM hog.
EDIT: One possible tactic to implement slower loading: if/when you receive a low memory warning, pause or slow loading down for a few seconds to give other apps time to lower their memory usage, then continue loading at normal speed.
Even if I'm wrong (if LowMemory...log files show Virtual+Physical usage and thus your app isn't even doing that much), I would suggest then integrating bug reporting such as QuincyKit so that you get emailed a backtrace and crash description when this bug IS encountered in the wild.