I'm planning to create an app for blackberry. The Android and iPhone Versions are almost done. Next on the list is the blackberry one.
So my question is: how good are the simulators? Do I need a real device to test? Which one would you recommend?
It is unlikely you'll be able to create a bugless app without a real device. However totally ignoring simulators would also be a mistake. Simulators are quite good, because they allow to test a substantial part of features on a wide range of device models/OS versions. It would be quite expensive to have a dozen of real devices. :)
Usual points to bear in mind while working on simulators:
real devices are slower in times.
simulators do not support permissions (simulators act as if permissions are always granted regardless of what you actually see).
real device may not support the same type of network transport that your simulator has (TCP, WIFI, BES).
big wireless providers (e.g. Verizon) usually install a slightly customized version of BB OS on their devices, and sometimes it results in a different behaviour (or even bugs).
I've had fairly good luck with the simulators. I've been using the Storm, and I have a real curve. I have a colleague with a real storm so I run everything by him as well.
There have only been 2 times that having a real device has helped me. 1. Making sure I had my install files correct. Since Eclipse just copies them out to the simulator it took me 2 attempts on a real device.
and 2. When testing creating & using a database on the sim card. I got about 95% there on the simulator, and the other 5% was really just verification.
That said, which one depends on what version you're writing for. Storm and some curves are 5.0 (and can be upgraded to 6.0) The Torch is only 6.0
Blackberry simulators are really good
They are exact replicas of the devices
I have worked with storm and also the torch devices
I have faced problems during the connections to the internet
and while using the SD cards (SQLite databases)
Getting images from the SD Card which is not possible from the simulator
If you are developing an application which needs the basic UI components and the native blackberry components, simulators are fine
But if you are really developing something out of the box device is a must
It would be more better if an app can be tested in the device before rolling out
Try your luck
Thank you.
The Blackberry simulators are fine. I believe they are built using the same code as actual devices, so they are pretty much identical to real devices. The only thing that makes a difference is the software that they run.
The simulators will provide you with almost all the same functionality with the exception of things like GPS. I believe I've used an image from an SD card before using the simulator as well...
As far as I know, simulators are set at a specific OS version, whereas in the real world there are tons of different OS versions being used (minor revisions). There have been cases where a feature has worked on the simulator but once it was built and launched on the device, the device shows something different. So if you want to get your app tested, you should test on your TARGET device and OS on simulator and real device.
Related
I am trying to construct an up to date device matrix for our dev teams that lists the mobile devices that we should target.
This is a question that I see come up over an over again. I appreciate that there is no ideal device matrix as each project will have its own requirements, but I think that it could be easy to say that the most popular devices and OS' such as iPhone 5 and iOS 7 would be included as they are a market leading device and OS combo.
Other devices such as iPhone 3Gs would have fallen off the list because now it is just not a significant device in the market.
Ideally the matrix should include Android and iOS phones and tablets.
In general I am looking for one that captures the majority of the market but excludes those that are not worth testing for.
Are there any resources that present this in an up to date clear consumable format?
For iOS devices, take a look at the iOS Support Matrix.
Well, if you are interested in market shares of Android, then go to the play developer console, create an app and take a look at the statistics. It will show you which versions of android are most used in the category that you set for your app. To get an idea of what phones are most used, take a look at this chart http://www.appbrain.com/stats/top-android-phones Testing on some of these and some lower end phones should be satisfying.
In case of iOS for graphics performance, test on a iPhone 4. That has retina display but not very good GPU. If that performs well, then it will do well everywhere.
Otherwise, regarding any mobile stuff, test on any device that satisfies the requirements of your application. Test for screen size and performance if that is crucial.
i* resolutions: http://www.iosres.com/
Android resolutions: don't suppose anything, plan for flexibility
Once you are Ok with what you see on your device and in the simulator, then go ahead and start a private beta with TestFlight, Hockey App or any other platform.
In general, everything depends on your app. Whether it has hardware needs or should work on any device, or only with OS versions above some level... The definitive answer depends on your exact requirements. There is no holy grail and you should rather plan the way you'll handle any issues.
I'm currently developing a mobile website with jquery mobile, not exactly responsive web design. I know I can develop the project in the browser on my desktop PC with some plugins or use some online services or simulators available. But I'm not sure if I missing something really important for test.
Example:
touch/swype events or viewport rotation.
Is necessary purchase some physical devices (smartphone/tablets) to develop/test the project? Why?
Intro
First don't let anyone tell you it is not necessary to purchase a real devices for a test purpose. I will tell you why from an Android perspective, same thing, just in a much smaller manner also goes for iOS development.
Good sides of an emulator testing
It is free, you only need a computer which will run your emulator.
You can test your applications in different cases (different screen resolutions, different OS versions)
Faster I/O and network operations but this is not so much a good point if you calculate how much everything else is slow.
Bad sides
It is slooooooooow, if you never tried to use it you can comprehend how slow it is (iOS emulator is fast like hell in a comparison). It doesn't matter if you have a top of a line hardware PC or Mac it is just that slow.
Emulator is simply to darn buggy, there will be a lot of times when application will work just fine on a read device and it will brake on an emulator.
This also goes other way around, sometimes application will work just fine on an emulator but will brake on a real device, in some case it will not work at all or it will not work on some devices. This is usually a case when working with hybrid applications. for some reason Android web view acts differently on real devices and on an emulator.
Emulator simply don't have some functionalities to interact with a hardware nor it can successfully emulate it. Hardware connection it can emulate even don't work correctly sometimes.
I have talked about how slow it is (because of a converting ARM bytecodes to x86 ones on the fly) but from a graphics standpoint it tends to be even slower so don't expect to do any game development on it.
Real devices comes with much more preinstalled software which may slower your application or in some ways enhance its functionalities.
Real world GPS testing is out of a question
Final notes
If you are intending to work with iOS only emulator can be used to do much of a development. Sheer lack of different screens sizes and hardware diversity makes it a perfect platform for a test purposes. Android on the other hand is completely different story, its emulator is simply useless for test purposes. I have several real Android devices, ranging from Android 2.1 + , different screen sizes and finally hardware architecture. You don't need to believe me but everything I mentioned play a significant role while testing Android applications.
If your main concern is testing your jQuery Mobile application I would still advise you a use of a real device in case of Android while in case of iOS you can successfully use emulator. Android is problematic because transition effects are to darn slow and that includes everything else that requires animations. Swipe will not be a problem and I can vouch it works just fine. Second real problem is a device rotation. jQuery Mobile sometimes can have a problem with it, mostly when used with non responsive 3rd party jQuery plugins (carousels, sliders ....). Third problem is mentioned in my list of bad sides, basically web view used in a emulator acts different then one in a real phone so sometimes you will see one thing in your real device and one thing in your emulator.
It is not necessary to purchase such a device.
For Android there is an emulator provided by the Android Development Kit (ADK). You can use it to configure multiple emulated devices with defferent screen sizes, etc. to test for multiple resolutions and Android (browser) versions.
[edit] Though to really test it for iphones you would need that emulator too I suppose, to make sure the website is correctly displayed in the provided browser.
[edit 2] To test "real" smartphone apps (not webapps), it is better to have a real device at hand.
This very much depends to which level you want to test it before you are happy to hand it over for the usage. After you do that and someone reports a defect, will you be able to see where is the problem (if it works on your PC)?
The development itself can be done in your browser, you can even simluate swipe events by dragging mouse. You don't even need any simulators, you can just make chrome window smaller (most of the devices are using some version of webkit, same as chrome).
However, once it comes to testing, I wouldn't feel great if I didn't know how it looks on the device itself. So I think having at least one device (ideally two with different OS and resolution) is always beneficial.
I would also be unsatisfied if I was working on something of which I could not see the result :)
I would like to start developing for iOS. Coming from an Android developing background, I know that the more types of devices you can get your hands on, the better testing will be, as all devices have wildly different specs, and what may work perfectly in your test device may not even run in another one, let alone look good.
I know that testing on the actual device is very important, as there are many limitations on what you can test on an emulator, so I've decided to get an actual device.
However, there are also tons of devices available in the iOS world! There's the iPod touch, the iPad and the iPhone, each in several different generations and configurations (8GB version vs 16GB version, WiFi version, 3G version, etc.). Not also the screen sizes, but also the aspect ratio is very different across devices, and also the included sensors.
I think that getting an application to run in varied devices should not be difficult, but is it necessary to actually test on all the device types you plan to support? Apple is not renown for its low price, and I would like to keep the starting costs as low as possible.
So, to conclude: Is it necessary/recommended to test on as many device types as you can in the iOS development world?
A small clarification: I'm specially asking if it is possible for there to be compatibility issues related to a specific device/family-of-devices that I would not be able to catch either by testing on the emulator nor a totally different device.
Generally speaking, the major differences in capabilities between testing on the simulator and testing on a device are:
The simulator does not use exactly the same sandboxing as the device. So, for instance, if your provisioning profile is missing your Passbook credentials, this problem will show up on a physical device but not on the simulator.
The simulator doesn't generally support GPS, multitouch, push notifications, Bluetooth, and some other specific features.
On a non-retina display, the simulator view for an iPhone 5+ or (especially!) a retina iPad will be nigh unusable at 100% because its size will exceed the size of your screen.
There are a few, very rare, crashes that occur only on the simulator and a few that occur only on the device.
The simulator does not always support the earliest iOS versions your app supports. For instance, the current version of Xcode (which you must use if you want to build for the latest iOS version) only has simulators from 5.0+ available.
Certain profiling with Instruments is, as far as I can tell, only available in the Simulator.
Now, in my specific case, I try to test on one of each screen resolution I support and one of each major OS version I support.
This boils down to the following array of test devices:
(480x320) iPhone 3GS running 4.3.3
(1136x640) iPod 5gen running the latest 6.x
(960x640) iPhone 4S running 6.0
(1024x768) iPad 1st gen running 5.0
(2048x1536) iPad 4th gen running the latest 6.x
Note that the iPad mini is the same resolution as the iPad 1st gen.
(My choices are skewed towards later iOS versions since I like to implement integration with all of Apple's snazzy optional features as they roll them out. It would probably be a more balanced assortment if one of the 6.x devices were running 5.1 instead.)
If you don't need to support 4.x, I would personally advise against it, since iTunes Connect no longer collects crash reports for it and the simulator no longer offers it. Of course, only you can decide whether you really need to or not, and if you do, focus a lot of your testing there as Xcode does not warn you if you are using APIs that were only introduced in 5.0, which will crash any device running 4.x.
Please note that there are ways to (with significant preparation) downgrade the version of iOS on a device, so if you really want to test more versions than you have devices for, you can (with a lot of effort). But you're probably better off cultivating a strong pool of beta testers for this, anyway.
Whilst it's obviously great to test on all possible devices, the iOS ecosystem is much tighter than Android, so you can narrow down the field somewhat.
You can start by limiting your target iOS versions. That will anyway cut out a number of older devices. iOS6 share of all iOS devices is now probably around 75% 4 months after release; iOS5+iOS6 upwards of 90%. If you are just now starting to develop for iOS, you could probably just target iOS6.
That means your minimum hardware platform is iPad2 / iphone3GS / ipodTouch4
Total list of devices
iPad: 2 3 4 mini
iPhone: 3GS 4 4S 5
iPod: 4 5
Ten devices.
But you won't need to test them all. You could sensibly narrow it to...
iPad: mini + one of the retina models
iPhone: 3GS + 5
for everyday testing.
Obviously if you do want to be more back-compatible, just replace the lower-end testing model for a lower-spec device (iphone 3, iPad 1).
The difference in storage capacity (8GB vs 16GB for example) will be mostly immaterial.
There are some hardware features you will have to pay special attention to, depending on your project. The obvious one is retina vs non-retina displays. Hardware features for location services is particularly nuanced between models.
Lowendmac have a pretty thorough iphone comparison chart...
I think that getting an application to run in varied devices should
not be difficult, but is it necessary to actually test on all the
device types you plan to support?
That depends a lot of what kind of apps you intend to build. For example, universal apps run on both small- and large-screen devices but may present themselves differently on each, so you'd want to be able to try both. Many apps target iPad specifically, so obviously testing on small devices isn't necessary. iPod Touch and iPhone are very similar, so testing on one or the other is often sufficient.
In short, you don't have to own every version of every device, and you don't probably don't have to test on every single version of iOS that's ever been released. But you do want to get your product tested on as many different devices and operating system versions as you can. So, cultivate a group of beta testers who will help you out by trying your app on their devices. The iOS developer program lets you add up to 100 test devices to your account precisely so that you can get your app tested in lots of different circumstances.
Testing on the iPod touch is generally not worth it.
In the iOS world, there is generally the iPhone/iPod, the iPhone 5/iPod 5th generation, and the iPad.
So, that is a total of 3 screen sizes.
If you plan on targeting more than the latest OS (iOS 6), that is where the complexity of testing comes in. Simply targeting iOS 5, and iOS 6 nearly doubles the amount of targets you need to test for.
I try and keep it simple. I test on my iPhone 5, and my iPad. Both of those run iOS 6. For iOS 5 and the small iPhone, I rely on the simulator.
I develop for both Android and iOS, so I get where this question is coming from. I have the luxury of being able to develop on most of the different iOS devices and I would say that in most cases I would feel comfortable testing on the least advanced device my user will be using. If it runs smooth on a iPOd 3rd gen then it is going to run very smooth on an iPhone 4S, 5, etc. For the different screen height for the iPhone 5, the simulator works very good with laying it out.
Things you will need to consider is if your apps have the option to use certain feature only a phone would have, like making calls. Also if you want to make a iPad or universal app, it would be very handy to have an actual iPad, but the simulator does work very good.
I'm learning iOS development and I need to know what hardware I need to test my apps.
Is the iPhone/iPad simulator in Xcode sufficient? Or do I need the hardware? I have an iPad 2, and an iPhone 3G. The iPad 2 is one generation old, while my iPhone 3G is three generations old.
My first project is a basic card game with networking, based on a tutorial.
Opinion: Considering the number of questions I see of the form "this works great on the simulator but not on my device" I'd say that having hardware for testing is necessary. I don't think you need every possible device but certainly ones that cover the features that your app uses.
It depends on features you need.
Example of things you can't test in the simulator:
Push notifications
Performance of an OpenGL game (usually you need a wide set of device to test OpenGL)
The simulator can be used for development, but the simulator is not relevant for efficiency. It is very recommended to testing on a real device too. Some of the services can not be developed on the simulator:
the push notifications
in-app purchase
iCloud services
And you know, that the iPhone 3G is not able to be updated for the lastest iOS (your iPad is able).
The first answer is YES, you need hardware as there are differences between the behaviour of the simulator and the devices. They won't always act the same as the simulator is a bit more forgiving than the device.
For example the simulator will find files (images/sounds/models etc.) even if the case is different between the request and the file name, the device will not find them. And there are more.
An other point is whether to buy/have devices to hold different iOS versions. I don't have them all as this is too expensive for me but I can say that this is a problem. No matter how much you will try to consider the differences between the devices you will always miss something and your app might not work or crash on this device.
Still you can consider this question by looking at the apps that you are going to work on. I would say that if your apps don't use the device hardware (camera for example) and don't have features that might cause problems on different devices you will be able to start with out the devices.
Bottom line is that if you want to deploy good working apps, in most cases it will be better if you could test your apps on a variety of devices.
It's not a requirement to have a equipment to test, but certainly very important. You can test FPS of your app, even not containing hand-made OpenGL. All features that you use on your app, like view effects, are tested for sure on a device. Since simulator uses your mac memory, you won't see any side effects from memory shortage. I believe your best chance is to have a iPhone 4 and your iPad 2.
I've got a newbie question about phoneGap and creating apps for iOs.
If my phoneGap app runs in device emulator in xCode and everything is ok, can I be sure that it will work the same way on a real iOS device when I publish it in the appstore?
The XCode simulator does what the name suggest - 'simulates'. It is not identical to the actual hardware, for a number of reasons. Here are just a few of them:
Performance - your computer has much more memory and processing power available than the phone itself. Your app may run fine on the simulator, but quite slow on the device. This is why it's a good idea to run on the device itself, especially if you're doing stuff which could use up a lot of memory.
Missing features - the simulator doesn't allow certain things to be tested, like in app purchase or media/asset management. And obviously you're missing things like the camera, the accelerometers, compass, etc. You can from iOS 5 simulate certain things like the GPS, but nowhere near everything.
Visuals - the simulator runs at your monitor resolution, whereas the phone itself has a much higher DPI display. Things that look readable or fine on the simulator can on device look very different.
If you're not testing your app on an actual device before releasing it to the app store you're doing a disservice to your users - this is Apple's point of view, and one shared by most developers. It's not necessarily what you might want to here, but unfortunately the simulator really is just that - a simulator. You wouldn't want to be in a plane piloted by somebody who had only trained on a simulator. And you probably wouldn't want to use an app that had only been run on the iOS simulator.
Here's what Apple have to say about it in their own documentation:
Although you can do much of your debugging and testing of an iOS application using iOS Simulator, simulation cannot completely match the results of running your application on the target devices; you must test your application on actual devices to ensure that it runs as intended and to tune it for performance on actual hardware.