I am working on automating my iOS project's build process using Jenkins and xcodebuild command line tool. I am also able to test the application on simulator using UIAutomation testing scripts (instruments command line utility). But I would like to use other instruments like Time Profiler, Leaks, etc.. along with the automation to automatically test for irregular behavior.
Although I can make a custom template and run it with automation, but the problem is I can not parse the results of these other Instruments. Is there a way, for example, to know that there is a leak in my code, and trigger an email (from jenkins)?
It's possible (I've done it), but it is terribly complicated to do well. You need something to translate the Instruments results plist to something Jenkins can understand. You can use XSLT (I wouldn't recommend it), or write a script in perl/python/etc.
The Jenkins XCode plugin is an excellent tool for automating Xcode from Jenkins.
https://wiki.jenkins-ci.org/display/JENKINS/Xcode+Plugin
That aside, building a CI environment around UIAutomation is doable, but there are a lot of gotchas - the Jenkins user has to have the correct set of permissions to use the dev tools, has to be in the right group, and has to be logged into the UI for the simulator to run correctly from Instruments. Make sure you can get that far before you invest too much time into parsing the output of the Instruments run.
Related
I am using Appium for automation test cases in iOS project. I would like to create a report of code coverage. I have gone through appium documentation as well as on internet forums, but didnt find any way.
Is there any way to get it for iOS (Appium)?
It is not possible to do a code coverage because appium doesn't test the actual source code. Appium tests only from the UI level. You might have to manually map the functional test cases and see which all functions it get called when the test is done. But certainly there is no way to figure out like UnitTests with a report to see percent wise code covered report.
I need to know if I can do Continuous Integration with XCode server. In example: set run the tests every night or when someone commit changes, and more..
I am trying to decide one iOS Ui Automation tool to integrate with my Xcode server
Thanks
There are a few problems here:
UIAutomation has no built in support in Xcode Server. I've filed bugs, I've chased down people at WWDC. Most I've ever gotten on this problem is basically "shrug." I'm not sure UIAutomation is a priority for Apple right now. So you're not going to get any official support.
As was noted, you might be able to use a trigger. The trigger won't be able to add anything to the Xcode Server report, besides possibly the error logging. But you're not going to get anything added to the nice report table.
Running on actual devices has traditionally been a problem (if you care about that.) The loading the app part has been a problem for us, but Xcode Server might be able to preload the app for you. In addition, it seems like this might have changed in the iOS 8 SDK.
There is just a lot of uncertainty around this sort of workflow. I'm hoping Apple eventually makes an announcement or adds a new tool, but the best answer I've gotten is if you want to go down this path, use UI Unit Tests. That's a shame because it requires knowledge of Obj-C or Swift, and means interacting with the app at an API level instead of an abstract level, but if you're looking for the direction Apple wants to see people go, that's it.
Edit 7/4/2015: As of WWDC 2015, there is a new UI Testing component as part of Xcode 7 that, in my experience, seems totally supported, and is promising Xcode Server support. I would very strongly recommend using that, and not using the Instruments UIAutomation tool.
With Xcode6 right around the corner they are adding some features to XCode Server specifically it looks like "Triggers" will be helpful for running iOS UIAutomation tools. Since you can run UI automation scripts from cmd line it should be possible to utilize triggers to run your scripts post builds. This along side the logic for when a bot should run will let you decide if it should be nightly or on every commit.
https://developer.apple.com/library/prerelease/ios/documentation/DeveloperTools/Conceptual/WhatsNewXcode/Articles/xcode_6_0.html#//apple_ref/doc/uid/TP40014509-SW1
I wrote a framework around UIAutomation called Illuminator to handle tasks like nightly test runs, pull request tests, and other automated conveniences.
It provides a flexible and extensible command line that can execute any particular subset of tests that you'd like, and produces reports in JUnit format (used by Jenkins).
I want to test my iOS app along with a memory profiler. I hope to use XCode inbuilt memory profiler with Calabash testing tool. But I can't run both at once. What I want is to integrate the both and analyze memory usage while testing the app. If calabash can't do this I want to know atleast XCode Automation tool can integrate with the XCode memory profiler.
can someone help me out
I couldn't find a way to use Calabash tool and Allocations tool together. But I found a way to use Allocations tool and Automation tool together. In XCode profiler first choose Automation tool and then click library tool in the tool bar. After that add the tools you need(In this case Allocations tool)
This is possible, but the results are pretty hard to understand. I use this to track down leaks in the Calabash iOS Server.
You'll need an Automation + Leaks/Allocations template and you'll need to use the TEMPLATE environment variable.
The next time I use this technique, I will generate a document that describes the process.
For some of my iOS application projects, I would like my CI server to be able to report the following properties:
startup time
frame rate
both as a graph over time, and with "low water marks" so the build fails if the measured results aren't within certain criteria. I've already found some of the things I need.
The CI server will be Jenkins.
I can use Transporter Chief to get the built app onto an iPad.
To measure the startup time I could find the duration between launching main() and leaving application:didFinishLaunchingWithOptions:.
To measure frame rate I can put a CADisplayLink into the app and sample its duration property.
If those tests output JMeter XML, then Jenkins can display the output via the Performance plugin.
What I haven't worked out is, how should I embed those tests into my app and launch it on the iPad? As described above I can deploy the app to the iPad, but then I don't know how I would launch it to gather the results of the tests. My unit tests are running on the simulator - I don't want to run the performance tests there obviously :-).
I imagine that there's a solution involving jailbreaking the iPad and controlling the app over SSH, I'd prefer not to go down that route if it's possible. If you have done that and can explain how it works, I'd still like to hear about it.
I'm also using fruitstrap to install apps on the device in CI. In terms of booting the app, I know of two ways:
Use fruitstrap with the debugger attached
I know teams that have done this to run KIF integration tests on devices in CI. I have played around with fruitstrap to get it booting apps on the device, but haven't myself taken the extra step of automating the whole thing
shameless plug for my post on fruitstrap: http://www.stewgleadow.com/blog/2011/11/05/installing-ios-apps-on-the-device-from-the-command-line/
Use the instruments command line tool with UIAutomation
I know the instruments tool can boot apps on the device automatically in CI (I wish it also installed them, but we have fruitstrap for that until Apple fixes it). So you could write a really simple little UIAutomation test that gave your app enough time to do its performance analysis.
Jonathan Penn has a nice little demo project for UIAutomation and build script that could be integrated with an 'install' step using fruistrap to try it out on the device
In both cases, I uses a little wrapper around libusb to give me the device ID of attached devices, so the more devices I plug into my CI machine, the more devices it runs tests on, https://github.com/sgleadow/iphone_detect
Can you launch the app on the device using lldb?
If so, it may also be able to capture the log output.
Is there any tool in blackberry sdk like adb in android which can be used to install or run apps from command prompt on simulator.
Once you've packaged a .cod file to deploy, you basically have two options, which Howard and seand mentioned:
Using javaloader: javaloader -u load <path-to-cod>.cod
Using fledge and fledgecontroller
I'm guessing based on your question that this is part of an automated deployment scheme? I'd recommend using javaloader when deploying to real devices, and using fledge/fledgecontroller for deploying to the simulator.
javaloader will require that the simulator is simulating a usb connection, and a lot of people seem to have had trouble getting it to work correctly - when we tried there was about a 50% chance for any given run that it wouldn't connect or deploy correctly. But it works a bit better when trying to connect to a real device, at least in my experience. And if your project and requirements are fairly simple, you might be able to get away with using it on a simulator without too many problems.
Fledge/fledgecontroller have more commands that are useful for doing automated deployment and testing, which is primarily why I recommend them. They are a bit more complicated to use, however. To deploy to the simulator with fledge and fledgecontroller, you'd want to do the following:
Start fledge (located in the simulator folder) with fledge.exe /handheld=<handheld-id> /session=<session-name> /app=<simulator-folder-path>/Jvm.dll /automate
Use fledgecontroller (also located in the simulator folder) to load the cod with fledgecontroller.exe /session=<session-name> /execute=LoadCod("<path-to-cod>.cod")
<handheld-id> is the id of the device you want to test on - e.g. "8300" would run the 8300 simulator for BB 4.5
<session-name> can be anything, but should match between the call to fledge and the call to fledgecontroller
Note that it will take a little bit of time (anywhere from 15 seconds to 5 minutes, depending on which simulator you are using) for the simulator to boot after calling fledge. You'll need to wait for it to finish before calling fledgecontroller.
Fledge also has a bunch more commands that you can use to configure the session, for example by setting up an sd card or language options. If you run fledge.exe /help, it will describe the different parameters; and fledge.exe /controller-help will list the different commands you can pass through fledgecontroller.
In summary: if you are using real devices, or possibly if you are doing very simple deployments on a simualtor, you can probably get away with using javaloader. If you want more control over the simulator, you should use fledgecontroller.
Most probably the JavaLoader is the tool you are looking for.
Simulators are launched using the 'fledge.exe' CLI tool. Look at the built in scripts for launching simulators to see the myriad of options.