I am using Spoon and Espresso to automate UI/Functional instrumentation tests on our android app.
I would like to know if there is a way to distribute instrumentation tests across the multiple connected devices and/or emulators so that i can reduce the test execution time.
Ex: I have say 300 tests that take 15 mins to run on 1 emulator. Is there a way i can add more emulators (say 4), distribute 75 tests to each emulator and reduce the test execution time?
Appreciate your inputs on this.
What you are looking for is called auto-sharding. You have to call the spoon-runner with --shard and add the serials from all connected devices with -serial. You can find the serials with adb devices.
You can choose more than one device in Choose dialog. Press Shift or CTRL key when click.
Another solution is to use Gradle. On the right side of Android Studio choose Gradle, then verification, finally connectedAndroidTest. It would give you the same effect as you would put in console:
./gradlew connectedAndroidTest or gradlew.bat connectedAndroidTest
I mean I would run all test cases on all available devices (physical and emulators). For choosing exactly which test classes you should make tasks in build.gradle.
Learn basics of Groovy programming language to make writing Gradle task scripts more effectively. Here's an example of task written in Groovy: Run gradle task X after "connectedAndroidTest" task is successful
You may also learn about Continuous Integration and its tools like Jenkins, or Travis, which you can configure to run specific test cases on every commit.
As an example please take a look at this build log of my Android project: https://travis-ci.org/piotrek1543/WeatherforPoznan/builds/126944044
and here is a configuration of Travis: https://github.com/piotrek1543/WeatherforPoznan/blob/master/.travis.yml
Have any more question? Please free to ask.
Hope it help
Related
I've build Jenkinsfile for multibranch-pipeline as on this gist: https://gist.github.com/nysander/0911f439bca7e046c765c0dc79e35e9f
My problem is that I want to automate testing on multiple simulators and multiple iOS SDK versions. To make this work I make a lot of duplication in attached code.
Is there any way to make this work in loop and pull list of simulators / SDK's to test from some library, array, etc?
The other thing is that testing as in gist is made in sequence (when I made it parallel it broke - something like Xcode database locked)
Other issue is that tests on tests results summary are shown now 3 times every test, and if some fail on one simulator/SDK I have no idea how to know on which SDK it failed.
Any comments and help appreciated, also if such workflow is bad from the beginning.
I used to develop unit tests on Jenkins, running on multiple simulators. I came up with this https://github.com/plu/pxctest allow me to run testing in parallel, saving time as well. In your case it should be multiple simulators with different SDKs.
Regarding the summary, maybe you can export environment variables to tag every test.
Hope it helps!
I have a project which has an Xcode Tests suite and I want to implement a CI system by using "xcodebuild ... tests" command. The only limitation that I have is the hardware. I'm limited to only 2 Macs and 2 iPads and I also want to build for Debug and Release. The problem is that I have different app flavors and I would like to run a part of them (e.g. all the flavours on debug) on one iPad and the others on the other iPad.
I tried to call multiple times "xcodebuild" commands using the "tests" argument, hoping that it would be parallelized, but the problem is that it kills the tests app instance that is running already on the device.
I spent some time searching for someone who did this, but the only thing that was close to what I need was the usage of multiple iOS Simulators which I'm not comfortable because real device testing is required for the project.
Is it doable to run parallel tests on the same device? Am I missing some arguments that I need to add to the xcodebuild command?
No you can only run one test operation on 1 device at a time. Thats why people use multiple simulators as you can set them all to the same OS if you need to run the same test multiple times.
Not the answer you are looking for, but you could use Xcode server to have multiple bots, one for each flavour. If they all point to the same repo / branch, then they should all run one after the other. It would mean less manual effort building a tool to do it.
I also thought I read somewhere that with Xcode 10 you can load balance across multiple Xcode servers, but can't find a link to it. That might be helpful if you have 2 machines. If not you could duplicate the bots across both Xcode servers. Maybe using the API to control the creation of bots?
Xcode server setup guide
I am new to continuous integration. I have developed windows application and its smoke testing using coded ui. I want to create build and release process in TFS and want to execute coded ui tests bases on every time new build is created. I would like to know if there are any limitations for coded ui when it comes to continuous integration? I am not able to find any article so far which explains about behavior of coded ui in TFS continuous integration
Your guidance in this regard is highly appreciated.
First off, it's worth noting that Coded UI is effectively deprecated; Selenium is recommended for web apps, and Appium is recommended for desktop apps.
That said, you need to use the Deploy Test Agent task to deploy interactive test agents to your target machines, followed by a combination of the Windows Machine File Copy (to deploy your tests to the test agents) and Run Functional Tests task to kick off your tests.
What I need Actually?
We create the iPhone application for Mobile & iPad and the code is always checked in to the repository.
1) When ever the code is checked in to the code repository, that has to under go the automation testing and confirm the build does not failed or the app itself will works as per teh functional test scripts.
2) If there is any Build failure, mail has to be triggered to the developers.
3) The build is sucess and automation scripts are executed and that is also passed, next step is to deploy to the apple store and submit for review, necessary information for apple store is made available in configuration files.
Existing reference in stack overflow:
Continuous Integration for Xcode projects?
**Reference**: http://stackoverflow.com/questions/212999/continuous-integration-for-xcode-projects/17097018#17097018
Continuous integration for iphone xcode
**Reference**: http://stackoverflow.com/questions/1544119/continous-integration-for-iphone-xcode
Some of other references also was checked, which just give me the idea of how to execute functional script during code checkin, which is actual works like any CI tools likes Jenkins etc.
Above said reference are also discussed during 2009/2013, which are evry old.
What is available when researched?
I came to know about using using Hudson on the mac, which is very old version and not much supportive and also found Xcode OS X Server which is a product of apple itself where the reviews are not good and implementation is not feasible for my requirement.
Please share me the the approach of how to achieve this, also is that is possible to do CI process a one touch go for IOS, I found something similar to android with few confirmation from user.
At-least execution of Tests and creating an .ipa file in ios will be great.
I am putting together a bunch of applications to create an automatic building for microsoft platform (the products I chose and the software I will build, both, runs on windows). The products I've chosen are:
Code repository: SubVersion
Continuous integration: CruiseControl
Unit testing: NUnit
Test coverage: NCover
Static code analysis: FXCop
Now I need to choose a bug/issue tracking system (free if possible) that can be, in some way, integrated with the previous products.
What I mean by integration? Well, all these products have a file as output I want to be able to publish errors and bugs found by them into the tracking system.
Do you know some product, some technique or trick that can help me to do this?
Thanks in advance.
First off, these are all tools I have experience with and congratulate you on your choices - these tools will serve you well if you use them wisely.
The most common usage of these tools is that CC would fail the build if certain criteria are not met, e.g.:
A unit test fails
Code coverage falls below a certain threshold
FXCop detects a violation of a certain severity
Because the build would fail and in continuous integration a failed build should be fixed immediately, you wouldn't really need to put those issues into a bug tracking system. Think of build-failing errors as being as severe as the code not compiling - you drop everything and fix right away.