I have a project which has an Xcode Tests suite and I want to implement a CI system by using "xcodebuild ... tests" command. The only limitation that I have is the hardware. I'm limited to only 2 Macs and 2 iPads and I also want to build for Debug and Release. The problem is that I have different app flavors and I would like to run a part of them (e.g. all the flavours on debug) on one iPad and the others on the other iPad.
I tried to call multiple times "xcodebuild" commands using the "tests" argument, hoping that it would be parallelized, but the problem is that it kills the tests app instance that is running already on the device.
I spent some time searching for someone who did this, but the only thing that was close to what I need was the usage of multiple iOS Simulators which I'm not comfortable because real device testing is required for the project.
Is it doable to run parallel tests on the same device? Am I missing some arguments that I need to add to the xcodebuild command?
No you can only run one test operation on 1 device at a time. Thats why people use multiple simulators as you can set them all to the same OS if you need to run the same test multiple times.
Not the answer you are looking for, but you could use Xcode server to have multiple bots, one for each flavour. If they all point to the same repo / branch, then they should all run one after the other. It would mean less manual effort building a tool to do it.
I also thought I read somewhere that with Xcode 10 you can load balance across multiple Xcode servers, but can't find a link to it. That might be helpful if you have 2 machines. If not you could duplicate the bots across both Xcode servers. Maybe using the API to control the creation of bots?
Xcode server setup guide
Related
How can we setup access to different builds so that they can easily / quickly viewed in the simulator..
We have several builds - production, development and one or more feature branches.
From a development and testing point of view, it would be really useful to be able to quickly see each of these versions of our app.
I'm just wondering if there are any services or some sort of configuration we could employ to make this easier.
I'm thinking one major obsticle might be the app id is always the same, so this would stop of from running the builds / app in the same simulator...
I'm wondering if the app could be bundled locally (perhaps from some kind of snapshot) and a script could copy in app to the simulator ?
I guess this could be some kind of tool which would use use the simulator command line tool and swap app builds.
I found a solution, all you need to do is keep a copy of the app and drag and drop it into the simulator
https://medium.com/swift2go/how-transfer-your-app-from-one-xcode-simulator-to-another-f4225da9eb7b
We create iOS and Android apps that are white-labeled. They all use a single code base (one for iOS and one for Android). Whenever we need to make changes to all of our apps (> 100 live in App Store) we rely on Fastlane. We have a "bulk" command that submits each new build to Apple, changing out config variables first and a few files so each app is unique.
This has worked well for us... but... its getting really slow. We'd love to be able to take advantage of some of the continuous development services out there. It seems like they weren't necessarily made for this use case but it might still work?
Ideally instead of running bulk on a local machine we could spin up 100 instances on something like CircleCI and they all run side by side, using our fastlane script to build, submit, etc.
We started by looking into CircleCI. The problem we are running into is they don't allow injection of variables into a job (https://ideas.circleci.com/ideas/CCI-I-690).
Is there a better service for this goal? Is there a tool that was built to achieve this? Struggling to find an alternative to hacking together a bunch of smaller tools.
I think you already identified your first step: You will have to split your fastlane (and other tooling) configuration, so it is possible to build each app in isolation.
Then you can trigger a job for each app on a CI service like for example Travis CI or Azure Pipelines (both have a simple API you can use to start jobs and give them some parameters that will be available to your job) that builds and releases the app.
All the other things (e.g. one big build vs. many small build steps etc.) are just implementation details and will depend on the individual service or tools you choose.
I've build Jenkinsfile for multibranch-pipeline as on this gist: https://gist.github.com/nysander/0911f439bca7e046c765c0dc79e35e9f
My problem is that I want to automate testing on multiple simulators and multiple iOS SDK versions. To make this work I make a lot of duplication in attached code.
Is there any way to make this work in loop and pull list of simulators / SDK's to test from some library, array, etc?
The other thing is that testing as in gist is made in sequence (when I made it parallel it broke - something like Xcode database locked)
Other issue is that tests on tests results summary are shown now 3 times every test, and if some fail on one simulator/SDK I have no idea how to know on which SDK it failed.
Any comments and help appreciated, also if such workflow is bad from the beginning.
I used to develop unit tests on Jenkins, running on multiple simulators. I came up with this https://github.com/plu/pxctest allow me to run testing in parallel, saving time as well. In your case it should be multiple simulators with different SDKs.
Regarding the summary, maybe you can export environment variables to tag every test.
Hope it helps!
What I need Actually?
We create the iPhone application for Mobile & iPad and the code is always checked in to the repository.
1) When ever the code is checked in to the code repository, that has to under go the automation testing and confirm the build does not failed or the app itself will works as per teh functional test scripts.
2) If there is any Build failure, mail has to be triggered to the developers.
3) The build is sucess and automation scripts are executed and that is also passed, next step is to deploy to the apple store and submit for review, necessary information for apple store is made available in configuration files.
Existing reference in stack overflow:
Continuous Integration for Xcode projects?
**Reference**: http://stackoverflow.com/questions/212999/continuous-integration-for-xcode-projects/17097018#17097018
Continuous integration for iphone xcode
**Reference**: http://stackoverflow.com/questions/1544119/continous-integration-for-iphone-xcode
Some of other references also was checked, which just give me the idea of how to execute functional script during code checkin, which is actual works like any CI tools likes Jenkins etc.
Above said reference are also discussed during 2009/2013, which are evry old.
What is available when researched?
I came to know about using using Hudson on the mac, which is very old version and not much supportive and also found Xcode OS X Server which is a product of apple itself where the reviews are not good and implementation is not feasible for my requirement.
Please share me the the approach of how to achieve this, also is that is possible to do CI process a one touch go for IOS, I found something similar to android with few confirmation from user.
At-least execution of Tests and creating an .ipa file in ios will be great.
I am using Spoon and Espresso to automate UI/Functional instrumentation tests on our android app.
I would like to know if there is a way to distribute instrumentation tests across the multiple connected devices and/or emulators so that i can reduce the test execution time.
Ex: I have say 300 tests that take 15 mins to run on 1 emulator. Is there a way i can add more emulators (say 4), distribute 75 tests to each emulator and reduce the test execution time?
Appreciate your inputs on this.
What you are looking for is called auto-sharding. You have to call the spoon-runner with --shard and add the serials from all connected devices with -serial. You can find the serials with adb devices.
You can choose more than one device in Choose dialog. Press Shift or CTRL key when click.
Another solution is to use Gradle. On the right side of Android Studio choose Gradle, then verification, finally connectedAndroidTest. It would give you the same effect as you would put in console:
./gradlew connectedAndroidTest or gradlew.bat connectedAndroidTest
I mean I would run all test cases on all available devices (physical and emulators). For choosing exactly which test classes you should make tasks in build.gradle.
Learn basics of Groovy programming language to make writing Gradle task scripts more effectively. Here's an example of task written in Groovy: Run gradle task X after "connectedAndroidTest" task is successful
You may also learn about Continuous Integration and its tools like Jenkins, or Travis, which you can configure to run specific test cases on every commit.
As an example please take a look at this build log of my Android project: https://travis-ci.org/piotrek1543/WeatherforPoznan/builds/126944044
and here is a configuration of Travis: https://github.com/piotrek1543/WeatherforPoznan/blob/master/.travis.yml
Have any more question? Please free to ask.
Hope it help