I want to test an app on multiple emulators with different screen sizes. I prefer to have just one build plan instead of a build plan for every emulator configuration.
Is is possible to run e.g. 3 emulators with different configurations using Android Emulator Plugin for Jenkins?
Is there some other plugin to achieve this?
Using the Matrix plugin is a good solution for this, as discussed on the wiki page for the emulator plugin.
This lets you have a single Jenkins job, where you define the various configuration combinations you want — Jenkins will then run a series of sub-jobs to fulfil each combination, in turn creating the desired Android emulator config.
Related
We create iOS and Android apps that are white-labeled. They all use a single code base (one for iOS and one for Android). Whenever we need to make changes to all of our apps (> 100 live in App Store) we rely on Fastlane. We have a "bulk" command that submits each new build to Apple, changing out config variables first and a few files so each app is unique.
This has worked well for us... but... its getting really slow. We'd love to be able to take advantage of some of the continuous development services out there. It seems like they weren't necessarily made for this use case but it might still work?
Ideally instead of running bulk on a local machine we could spin up 100 instances on something like CircleCI and they all run side by side, using our fastlane script to build, submit, etc.
We started by looking into CircleCI. The problem we are running into is they don't allow injection of variables into a job (https://ideas.circleci.com/ideas/CCI-I-690).
Is there a better service for this goal? Is there a tool that was built to achieve this? Struggling to find an alternative to hacking together a bunch of smaller tools.
I think you already identified your first step: You will have to split your fastlane (and other tooling) configuration, so it is possible to build each app in isolation.
Then you can trigger a job for each app on a CI service like for example Travis CI or Azure Pipelines (both have a simple API you can use to start jobs and give them some parameters that will be available to your job) that builds and releases the app.
All the other things (e.g. one big build vs. many small build steps etc.) are just implementation details and will depend on the individual service or tools you choose.
I've build Jenkinsfile for multibranch-pipeline as on this gist: https://gist.github.com/nysander/0911f439bca7e046c765c0dc79e35e9f
My problem is that I want to automate testing on multiple simulators and multiple iOS SDK versions. To make this work I make a lot of duplication in attached code.
Is there any way to make this work in loop and pull list of simulators / SDK's to test from some library, array, etc?
The other thing is that testing as in gist is made in sequence (when I made it parallel it broke - something like Xcode database locked)
Other issue is that tests on tests results summary are shown now 3 times every test, and if some fail on one simulator/SDK I have no idea how to know on which SDK it failed.
Any comments and help appreciated, also if such workflow is bad from the beginning.
I used to develop unit tests on Jenkins, running on multiple simulators. I came up with this https://github.com/plu/pxctest allow me to run testing in parallel, saving time as well. In your case it should be multiple simulators with different SDKs.
Regarding the summary, maybe you can export environment variables to tag every test.
Hope it helps!
So I have 3 Appium/Selenium Junit classes, each for a different platform (iOS, Android, web). I'd like to create 3 Jenkins jobs and then run some or all of them in parallel based on parameters that I'll input somehow (file, command, etc.).
Example: I want to run the iOS and Android tests with specific devices (UDIDs). So I'd somehow input this information (which jobs to run and with which parameters=devices), and the 2 jobs will run concurrently with this input.
I'm a Jenkins beginner, and I have tried searching and found many Jenkins plugins that seem like they can help (i.e. Parameterized Trigger Plugin), but I didn't understand how to use these correctly (how to configure the jobs correctly with these plugins). Any help would be appreciated (as "dumbed-down" as possible...)
So it's like this. You can run tests on Jenkins in parallel or in multiple builds. I'm using maven, multiple Jenkins nodes, selenium-grid and bunch of other tools. You can call single maven command so it can run parallel tests at once, for eg. only mobile (Android/iOS), and web in separate build to run tests in parallel run for multiple browsers eg. (Chrome & Firefox...).
There is a several ways to achieve this, but there is lot o preparation and setup:
via java/testNg (xml test-suites),
set up selenium-grid for multiple instances,
Here is great starter article for this: https://www.swtestacademy.com/selenium-parallel-tests-grid-testng/
https://learn.techbeacon.com/units/how-use-testng-parallel-test-execution
(if using for mobile testing) setup Appium to run several instances, here is also nice article to start with: https://appium.io/docs/en/advanced-concepts/parallel-tests/
have multiple Jenkins nodes, here is something about that, read entire article: https://wiki.jenkins.io/display/JENKINS/Distributed+builds
But main question was about Jenkins, You can achieve this by creating simple Maven build, create Your job, connect it with Your code on git, enter maven command to call Your code, and You should be set to go, again one more article: https://developers.perfectomobile.com/pages/viewpage.action?pageId=21435209.
Good parallelism is hard to achieve, but when You do its is going to make You worth all time invested.
Hope It will help You!
I am using Spoon and Espresso to automate UI/Functional instrumentation tests on our android app.
I would like to know if there is a way to distribute instrumentation tests across the multiple connected devices and/or emulators so that i can reduce the test execution time.
Ex: I have say 300 tests that take 15 mins to run on 1 emulator. Is there a way i can add more emulators (say 4), distribute 75 tests to each emulator and reduce the test execution time?
Appreciate your inputs on this.
What you are looking for is called auto-sharding. You have to call the spoon-runner with --shard and add the serials from all connected devices with -serial. You can find the serials with adb devices.
You can choose more than one device in Choose dialog. Press Shift or CTRL key when click.
Another solution is to use Gradle. On the right side of Android Studio choose Gradle, then verification, finally connectedAndroidTest. It would give you the same effect as you would put in console:
./gradlew connectedAndroidTest or gradlew.bat connectedAndroidTest
I mean I would run all test cases on all available devices (physical and emulators). For choosing exactly which test classes you should make tasks in build.gradle.
Learn basics of Groovy programming language to make writing Gradle task scripts more effectively. Here's an example of task written in Groovy: Run gradle task X after "connectedAndroidTest" task is successful
You may also learn about Continuous Integration and its tools like Jenkins, or Travis, which you can configure to run specific test cases on every commit.
As an example please take a look at this build log of my Android project: https://travis-ci.org/piotrek1543/WeatherforPoznan/builds/126944044
and here is a configuration of Travis: https://github.com/piotrek1543/WeatherforPoznan/blob/master/.travis.yml
Have any more question? Please free to ask.
Hope it help
I am using the combination of Jenkins, python, unittest, nosetests to run test suites. I publish the results in nosetests.xml to Jenkins using Junit plugin.
My question: How can I run the same test suites with different browsers (Chrome, FF, IE,…,etc) and publish all the results in the same Jenkins job and still be able to distinguish each test result with respect to browser it used?
I am thinking about running the test suite many time, each with different browser and rename the tests in each nosetest report, nosetests.xml, before publishing to Jenkins but I don’t think that it is an elegant solution.
Thanks for your help
This is what multi-configuration jobs (or matrix jobs) are designed for in Jenkins.
You specify your job configuration once, but add one or more variables that should change each time, building a matrix of combinations (in your case, the matrix has one dimension: browser).
Jenkins then runs one main build with multiple sub-builds — one for each combination in the matrix. You can then clearly see the results for each combination.
This requires that your test job can be parameterised, i.e. you can choose at runtime which browser should be run, rather than running all tests together in a single job.
The Jenkins wiki has minimal documentation on this feature, but there are a few good blog posts (and Stack Overflow questions) out there on how to set it up.