I am using the combination of Jenkins, python, unittest, nosetests to run test suites. I publish the results in nosetests.xml to Jenkins using Junit plugin.
My question: How can I run the same test suites with different browsers (Chrome, FF, IE,…,etc) and publish all the results in the same Jenkins job and still be able to distinguish each test result with respect to browser it used?
I am thinking about running the test suite many time, each with different browser and rename the tests in each nosetest report, nosetests.xml, before publishing to Jenkins but I don’t think that it is an elegant solution.
Thanks for your help
This is what multi-configuration jobs (or matrix jobs) are designed for in Jenkins.
You specify your job configuration once, but add one or more variables that should change each time, building a matrix of combinations (in your case, the matrix has one dimension: browser).
Jenkins then runs one main build with multiple sub-builds — one for each combination in the matrix. You can then clearly see the results for each combination.
This requires that your test job can be parameterised, i.e. you can choose at runtime which browser should be run, rather than running all tests together in a single job.
The Jenkins wiki has minimal documentation on this feature, but there are a few good blog posts (and Stack Overflow questions) out there on how to set it up.
Related
So I have 3 Appium/Selenium Junit classes, each for a different platform (iOS, Android, web). I'd like to create 3 Jenkins jobs and then run some or all of them in parallel based on parameters that I'll input somehow (file, command, etc.).
Example: I want to run the iOS and Android tests with specific devices (UDIDs). So I'd somehow input this information (which jobs to run and with which parameters=devices), and the 2 jobs will run concurrently with this input.
I'm a Jenkins beginner, and I have tried searching and found many Jenkins plugins that seem like they can help (i.e. Parameterized Trigger Plugin), but I didn't understand how to use these correctly (how to configure the jobs correctly with these plugins). Any help would be appreciated (as "dumbed-down" as possible...)
So it's like this. You can run tests on Jenkins in parallel or in multiple builds. I'm using maven, multiple Jenkins nodes, selenium-grid and bunch of other tools. You can call single maven command so it can run parallel tests at once, for eg. only mobile (Android/iOS), and web in separate build to run tests in parallel run for multiple browsers eg. (Chrome & Firefox...).
There is a several ways to achieve this, but there is lot o preparation and setup:
via java/testNg (xml test-suites),
set up selenium-grid for multiple instances,
Here is great starter article for this: https://www.swtestacademy.com/selenium-parallel-tests-grid-testng/
https://learn.techbeacon.com/units/how-use-testng-parallel-test-execution
(if using for mobile testing) setup Appium to run several instances, here is also nice article to start with: https://appium.io/docs/en/advanced-concepts/parallel-tests/
have multiple Jenkins nodes, here is something about that, read entire article: https://wiki.jenkins.io/display/JENKINS/Distributed+builds
But main question was about Jenkins, You can achieve this by creating simple Maven build, create Your job, connect it with Your code on git, enter maven command to call Your code, and You should be set to go, again one more article: https://developers.perfectomobile.com/pages/viewpage.action?pageId=21435209.
Good parallelism is hard to achieve, but when You do its is going to make You worth all time invested.
Hope It will help You!
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
I am trying to run UI tests on multiple identical instances of the web application. For example, let's say the identical version of the application is available at 3 places:
https://some1.app.com
https://some2.app.com
https://some3.app.com
The intended system should check which instance is available and run a test (that is not already run) on it. It should be able to run 3 tests on the 3 instances simultaneously in the Jenkins environment.
I have explored the Jenkins Matrix Configuration, but that appears to run all tests on all possible combinations in the matrix. My intention is to divide and load balance the tests, not run on all combinations. Any ideas on how this can be done?
I am using JUnit4 with Ant for running the tests on Jenkins.
One solution would be a Matrix Project Plugin. You could configure your url as parameter a bit like in here: Building a matrix project
Long story short,
I was wondering if anyone ever felt the need for (and knows of any implementation of) the possibility of "instantiating" (OO terminology) a parametrized build.
What I mean is treating a parametrized build as a template, from which many "instances" can be generated.
Each instance is supposed to define a different combination of values for the parameters.
The final goal is twofold:
DRY (which is given simply by the parametrized build concept)
having separate build histories / test reports for each instance (otherwise it would be a mess)
the instances would be schedulable directly in jenkins UI (while a parametrized build is not)
The template would then be used only for:
manual builds
changing the config for all of the instances at once
Now, time for some context, as I may be missing something in my overall approach.
You are welcome to point me in the right direction :)
I have a maven project with a suite of selenium tests that I want jenkins to run.
The suite is parametrized: browser, OS, test environment.
So, I can run it e.g. with mvn test -Dbrowser=chrome -Dplatform=win [..].
I want a separate test report for each combination of my parameters.
As a newbie, my first solution was "Copy existing job".
Quick and dirty. But effective.
As you will know, problems arise when you need to make a change to the configuration of the job, and you want to keep in sync all of these copy&pasted jobs.
Then I found the parametrized build feature.
It's very cool (code reuse/maintainability++), but the test report and the build history is shared among all of the actual builds, therefore I can not rely on them for a tidy reporting like "this test is always failing on IE; but it isn't on chrome", and so on.
Thank you very much in advance
I think what you are describing is the matrix project
There are also selenium plugins, I put one together to work with matrix jobs https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Axis+Plugin
One lack I can see: you can't build a single combination, as the build btn is present only at the "top level".
Have you tried the Matrix Combination plugin
https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Combinations+Plugin
I'm looking for a jenkins plugin that would allow selection of file(s)/folder(s) for a parameterized build.
The purpose is to be able to select different tests to execute, each test being defined as a .xml file in a svn repo.
eg of repo structure:
tests/business/cars/buy.xml
tests/business/cars/sell.xml
tests/system/core/stuff.xml
I'm not sure if these quite get at what you're looking for, but I found the following two possibilities:
Test In Progress which, based on their GitHub information, should allow you to run tests based on pattern matching and view the progress as they run.
RQM is an IBM tool made to allow you to run specific test cases in a test plan that have a custom property that you configured and provide at test execution time.
edit:
I just found Multi Module Test Publisher, which may be a little closer to what you're looking for. It acts as a replacement for the normal junit plugin and allows you to group junit tests into suites and view statistics on each separately.