I am using Serenity BDD for my project. I have divided feature files into different functional areas and they are in turn represented as separate Jenkins jobs. The problem I am facing is the Jenkins jobs generate separate report and the client requires a single report. I was wondering if somebody has tried Jenkins Copy artifact + Serenity aggregate to consolidate reports into a single serenity report. It would be great if you can share may be another (alternate) simpler report to do it. Cheers
Related
So I have 3 Appium/Selenium Junit classes, each for a different platform (iOS, Android, web). I'd like to create 3 Jenkins jobs and then run some or all of them in parallel based on parameters that I'll input somehow (file, command, etc.).
Example: I want to run the iOS and Android tests with specific devices (UDIDs). So I'd somehow input this information (which jobs to run and with which parameters=devices), and the 2 jobs will run concurrently with this input.
I'm a Jenkins beginner, and I have tried searching and found many Jenkins plugins that seem like they can help (i.e. Parameterized Trigger Plugin), but I didn't understand how to use these correctly (how to configure the jobs correctly with these plugins). Any help would be appreciated (as "dumbed-down" as possible...)
So it's like this. You can run tests on Jenkins in parallel or in multiple builds. I'm using maven, multiple Jenkins nodes, selenium-grid and bunch of other tools. You can call single maven command so it can run parallel tests at once, for eg. only mobile (Android/iOS), and web in separate build to run tests in parallel run for multiple browsers eg. (Chrome & Firefox...).
There is a several ways to achieve this, but there is lot o preparation and setup:
via java/testNg (xml test-suites),
set up selenium-grid for multiple instances,
Here is great starter article for this: https://www.swtestacademy.com/selenium-parallel-tests-grid-testng/
https://learn.techbeacon.com/units/how-use-testng-parallel-test-execution
(if using for mobile testing) setup Appium to run several instances, here is also nice article to start with: https://appium.io/docs/en/advanced-concepts/parallel-tests/
have multiple Jenkins nodes, here is something about that, read entire article: https://wiki.jenkins.io/display/JENKINS/Distributed+builds
But main question was about Jenkins, You can achieve this by creating simple Maven build, create Your job, connect it with Your code on git, enter maven command to call Your code, and You should be set to go, again one more article: https://developers.perfectomobile.com/pages/viewpage.action?pageId=21435209.
Good parallelism is hard to achieve, but when You do its is going to make You worth all time invested.
Hope It will help You!
i'm running a nightly Cucumber test automation suite through Jenkins, but i'm interested in archiving the reports somewhere and using them for future metrics and auditing activities.
What are my options for archiving Cucumber Reports somewhere?
Should i be looking at tools such as Splunk to consume the reports and build a bespoke Splunk report based on the data? Are there any other tools out there which would better suit this task? Are there any standard tools used for the archiving of Test Automation Reports, which also allows the archived reports to be accessed and displayed.
I'm trying to get an idea of what's the best tool or approach for this task.
You could post the reports to a data store of your choosing as well as Jenkins. That way you could create a historical store with a scheduled task to compress the reports say weekly or monthly.
This way would remove adding complications into Jenkins itself.
If you are in AWS say, you could look at s3 buckets for example.
I am currently working on a project which consists of several applications which in our infrastructure are tested with SonarQube and JUnit. For the moment I receive test reports for each subproject (application) separately.
My question is : what tools or solutions do you use to generate a report across multiple independent Maven projects to aggregate the test results generated by JUnit / Maven / Surefire ?
For the moment I came across SonarQube Governance plugin, however I would like to know the alternatives.
The Governance plugin ($) is the only solution to aggregate projects. However, if you're talking about aggregating multiple test reports within a single project, that functionality is native in SonarQube 6.2. Further, for projects with sub-projects, the aggregation of sub-project data should happen naturally in the parent project.
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
I am using the combination of Jenkins, python, unittest, nosetests to run test suites. I publish the results in nosetests.xml to Jenkins using Junit plugin.
My question: How can I run the same test suites with different browsers (Chrome, FF, IE,…,etc) and publish all the results in the same Jenkins job and still be able to distinguish each test result with respect to browser it used?
I am thinking about running the test suite many time, each with different browser and rename the tests in each nosetest report, nosetests.xml, before publishing to Jenkins but I don’t think that it is an elegant solution.
Thanks for your help
This is what multi-configuration jobs (or matrix jobs) are designed for in Jenkins.
You specify your job configuration once, but add one or more variables that should change each time, building a matrix of combinations (in your case, the matrix has one dimension: browser).
Jenkins then runs one main build with multiple sub-builds — one for each combination in the matrix. You can then clearly see the results for each combination.
This requires that your test job can be parameterised, i.e. you can choose at runtime which browser should be run, rather than running all tests together in a single job.
The Jenkins wiki has minimal documentation on this feature, but there are a few good blog posts (and Stack Overflow questions) out there on how to set it up.