i'm running a nightly Cucumber test automation suite through Jenkins, but i'm interested in archiving the reports somewhere and using them for future metrics and auditing activities.
What are my options for archiving Cucumber Reports somewhere?
Should i be looking at tools such as Splunk to consume the reports and build a bespoke Splunk report based on the data? Are there any other tools out there which would better suit this task? Are there any standard tools used for the archiving of Test Automation Reports, which also allows the archived reports to be accessed and displayed.
I'm trying to get an idea of what's the best tool or approach for this task.
You could post the reports to a data store of your choosing as well as Jenkins. That way you could create a historical store with a scheduled task to compress the reports say weekly or monthly.
This way would remove adding complications into Jenkins itself.
If you are in AWS say, you could look at s3 buckets for example.
Related
The current situation: We're manually putting together a weekly report using the information available to us in TFS. It's a mind-numbing process that takes each scrum master several hours a week to piece together. The reports include information on sprint progress, blocked stories/features, workload per team, etc.
My question: Is there a way of automating this process to run on a weekly basis and export the results to an Excel file? TFS data must be housed in a database somewhere, but I'm not sure how restricted access to this database is.
After you create and publish reports, you can use Report Manager to view, organize, and configure those reports. This could be achieved by using Report Manager. Take a look at this tutorial: View, organize, and configure reports using Report Manager
After you create and publish reports in SQL Server Report Designer,
you can use SQL Server Reporting Services' Report Manager to view,
organize, and configure those reports. By using Report Manager, you
can group related reports in folders, adjust parameters and data
sources, and schedule automated reports.
You can also configure different methods by which your reports are
saved. For example, you can save a copy of a report (sometimes
referred to as a snapshot) as report history. You can also export and
save reports and have reports copied automatically to a file share.
If you want to import report to Excel, you could write a small PowerShell script using Rest API to query info such as related work items, then you add a scheduled job that'll execute that script .
We have TFS 2017 installed on our server and JMeter performance tests which i want to run through non-gui mode on TFS build agent.
I can't find way how to publish jmeter results to TFS?
There is no approved plugin for JMeter (at least none I could find) that incorporate your results into VSTS test results. You would need to dump your JMeter results locally, add a task that captures the output and puts it with the $(build.artifactstagingdirectory) path. You will then need to make sure that your final task is a Publish Build Artifacts. This will ensure that your JMeter results will get published as artifacts as part of the build.
Now the fun part, you obviously want to view the results so you are left with two options. The first is to create a web app that contacts VSTS via the API, pulls the JMeter artifact and displays the results as desired. There may actually be
Last option is to create a hub page for VSTS. This could query the build, get the JMeter artifacts and render them as desired.
These are the best options I could find for you at present. You can always request such a feature in the VSTS User Voice Forum.
There is not any build-in tool to integrate the JMeter with TFS for now.
You can have a try for Cloud-based Apache JMeter Load Test task. However it's cloud based, and need Visual Studio Team Services account.
There are detailed reports and analysis tool when using could based jmeter test such as this tutorial.
Please reference this article for detail steps : Running Apache JMeter based load tests in the cloud – how to
In your case, the simplest way is as virusstorm suggested using custom extensions to catch your test results and publish test results. If you are not familiar with this area, there are detail tutorials here. Moreover you could also ask help of some ALM company and team.
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
The builds in my project have started to take more time recently. I need to optimize them, but I don't want to do blind optimizations. I need how many time is consumed by respective steps of the build process (fetching, preparing, build script, publishing etc). It there any jenkins plugin that enables cuch profiling?
Not really a profiling tool, but you should start with the Timestamp plugin
https://wiki.jenkins-ci.org/display/JENKINS/Timestamper
This will put a timestamp for every line in your console log. You can then easily see how much time was spent on each particular section (checkout, building, publishing) in the log
If your build calls out to standard build tools like Ant, Maven etc., you may want to give YouMonitor a try: https://www.yourkit.com/youmonitor/features/. Be aware this does not show you any details of the Jenkins pipeline or job, but summarizes the external build tools quite nicely by their goals/targets etc.
What kind of agile tools are you using for Erlang development? What continuous integration (CI) server are you using to build Erlang code? The only reference I got was from Quora question How do I integrate Erlang unit tests in Jenkins (Hudson)?.
I am also interested in the nifty details of setting them up and making talk to each other.
As a company using Erlang actively, Klarna (www.klarna.com) use Jenkins (formerly Hudson) for daily regression test on nearly every dev commit. It's an org with about 80 people total in rnd and we use distribute mode of Jenkins which allows us to have more than 10 build slaves mastered by only one Jenkins server. Basically we have a code base with Eralng code which is version controlled by tools like svn or git. All these testcases are under common test framework and all works well under Jenkins.
Previously, we tried Cruise Control and gave it up since Jenkins does much better.
As Lukas mentioned, you probably will need a tool to gen xml files sine common test doesn't export them directly. Haven't really tried that module though, we do have an implementation of common test event handler to do the job, but it was abandoned due to performance, we do have a a critical requirement on test time. right now, we use a own made script to export xml from common test log directly.
There are a lot more you could do with Erlang and Jenkins, like code coverage analyze if you compile properly and export formatted xml to Cobertour plugin, gui test with selenium etc.
For setting up Jenkins, I think Jenkins home page has a good introduction.
Regarding agile tools, I guess it's really hard to define what a agile tool. Also what I believe is it's very much depend on the size of you org. You will probably need a good process view tool (team level or depart level), a good ticket tracking tool, code review tool, communication tool. There are bunch of them implemented under open source. According to our exp, none of them seems to be able to work seamlessly with Jenkins which means you will need to select and tweak by your own requirement. BUT that's the beauty of open source isn't it :)?
If you want to do it using Jenkins, I have written a common test hook which generates JUnit XML output for your tests which Jenkins can use to produce test statistics.
https://github.com/garazdawi/cth_tools/blob/master/src/cth_junit.erl
We use Jenkins for our Python code, so I think you may use Jenkins with Erlang code.
We use buildbot with our own recipes to hook unit tests.