We are using JIRA, SpecFlow, Jenkins in our project.
Can anybody tell us a tool that can be used as JIRA add in to organize, maintain, and track the testing process, progress, and results of execution the test scenarios in the BDD Feature files?
I want to record and track both the manual tests my team will run and all the automated tests generated through BDD Feature files in JIRA.
We used BehavePro which worked excellent. The business could create the user stories in Gherkin and we would export it to Visual Studio and automate the tests.
Besides that I have no experience in other tools/plugins.
Related
My managers want we to determine which tests might have to be run, based on coding changes that were made to the application we are testing.
But, it is hard to know which tests are actually needed to be re-verified as a result of a code change. What we have done is common to test the entire area where the code change occurred / or the entire proj, solution.
We were told this could be achieved by TFS build or MTM tools. Could someone share the details?
PM:We are running on TFS 2015 update4,VS2017.
There is a concept of Test Impact Analysis which helps in analysis of impact of development on existing tests. Using TIA, developers know exactly which tests need to be verified as a result of their code change.
The Test Impact Analysis (TIA) feature specifically enables this – TIA
is all about incremental validation by automatic test selection. For a
given code commit entering the pipeline TIA will select and run only
the relevant tests required to validate that commit. Thus, that test
run is going to complete faster, if there is a failure you will get to
know about it faster, and because it is all scoped by relevance,
analysis will be faster as well.
Test Impact Analysis for managed automated tests is available via a checkbox in the 2.* preview version of the VSTest task.
If enabled, only the relevant set of managed automated tests that need to be run to validate a given code change will run. Test Impact Analysis requires the latest version of Visual Studio, and is presently supported in CI for managed automated tests.
However this is only available with TFS2017 update1(need 2.* preview version of VSTS task). More details please refer this blog: Accelerated Continuous Testing with Test Impact Analysis
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
I manage to use bddlib(storyq) with combination of xunit.net using resharper runner. Than decided to try to specflow, since have read its advantages over storyq.
Now I do have also another requirement to integrate this all with tfs build 2010. Am bit lost there how the big picture gonna look like. I found some articles on how to make work xunit.net with tfsbuild 2010, however there is no single word on bdd lib integration with tfsbuild more specifically continuous integration (ci).
Anyone could have helped ? thanks.
If you can run it from the command-line you can simply customize your TFS Build workflow by tossing an InvokeProcess activity in there. An example of customizing the workflow can be seen here: http://www.ewaldhofman.nl/post/2010/04/28/Customize-Team-Build-2010-e28093-Part-3-Use-more-complex-arguments.aspx
What kind of agile tools are you using for Erlang development? What continuous integration (CI) server are you using to build Erlang code? The only reference I got was from Quora question How do I integrate Erlang unit tests in Jenkins (Hudson)?.
I am also interested in the nifty details of setting them up and making talk to each other.
As a company using Erlang actively, Klarna (www.klarna.com) use Jenkins (formerly Hudson) for daily regression test on nearly every dev commit. It's an org with about 80 people total in rnd and we use distribute mode of Jenkins which allows us to have more than 10 build slaves mastered by only one Jenkins server. Basically we have a code base with Eralng code which is version controlled by tools like svn or git. All these testcases are under common test framework and all works well under Jenkins.
Previously, we tried Cruise Control and gave it up since Jenkins does much better.
As Lukas mentioned, you probably will need a tool to gen xml files sine common test doesn't export them directly. Haven't really tried that module though, we do have an implementation of common test event handler to do the job, but it was abandoned due to performance, we do have a a critical requirement on test time. right now, we use a own made script to export xml from common test log directly.
There are a lot more you could do with Erlang and Jenkins, like code coverage analyze if you compile properly and export formatted xml to Cobertour plugin, gui test with selenium etc.
For setting up Jenkins, I think Jenkins home page has a good introduction.
Regarding agile tools, I guess it's really hard to define what a agile tool. Also what I believe is it's very much depend on the size of you org. You will probably need a good process view tool (team level or depart level), a good ticket tracking tool, code review tool, communication tool. There are bunch of them implemented under open source. According to our exp, none of them seems to be able to work seamlessly with Jenkins which means you will need to select and tweak by your own requirement. BUT that's the beauty of open source isn't it :)?
If you want to do it using Jenkins, I have written a common test hook which generates JUnit XML output for your tests which Jenkins can use to produce test statistics.
https://github.com/garazdawi/cth_tools/blob/master/src/cth_junit.erl
We use Jenkins for our Python code, so I think you may use Jenkins with Erlang code.
We use buildbot with our own recipes to hook unit tests.
Some background: I am not a tfs guy and I dont know much about build scripts etc.
1 - Is there a way to run tests for every check-in TFS? What I'm dreaming is, if any of the tests is failing then build server rejects the changeset. Is it possible with TFS or should it be some other tool like Hudson, Cruise Control etc? What are the other powerful tools?
2 - Does using such a tool make it possible to run only portion of tests, not all of them (ie only unit tests, not integration tests)?
I am not interested in technical details like how it can be done technically, as it is our tfs team's job. Rather I am after some high level info about the possibilities?
In TFS you have what's called check-in policies. With those in place you could forbid checking-in something without all of the unit tests passing. You could even enforce FxCop rules, etc... but that would be cruel to your developers.
If you already have the Continuous Integration build set up, then change the trigger to Gated Check In, this will do exactly what you want. When a developer tries to commit, TFS will start the build, if the build fails then the check in will be aborted and instead TFS will create a shelveset of the changes.
As for running a portion of the tests, you would probably need to create a test list in your VSMDI that defines the tests you want to run and then configure the build to use that list.