Is it possible to select test cases by SpecFlow tag in Microsoft Test Manager? And if so, how?
Tags in SpecFlow get translated into TestCategory attributes in the generated code. As far as I know (worked with MTM until a year ago) you can:
Execute tests with MSTest filtered on TestCategory by Categories
Execute tests with VSTest.Console filtered on TestCategory by TestCaseFilter
Import test cases into MTM filtered on TestCategory
With the last option, you can create with a bit of creativity a set of testplans with different tests sorted by test category. And I am afraid that is the best you can do without writing your own wrapper around MTM. The TestCategory information is available in TFS, but not exposed to the user in MTM.
EDIT
To clear things up after the comment.
Given this feature file:
#timtim
Feature: MSTest Tags Test
In order to get feedback at the right time
As a test automation specialist
I want to categorize my tests and run them accordingly
#Jabberwocky #manxome
Scenario: A test with tags
Given 'twas brillig
When gyre and gimble in the wabe
Then all mimsy were the borogoves
It gets generated to this code:
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("A test with tags")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "MSTest Tags Test")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("timtim")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("Jabberwocky")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("manxome")]
public virtual void ATestWithTags()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("A test with tags", new string[] {
"Jabberwocky",
"manxome"});
#line 8
this.ScenarioSetup(scenarioInfo);
#line 9
testRunner.Given("\'twas brillig", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Given ");
#line 10
testRunner.When("gyre and gimble in the wabe", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 11
testRunner.Then("all mimsy were the borogoves", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
The scenario becomes a (by MSTest.exe) executable testmethod with three TestCategories: timtim, Jabberwocky and manxome. These are the same testcategories as mentioned in the articles. Coded UI does have a Test Category property that can be used to order the tests, but this category boiles down to using the same TestCategory attribute.
Related
After execution automated tests in VStest task, vstest.exe generate trx result file. This trx contains only 3 types of outcome: Passed, Failed and Skipped. This types of outcome publish into TFS test plan.
Is there any possibility to add "Not applicable" or "Blocked" outcome into trx and update test cases in TFS test plan, based on this outcome?
I believe custom logger should be used, but what type of outcome can I use for "Not applicable" or "Blocked"?
<Counters total="1" executed="0" passed="0" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
There is 9 possible test outcomes for UnitTestOutcome
https://learn.microsoft.com/en-us/dotnet/api/microsoft.visualstudio.testtools.unittesting.unittestoutcome?view=mstest-net-1.2.0
But actually vstest.exe generate and put into trx only 3 of this types.
We have an automation test framework written in Java.
At present, I can post a new test run to VSTS.
I can then post a new test result to the test run, eg:
{'testCaseTitle':'vstsPostTest',
'automatedTestName':'myAutomatedTestName',
'outcome':'Passed',
'errorMessage':'successfully completed vstsPostTest'}
However, when viewing the test result entry in VSTS, the testCaseTitle always seems to be overridden with value: 'myTestCaseTitle'.
Thinking this may be because I haven't associated the posted result with a testcase that resides in VSTS, I have added the testCase id parameter, and specified the ID of a random existing testCase in VSTS, eg,
{'testCaseTitle':'is title now necessary?',
'testCase'{'id':283},
'automatedTestName':'myAutomatedTestName',
'outcome':'Passed',
'errorMessage':'successfully completed vstsPostTest'}
My posted result still has the generic title 'myTestCaseTitle' - and I have seen nothing yet in VSTS which illustrates a link between the posted result and the specified testcase which resides in VSTS.
What else is necessary to:
a) Ensure that a posted result uses the specified testCaseTitle value (or receives it from the ID of the associated testCase)
b) See that either a testCase or story in VSTS is flagged as failing/passed, based on the latest testrun results. (ie, identify where stories are failing as a result of automated regression testing)
Many thanks
Try to specify test point ID:
POST: https://[team project url]/_apis/test/runs/[test run id]/results?api-version=3.0-preview
[
{
"testPoint":{
"id":XX
},
"priority": 1,
"outcome": "Passed"
}
]
You can get test point id by using Test points REST API
With thanks to Starains help. As commented: I ended up carefully following this MS post: social.msdn.microsoft.com/Forums/sqlserver/en-US/… To others reading: after a VSTS test plan, suite, case is created: post a new test run (specify plan ID), then post a new test result (specify testPoint ID, outcome, etc) then get the testResult ID and update that test result to state: Completed. This is the minimum to affect the outcome of a test case in VSTS.
Environment
TFS2013.Update 2 12.0.30324.0
C#
Visual Studio 2013
Visual Studio Test Runner
Gated Check-In
Problem
As shown here I decorated my TestMethod like
[TestCategory("Gated"), TestMethod()]
As shown here I filtered, like
TestCategory=Gated
When I build without any filter, the TestRun run successfully with all unit tests. But when I tried to filter over TestCaterory, no TestRun started at all.
I tried without success ...
... to put the filter in quotation:
"TestCategory=Gated"
... to separate the attributes:
[TestMethod]
[TestCategory("Gated")]
... a similar question has been posted here, but I can't revert back to using the old default.xaml.
... I found just one warning in the build log under "Run Visual Studio Test Runner for Test Sources": "Dependent Assembly Newtonsoft.Json, Version=4.5.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed of C:\Builds2\56\614\bin\External\Microsoft.TeamFoundation.TestManagement.Common.dll not found. Can be ignored if not a NUnit project."
-> I added the reference to Newtonsoft.Json in every single NUnit-Project in all the solutions, that were build during the gated check-in - also without success.
The problem was not the TestCategory, but another pretty simple one:
The unittest-project in which the “broken” testmethod resides was not part of ”projects to Build” of the Build Definition.
I am facing the issue of a test that has undefined step(s) not being flagged as a failed test.
In the Java code we use Selenium 2/WebDriver and tests are driven by Ant and run in a Continuous Integration environment.
For the following scenario:
#test1
Scenario: Run test with an undefined step
Given I am logged in to the application //working
And I view the test example //working
Then the tree panel exists in the layout //undefined step
The following is a snippet of what is seen in the console:
#test1
Scenario: Run test with an undefined step
Given I am logged in to the application
And I view the test example
Then the tree panel exists in the layout
1 scenario (1 undefined)
3 steps (1 undefined, 2 passed)
The ant target used to run the test:
ant test.cuke.firefox -Dwebtest.server="http://localhost:9944" -Dwebtest.cuke.options="--tags #test1"|wac
I read that using the --strict flag gets the tests to fail.
But I've no idea of where I need to mention the flag.
Is it in the build.xml file? If so, where exactly - as wherever I've tried hasn't helped.
Is it in the cucumber.yml file?
There are 2 such files:
i) \lib\cucumber.jruby\gems\cucumber-0.8.7
ii) \lib\cucumber.jruby\gems\gherkin-2.1.5-java
If not in these files, where else?
Could you please point to where and how the flag needs to be set?
I've tried looking up the help but nothing has helped (probably I'm looking in all the wrong places!)
Thanks!
You need to set the strict option:
http://cukes.info/api/cucumber/jvm/javadoc/cucumber/api/junit/Cucumber.Options.html#strict()
Edit: You can set this flag in the RunCukesTest like:
#RunWith(Cucumber.class)
#Cucumber.Options(
format = {"html:target/cucumber-html-report"},
strict = true)
public class RunCukesTest {
}
I've setup teamcity with my sln file and got the unit tests to show up with the CppUnit plugin that teamcity has. And I get test results in the TeamCity UI.
Now I'm trying to get trending reports to show up for my unit tests and code coverage.
As of code coverage, we're using vsinstr.exe and vsperfmon.exe which produces an XML file.
I'm not quite sure as of what steps I should be taking to make the trending reports and code coverage(not as important) to show up.
I've already seen this post, but the answer seems to require editing the build script, which I don't think would work for my case since I'm building through MSBuild and the .sln file, and the tests are being ran through that build.
So basically I'm trying to get the Statistics tab to show up, and I'm not sure where to begin.
Just add simple Powershell step into your build configuration. Something like this:
function TeamCity-SetBuildStatistic([string]$key, [string]$value) {
Write-Output "##teamcity[buildStatisticValue key='$key' value='$value']"
}
$outputFile = 'MetricsResults.xml'
$xml = [xml] (Get-Content $outputFile)
$metrics = $xml.CodeMetricsReport.Targets.Target[0].Modules.Module.Metrics
$metrics.Metric
| foreach { TeamCity-SetBuildStatistic "$($_.Name)" $_.Value.Replace(',', '') }
It uses XML output from FxCop Metrics. You have to update the script for your actual schema.