Can anyone advise how you can show the test category where "unspecified" is shown in the screenshot as that I can group test result output by category/tag.
I would like to be able to do this from both nunit and specflow tests.
There are some groups that can group tests, but aren’t existing in nunit and specflow tests, so you can’t group test with them (the result is Unspecified), you can group by Container, test run.
A related user voice that you can vote: VSTS Group Test Results by Namespace/Class, you also can submit a new user voice.
As #starain-MSFT wrote, it depends how you group your tests.
There is an option to group by Category and if the test case has no Category set, it is in the group Unspecified.
In NUnit you use the Category- Attribute (https://github.com/nunit/docs/wiki/Category-Attribute) which is then used from the NUnit VS Adapter to inform the test explorer (VSTest) about this.
For SpecFlow:
To get this category attribute generated on the test methods in the code-behind file, you have to use tags (see https://cucumber.io/docs/reference#gherkin in section "Tags")
Related
Using VS2015 the Test Explorer allows you to run a single scenario outline.
Now I need to do the same using NUnit3 console tool (I'm using NUnit as Unit Test Provider).
Currently I'm using the following command in order to run a test using console tool.
"C:\NUnit-3.0.1\bin\nunit3-console.exe" Path.Scripts.dll --test:Fully.Qualified.Name.TestAAAFeature.TestAAA --x86
I could run a single row of a specflow scenario outline example using the --testlist: option.
# list.txt
TestC112169Feature.TestCase112169("1","atomic",null)
# cmd
"C:\NUnit-3.0.1\bin\nunit3-console.exe" Path.Scripts.dll --testlist:"c:\list.txt" --x86
And that do the trick.
First and foresmost, I think you should rename your test cases to be more informative as a best practice.
Coming to your question, you should use filters which can be specified by using a where clause. For running a specific test case, you can use either method or name to filter down to one or more target test case(s).
Just append the following to your command and you should be good to go.
--where "name == 'TestCase11257'"
OR
--where "method == 'TestCase11257'"
or you can even combine multiple filters like
--where "name == 'TestCase11257' || method == 'TestCase11257'"
You can read more about filters here
I am trying to use Testrail as a test case management system and so,
integrating testrail with the Jenkins would be useful.
This is what I want to achieve:
Lets say I manually create three test cases in testrail with case ID's
C1, C2 and C3 and these test cases will have some unique automated test names such as A1, A2, and A3 (in more info, there will be a field in testrail with such a unique
information)
When I hit "Start Automated Tests" button and run a Jenkins job from testrail (considering I have already implemented UI script for testrail that has this button):
, I want to run a script/something that takes the case ID's of the selected test cases and annotate those IDs to the actual Java tests temporarily so that it can run those specific tests and post results back to the Testrail.
Approach I can think of:
When I hit "Start Automated Tests" button on Testrail, I can make a script to run to create an XML file that will include the desired selected test cases from Testrail. This XML will then be provided as a default input to the Jenkins job and it will run the test cases mentioned in the XML file. This XML will be temporary and will be replaced everytime the selection is made from the Testrail. However, how do you do it? I am a newbie to the Testrail and read its API and looks like API will be useful to post the results back to the Testrail. But, how do we achieve the mapping of the ID's?
Also, any advise on posting results back to the Testrail will be useful.
This isn't TestNG specific, but you can make a custom annotation in java. You can update a TestRail test in a test run through the api either by the test ID (using add_result), or both the case id and run id(using add_result_for_case). http://docs.gurock.com/testrail-api2/reference-results
The case id doesn't ever change, so you can just hard code these in your tests.
Here is what I'm using for this purpose:
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.METHOD)
public #interface TestData
{
int testId() default 0;
String[] tags() default "";
}
My test method then looks like this(Using Junit, but shouldn't be much different with other frameworks):
#Test
#TestData(
testId = 177,
tags = {"smoke", "authentication"}
)
public void testName()
{
//Do the test
}
I then use a JUnit specific way to get the test method name to use in my teardown method, but I'm sure there is a variety of ways to do that. Once you have the test method name here is how I read the annotation:
#After
public void baseTearDown() throws Exception
{
//Good place to record test results
Method testMethod = getClass().getMethod(testName);
if(testMethod.isAnnotationPresent(TestData.class))
{
TestData testData = testMethod.getAnnotation(TestData.class);
//Do something with testData.testId();
System.out.println("Test ID = " + testData.testId());
}
//other cleanups
}
This mkyong link gives some pretty basic examples of both creating an annotation and reading it with reflection. This is what I used to get started:
https://www.mkyong.com/java/java-custom-annotations-example/
If you're starting the test run in your code, then you can just keep track of the test run id and use it as needed. If not, my preference is to define and set some environment variables using Jenkins or other scripts that your code can read from so you don't have to deal with passing around files for some really basic key value pairs
In Fitnesse Commands: http://<host>:<port>/<suite path and test name>?responder=suite&startTest=TestTwo. I tried to execute. It is executing the test case which is passed in the url. If we pass the suite path and remove the test name, it is executing the whole suite. Is there any way we can run all tests coming after TestTwo?
No, you can run an entire suite or you can run an individual test. Perhaps you can break the suite into smaller sub-suites?
You can use firstTest parameters when calling the url. In your case, you can do:
http://<host>:<port>/<suite path>?responder=suite&firstTest=TestTwo
Note that, this only works based on alphabetical order of full path names of the test cases as mentioned in the fitnesse wiki here
firstTest: if present, only tests whose full path names are lexigraphically greater (later in alphabetical order) will be run. This is of questionable use since tests are not guaranteed to be run in any particular order.
Alternatively, you can tag certain test cases with tags, and execute them using parameter suiteFilter. You can find the relevant documentation in the same wiki page.
I am using specflow for writing my feature files .my feature files contains "#Tags"(like:#Authentication,#Login,#Permission,etc...) so i want to run all of them except #Authentication..
so can we use tag like:
~#Authentication so this will execute all test cases except test case containing #Authentication tag
As you have stated that you are running the tests from the command line using MSTest.exe then you should be able to run tests that are not in a category (according to the command line options like this:
"%VS100COMNTOOLS%..\IDE\MSTest.exe" /testcontainer:"Project.dll" /category:"!Authentication"
I am working on VSTS 2010 and using Microsoft Test Manager to manage our team's test cases. I just want to add a custom field for Test Categories (with values like BVT, FVT, Regression etc..) to the existing Test case template by following the steps here and here. But I am not able to select multiple values. I am able to create a drop down list with one of these values can be selected but not more than one. but since a test case can be part of more than one test category, how can I make this possible?
My steps:
In Process Editor, Work item types I Create a new field called Test Category with Ref Name as MyCustomField
In Rules, I selected AllowedValues as BVT, FVT, Regression.
After that, In Layout tab created a new control with name Test Categories
4.The field name of the control is set to MyCustomField
When I checked in MTM to create a new test case, the control is showing as drop down instead of one that was shown here
The control you are trying to use is strictly 1 to 1 (does not allow for multiple values), so there isn't really a simple answer to this question, but here are some options:
1- Create 3 custom fields BVT, FVT, and Regression each with a yes/no input.
2- Follow the steps here: Multi-value list control
3- Upgrade to VS2012 and use tagging.