How to associate a test result posted to VSTS with a testcase residing in VSTS? - azure-devops-rest-api

We have an automation test framework written in Java.
At present, I can post a new test run to VSTS.
I can then post a new test result to the test run, eg:
{'testCaseTitle':'vstsPostTest',
'automatedTestName':'myAutomatedTestName',
'outcome':'Passed',
'errorMessage':'successfully completed vstsPostTest'}
However, when viewing the test result entry in VSTS, the testCaseTitle always seems to be overridden with value: 'myTestCaseTitle'.
Thinking this may be because I haven't associated the posted result with a testcase that resides in VSTS, I have added the testCase id parameter, and specified the ID of a random existing testCase in VSTS, eg,
{'testCaseTitle':'is title now necessary?',
'testCase'{'id':283},
'automatedTestName':'myAutomatedTestName',
'outcome':'Passed',
'errorMessage':'successfully completed vstsPostTest'}
My posted result still has the generic title 'myTestCaseTitle' - and I have seen nothing yet in VSTS which illustrates a link between the posted result and the specified testcase which resides in VSTS.
What else is necessary to:
a) Ensure that a posted result uses the specified testCaseTitle value (or receives it from the ID of the associated testCase)
b) See that either a testCase or story in VSTS is flagged as failing/passed, based on the latest testrun results. (ie, identify where stories are failing as a result of automated regression testing)
Many thanks

Try to specify test point ID:
POST: https://[team project url]/_apis/test/runs/[test run id]/results?api-version=3.0-preview
[
{
"testPoint":{
"id":XX
},
"priority": 1,
"outcome": "Passed"
}
]
You can get test point id by using Test points REST API

With thanks to Starains help. As commented: I ended up carefully following this MS post: social.msdn.microsoft.com/Forums/sqlserver/en-US/… To others reading: after a VSTS test plan, suite, case is created: post a new test run (specify plan ID), then post a new test result (specify testPoint ID, outcome, etc) then get the testResult ID and update that test result to state: Completed. This is the minimum to affect the outcome of a test case in VSTS.

Related

TFS 2017 How do I know which test is being run (before it finishes)?

I have a TFS 2017 (version 15.105.25910.0) build which also runs tests, but one test is taking a very long time and the whole build is cancelled due to a timeout set in the 'general' tab of the build edit page. TFS log is included below. How can I check which test is faulty?
Notice the time difference between the first and second log. I assume a faulty test is being run after ReportAnalyzer_Blabla_SomethingTest, but with over 1k tests it's hard to guess which it is.
2017-08-30T11:30:09.7614471Z Passed ReportAnalyzer_Blabla_SomethingTest
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.FindMatchingFiles(rootFolder = D:\TfsBuildAgents\RmsBuild\_work\8\s\TestResults, matchPattern = *.trx, includeFiles = True, includeFolders = False
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.GetMatchingItems(includePatterns.Count = 1, excludePatterns.Count = 0, includeFiles = True, includeFolders = False
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.FindMatchingFiles - Found 0 matches
2017-08-30T11:53:52.1581687Z ##[debug]Processed: ##vso[task.logissue type=warning;code=002003;]
2017-08-30T11:53:52.1581687Z
2017-08-30T11:53:52.1581687Z
2017-08-30T11:53:52.1737949Z ##[warning]No results found to publish.
2017-08-30T11:53:52.1737949Z ##[debug]Processed: ##vso[task.logissue type=warning]No results found to publish.
2017-08-30T11:53:52.2050485Z ##[error]The operation was canceled.
2017-08-30T11:53:52.2050485Z ##[debug]System.OperationCanceledException: The operation was canceled.
Normally the faulty test should be the first test after ReportAnalyzer_Blabla_SomethingTest. But as you said, it with over 1k tests, according to the log you posted, if you didn't split the tests we can not exactly identify which test is the faulty one. In this case, I'm afraid that you have to debug that one by one.
So, yo can try to split the tests then debug them accordingly.
You can also try to check if there are any other detailed logs to track that.
See Review continuous test results after a build for more information.
I've found a messy workaround which helped me find the failing test. In all test classes (the messy part) I've added a code which appends the currently running unit test name to a file - the last entry was what I was interested in.
[ClassInitialize]
public static void ClassInitialzie(TestContext testContext)
{
// This is just an example!
File.AppendAllText("testRunLog.txt", testContext.TestName + Environment.NewLine);
}
The closes thing to "run the code before each test in the whole test project" seems to be the ClassInitialize attribute.
https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.classinitializeattribute.aspx

inconsistency in outcome status when execute test method between TFS2017 & TFS2015

I created a test method that uses "Assert.Inconclusive(skipMessage)" function. On TFS2015, the executing outcome is 'done', but on TFS2017 the executing outcome is 'not executed'.
Could you please help me to make the outcome is 'done' when using "Assert.Inconclusive(skipMessage)" on TFS2017?
In both TFS 2015 and TFS 2017, the Skipper test methods are both shown like 'not executed' not 'Done'.
This settings in TFS can't be changed. But if you want to change/edit the outcome of a specify test result, you could use TFS REST API to achieve this.
HTTP Method: PATCH
URL: http://SERVERNAME:8080/tfs/DefaultCollection/teamProject/_apis/test/runs/testrunId/results?api-version=3.0-preview
Json object:
{
"id": 100000, --this is test result id
"outcome": "None"
}
You could change the outcome from NotExecuted to None, Passed, Failed, Inconclusive, Timeout, Aborted, Blocked, Warning, Error, NotApplicable, Paused or InProgress.

Asana - How to get all tasks that are completed in a team

I am trying to develop summary data of length of time from task creation to task completion for all tasks in a certain team. I want to look at how lang it is taking from start to finish on tasks (on average) for any given month. My team has 30 - 40 projects and we move tasks from project to project until it is completed.
I have tried the following URL on the explorer page in their documentation:
GET
/tasks?opt_fields=completed_at,completed&completed_since=2012-02-22T02%3A06%3A58.147Z&limit=10&workspace=[myworkspaceid]&project=[myprojectid]
The response I get is:
{ "errors": [
{
"message": "Must specify exactly one of project, tag, or assignee + workspace",
"help": "For more information on API status codes and how to handle them, read the docs on errors: https://asana.com/developers/documentation/getting-started/errors"
} ] }
Since I am specifying a project ID why am I getting this error?
************************ EDIT ************************************
OK...thanks for that. I am usding the following code and using only a workspace did indeed fix my issue. When I use the following code using asana gem in ruby I get results....363 results.
allTasks = client.tasks.find_all(assignee: nil, workspace: nil, project: 999999999999999999,completed_since: nil, modified_since: nil, per_page: 100, options: {})
allTasks.each do |task|
puts task.name
end
Let's say the developer of the Asana gem messed up and is ignoring my "per_page: 100". Isn't Asana limited to 100 or less results? Why am I getting 363? How Do I know that this really is all of them?
When you specify a project for the /tasks endpoint, you mustn't specify a workspace (the Asana API infers the workspace from the project). (Admittedly, the workspace could be optional, so I've filed this as a bug.)
Also, to directly answer your original question, we don't currently support directly querying for every task in a team. Fetching tasks in each project is the best approach.

Select test cases by SpecFlow tag in Microsoft Test Manager

Is it possible to select test cases by SpecFlow tag in Microsoft Test Manager? And if so, how?
Tags in SpecFlow get translated into TestCategory attributes in the generated code. As far as I know (worked with MTM until a year ago) you can:
Execute tests with MSTest filtered on TestCategory by Categories
Execute tests with VSTest.Console filtered on TestCategory by TestCaseFilter
Import test cases into MTM filtered on TestCategory
With the last option, you can create with a bit of creativity a set of testplans with different tests sorted by test category. And I am afraid that is the best you can do without writing your own wrapper around MTM. The TestCategory information is available in TFS, but not exposed to the user in MTM.
EDIT
To clear things up after the comment.
Given this feature file:
#timtim
Feature: MSTest Tags Test
In order to get feedback at the right time
As a test automation specialist
I want to categorize my tests and run them accordingly
#Jabberwocky #manxome
Scenario: A test with tags
Given 'twas brillig
When gyre and gimble in the wabe
Then all mimsy were the borogoves
It gets generated to this code:
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("A test with tags")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "MSTest Tags Test")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("timtim")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("Jabberwocky")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("manxome")]
public virtual void ATestWithTags()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("A test with tags", new string[] {
"Jabberwocky",
"manxome"});
#line 8
this.ScenarioSetup(scenarioInfo);
#line 9
testRunner.Given("\'twas brillig", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Given ");
#line 10
testRunner.When("gyre and gimble in the wabe", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 11
testRunner.Then("all mimsy were the borogoves", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
The scenario becomes a (by MSTest.exe) executable testmethod with three TestCategories: timtim, Jabberwocky and manxome. These are the same testcategories as mentioned in the articles. Coded UI does have a Test Category property that can be used to order the tests, but this category boiles down to using the same TestCategory attribute.

TFS 2010 Issue with tracking Changesets in Builds that are a result of Gated Checkins

In order to retrieve the information which Changeset was included in which Build, we use "Label Sidekick" of Team Foundation Sidekicks, where we place the Label of the Build & expect to find the newly built Changeset.
Our development process in TFS 2010 is making use of 'Gated' checkins, so we are faced with the situation that the latest checkins are not presented in Sidekicks (we actually receive the changeset of the previous build). This is explainable, since at the time the labeling takes place, the latest changes have not yet been committed.
The BuildLog does report the associated Changeset correctly.
I 've made several experiments in our Build Process Template but can't seem to get what we need.
Placing, for example, the Labeling activity out of the "Run On Agent" scope, lead me to a build that fails at the very start with an "Object reference not set to an instance of an object." (I suppose this is related with fact I had to widen the scope for 'Label' & 'Workspace' variables to get the second part running).
The 'before' state of the build process template for this attempt is here (this works), the 'after' state ("Object ref not set..") is here.
So, to summarize, two different types of input could help me out:
How should I change our build process template so that the labeling happens after the Gated checkins have been committed? (-- This would rationalize the display in Sidekicks)
or
How can I programmatically retrieve the associated Changeset of each Build? (-- This would enable me to write a small app that could obsolete the Sidekicks angle)
You can use the TFS API to get this done.
public static void GetBuild()
{
var tfs = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(new Uri("http://tfsdevlonuk:8080/tfs/gazprom.mt"), new UICredentialsProvider());
tfs.EnsureAuthenticated();
var buildServer = tfs.GetService<IBuildServer>();
// Get Project Name
var versionControl = tfs.GetService<VersionControlServer>();
var teamProjects = versionControl.GetAllTeamProjects(true);
// Get Builds for a team project
var buildDetails = buildServer.QueryBuilds(teamProjects[0].Name);
// For each build
foreach (IBuildDetail buildDetail in buildDetails)
{
// Get the build details
var buildInfor = buildDetail.Information;
// More build infor like shelveset, etc
Debug.Write(buildDetail.LabelName + buildDetail.ShelvesetName);
}
The above code will help you get the build details programatically. I have some blog posts on how to connect to tfs programmatically and use the tfs api. http://geekswithblogs.net/TarunArora/archive/2011/06/18/tfs-2010-sdk-connecting-to-tfs-2010-programmaticallyndashpart-1.aspx

Resources