I created a test method that uses "Assert.Inconclusive(skipMessage)" function. On TFS2015, the executing outcome is 'done', but on TFS2017 the executing outcome is 'not executed'.
Could you please help me to make the outcome is 'done' when using "Assert.Inconclusive(skipMessage)" on TFS2017?
In both TFS 2015 and TFS 2017, the Skipper test methods are both shown like 'not executed' not 'Done'.
This settings in TFS can't be changed. But if you want to change/edit the outcome of a specify test result, you could use TFS REST API to achieve this.
HTTP Method: PATCH
URL: http://SERVERNAME:8080/tfs/DefaultCollection/teamProject/_apis/test/runs/testrunId/results?api-version=3.0-preview
Json object:
{
"id": 100000, --this is test result id
"outcome": "None"
}
You could change the outcome from NotExecuted to None, Passed, Failed, Inconclusive, Timeout, Aborted, Blocked, Warning, Error, NotApplicable, Paused or InProgress.
Related
Using TFS 2018 update 3, a step was defined to be executed only on scheduled builds., the custom condition was defined:
eq(variables['Build.Reason'], 'Schedule')
A build was queued, Despite the fact that the variable Build.Reason = 'Manual'. the step was executed, then the condition was changed to :
and(succeeded(), eq(variables['Build.Reason'], 'Schedule'))
The result was the same: the step is executed even in gated check-in and or manual builds.
After execution automated tests in VStest task, vstest.exe generate trx result file. This trx contains only 3 types of outcome: Passed, Failed and Skipped. This types of outcome publish into TFS test plan.
Is there any possibility to add "Not applicable" or "Blocked" outcome into trx and update test cases in TFS test plan, based on this outcome?
I believe custom logger should be used, but what type of outcome can I use for "Not applicable" or "Blocked"?
<Counters total="1" executed="0" passed="0" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
There is 9 possible test outcomes for UnitTestOutcome
https://learn.microsoft.com/en-us/dotnet/api/microsoft.visualstudio.testtools.unittesting.unittestoutcome?view=mstest-net-1.2.0
But actually vstest.exe generate and put into trx only 3 of this types.
I have a TFS 2017 (version 15.105.25910.0) build which also runs tests, but one test is taking a very long time and the whole build is cancelled due to a timeout set in the 'general' tab of the build edit page. TFS log is included below. How can I check which test is faulty?
Notice the time difference between the first and second log. I assume a faulty test is being run after ReportAnalyzer_Blabla_SomethingTest, but with over 1k tests it's hard to guess which it is.
2017-08-30T11:30:09.7614471Z Passed ReportAnalyzer_Blabla_SomethingTest
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.FindMatchingFiles(rootFolder = D:\TfsBuildAgents\RmsBuild\_work\8\s\TestResults, matchPattern = *.trx, includeFiles = True, includeFolders = False
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.GetMatchingItems(includePatterns.Count = 1, excludePatterns.Count = 0, includeFiles = True, includeFolders = False
2017-08-30T11:53:52.1581687Z ##[debug]FindFiles.FindMatchingFiles - Found 0 matches
2017-08-30T11:53:52.1581687Z ##[debug]Processed: ##vso[task.logissue type=warning;code=002003;]
2017-08-30T11:53:52.1581687Z
2017-08-30T11:53:52.1581687Z
2017-08-30T11:53:52.1737949Z ##[warning]No results found to publish.
2017-08-30T11:53:52.1737949Z ##[debug]Processed: ##vso[task.logissue type=warning]No results found to publish.
2017-08-30T11:53:52.2050485Z ##[error]The operation was canceled.
2017-08-30T11:53:52.2050485Z ##[debug]System.OperationCanceledException: The operation was canceled.
Normally the faulty test should be the first test after ReportAnalyzer_Blabla_SomethingTest. But as you said, it with over 1k tests, according to the log you posted, if you didn't split the tests we can not exactly identify which test is the faulty one. In this case, I'm afraid that you have to debug that one by one.
So, yo can try to split the tests then debug them accordingly.
You can also try to check if there are any other detailed logs to track that.
See Review continuous test results after a build for more information.
I've found a messy workaround which helped me find the failing test. In all test classes (the messy part) I've added a code which appends the currently running unit test name to a file - the last entry was what I was interested in.
[ClassInitialize]
public static void ClassInitialzie(TestContext testContext)
{
// This is just an example!
File.AppendAllText("testRunLog.txt", testContext.TestName + Environment.NewLine);
}
The closes thing to "run the code before each test in the whole test project" seems to be the ClassInitialize attribute.
https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.classinitializeattribute.aspx
We have an automation test framework written in Java.
At present, I can post a new test run to VSTS.
I can then post a new test result to the test run, eg:
{'testCaseTitle':'vstsPostTest',
'automatedTestName':'myAutomatedTestName',
'outcome':'Passed',
'errorMessage':'successfully completed vstsPostTest'}
However, when viewing the test result entry in VSTS, the testCaseTitle always seems to be overridden with value: 'myTestCaseTitle'.
Thinking this may be because I haven't associated the posted result with a testcase that resides in VSTS, I have added the testCase id parameter, and specified the ID of a random existing testCase in VSTS, eg,
{'testCaseTitle':'is title now necessary?',
'testCase'{'id':283},
'automatedTestName':'myAutomatedTestName',
'outcome':'Passed',
'errorMessage':'successfully completed vstsPostTest'}
My posted result still has the generic title 'myTestCaseTitle' - and I have seen nothing yet in VSTS which illustrates a link between the posted result and the specified testcase which resides in VSTS.
What else is necessary to:
a) Ensure that a posted result uses the specified testCaseTitle value (or receives it from the ID of the associated testCase)
b) See that either a testCase or story in VSTS is flagged as failing/passed, based on the latest testrun results. (ie, identify where stories are failing as a result of automated regression testing)
Many thanks
Try to specify test point ID:
POST: https://[team project url]/_apis/test/runs/[test run id]/results?api-version=3.0-preview
[
{
"testPoint":{
"id":XX
},
"priority": 1,
"outcome": "Passed"
}
]
You can get test point id by using Test points REST API
With thanks to Starains help. As commented: I ended up carefully following this MS post: social.msdn.microsoft.com/Forums/sqlserver/en-US/… To others reading: after a VSTS test plan, suite, case is created: post a new test run (specify plan ID), then post a new test result (specify testPoint ID, outcome, etc) then get the testResult ID and update that test result to state: Completed. This is the minimum to affect the outcome of a test case in VSTS.
I'm running NUnit tests using Jenkins (and the XUnit plugin), and Email-Ext to send out build result summaries.
I'd like to be able to email out something like "3 new test failures: [Names of tests that failed]." I can't work out how to get which tests changed from a previous run.
So far I have:
${TEST_COUNTS,var="total"} tests: ${TEST_COUNTS,var="pass"} pass,
${TEST_COUNTS,var="fail"} fail, ${TEST_COUNTS,var="skip"} skipped
giving
1914 tests: 1903 pass, 10 fail, 1 skipped
and ${FAILED_TESTS} giving the details of all tests failing - but I can't work out how to get just the changes from the previous run.
Viewing the job in Jenkins gives the information I need, so it ought to be possible.
Try this one:
============================
TESTS
There are ${TEST_COUNTS, var="total"} total tests of which ${TEST_COUNTS, var="fail"} test(s) failed.
$FAILED_TESTS
Try this:
CHANGES (All changes since first failure)
${CHANGES_SINCE_LAST_SUCCESS, reverse=true}