I am using a TFS Release to execute .loadtests written in Visual Studio.
I can execute the tests fine using MSTest, this produces the expected .trx file.
What I want is to be able to see the test results within TFS, I have tried using the 'Publish Test Results' task but have experienced issues.
The issue encountered was an Object not set to an instance of an object
Upon analysing the powershell scripts that 'Publish Test Results' task executes, from what I can see, the only results that are read in the TrxResultReader are from 'WebTest' and 'UnitTest'.
Has anyone managed to publish the results of a loadtest within TFS?
For TFS Publish Test Results task. This task supports popular test result formats including JUnit, NUnit 2, NUnit 3, Visual Studio Test (TRX), and xUnit 2. If you use the built-in tasks such as Visual Studio Test to run tests, results are automatically published and you do not need a separate publish test results task.
Based on the powershell script behind the build task, it use TestRunner. Since Vstest cannot run load test, it cannot publish load test result too. Take a look at below similar question:
Publish Test Results task fail when trying to publish a loadtest
It is not possible to use mstest to publish test results to build
made by a vNext. mstest can not identify the correct buildnummer /
buildid. Our solution will probably be that we store the *.trx file a
artifact. An other solution is to use a xaml build to perform
performance tests.
Related
Question:
How do I run NUnit tests on TFS?
Attempts:
I've searched the web and am still struggling to figure out the exact steps required to accomplish (to what I believe is) a trivial task.
I added the NUnit3TestAdapter package to my test project.
However, my tests are still not being discovered.
TFS displays the following message:
No test runs are available for this build. Enable automated tests in
your build pipeline by running your test framework of choice, such as
JUnit, Visual Studio Test, or xUnit. If you choose to run tests using
a custom task or runner, you can publish results using the Publish
Test Results task.
Expectation:
I'm surprised that NUnit is not an option that I can just select from.Hence, it's the standard unit test framework for .Net.
How do I run NUnit tests on TFS?
There is a blog here describes clearly how to run NUnit tests on TFS: Running NUnit Tests in a TFS 2015 Build vNext
Simply summarized as follows:
Add Nuget Package for NUnit Test Adapter
Specify path of custom Test Adapter inside build definition
Copying adapters inside Visual Studio TestWindows folder
Specify Path to Custom Test Adapter with nunit packages
Note：
You do need to edit the Visual Studio Test task’s advanced settings
to point at the NuGet packages folder for your solution (which will be
populated via NuGet restore) so the custom nUnit test adaptor can be
found i.e. usually setting it to $(Build.SourcesDirectory)\packages
Some other tutorial for you reference:
xUnit or NUnit with Visual Studio Online Build
Running nUnit and Jasmine.JS unit tests in TFS/VSO vNext build
I have a visual studio loadTest project and LoadTest2010 result database, when I run the test from visual studio then test work fine and test data save in the LoadTest2010 DB but when run the test from VNext then test run fine (did all the configurations) but data does not save in the DB LoadTest2010.
In VNext VsoAgent agent is running the test.
Did any one face this issue?
vNext is TFS Test Agent and it is VS2015 Test Agent, we ever used Test Controller and Test Agent for VS2013 Update5. vNext is Test Agent for VS2015.
According to your description, whether you have added a web test project in load test, if no, please make sure you have added it.
and then please use MSTest to run load test locally to see the results.
What test steps do you use in your build definition? If you use the steps Visual Studio Test Agent deployment and Run functional test to run your load test, it may get the result like you posted above. Because these 2 steps are using VS Test agent 2015 and if you want to run load test, you need to install VS Test Agent 2013 according to this document.
So about using the VS 2013 test agent in Vnext build, you could follow the step below:
Add a test setting file for your test solution.
Next, specify test controller and Load test result store in the setting file.
Then add a Visual Studio Test step in your build definition and in this step, remember to specify the testsetting file.
this is the only one option right now in the VNext to run the loadtest and data will save to the loadtest database. Below is the snapshot from VNext
We have a CI system using TFS 2015, and all was working well.
However, we added some new tests, and the "Gather Artifacts" stage stopped working.
I tracked this down to the fact that the MSTest command-line gets built thus:
MsTest.exe /testcontainer:blah /test:test1 /test:test2 [...] /resultsfile:"c:\blah.trx"
So when we go above a certain number of tests, the command-line ends up too long and becomes truncated, losing the /resultsfile switch and possibly some of the /test: switches also.
In this instance, TFS reports the suite as having passed, despite the fact that all tests did not run.
I see in the documentation for MSTest.exe that there is a /testlist: parameter that lets you specify which tests run by pointing at a file.
Can anyone help me work out how I switch over to using a testlist, or some other way of solving this issue?
I found the solution, it was to switch from using "Test Plan" to "Test Assembly"
Just as DaveShaw said in the comment, the limitation is not of MSTest or TFS, it's related to Windows Command Line AFAIK.
We can use both VSTEST and MSTEST to run automated unit and coded UI tests from a command line.
VSTest.Console.exe is optimized for performance and is used in place
of MSTest.exe in Visual Studio.
After finish running the test. It won’t save the results and there is no .trx file generated in the target directory.
MSTest.exe You can use the MSTest.exe program to run automated tests
in a test assembly from a command line.
MSTest is used for load tests and for compatibility with Visual Studio 2010 test projects.
MSTest can also be used to view the test results from these test runs, save the results to disk, and save your results to Team Foundation Server.
More detail info you could refer this link: How to choose between tcm.exe, mstest.exe and vstest.console.exe
Since you are using TFS2015, and if you don't have any VS2010 test projects,
we encourage you to use VStest instead of MStest.
We're using cake for defining our .NET builds, primarily so we can run the same build on developer stations as on the build server. TFS 2013 is our actual CI platform; the build workflow is effectively just a RunScript activity which invokes powershell and runs cake via its build.ps1 script.
The basic build is working well enough, and I'm on to having it generate reports (unit test results, coverage reports, etc). I'd like to have these reports appear in the build's Summary screen, but at the moment the only feedback the build gives me is the console output from cake under the build logs. The report files are being generated and dropped into the build's ./tst/ folder, but the contents of that folder aren't appearing anywhere in the build information.
How can I get test reports to be added to the build summary and/or information pages?
With the default build process template, when you build a test project, you will be able to get test result and code coverage (if you enable it) by default. So if you use VS Test Runner to run the tests, you can refer to the default process template.
If you need other tool to run the teats, you can add the InvokeProcess activity (execute the command line) to invoke the tool to test your project in build process template.
In addition, you can use the WriteCustomSummaryInformation activity in your workflow. The result is that you can display results, hyperlinks, and more on the build summary page.
My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.
Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.
If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?
Here is our pattern matching for our SpecFlow test project:
And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests:
UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:
Test method
AccessTheNDCHomePage threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)
After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?
If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.
The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.
In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.
Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.
if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.
So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.
EDIT
It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.
Here were some of the issues and solutions we found to set up SpecFlow on our build server:
The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue
WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.
I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.
The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.
https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive