We have a CI system using TFS 2015, and all was working well.
However, we added some new tests, and the "Gather Artifacts" stage stopped working.
I tracked this down to the fact that the MSTest command-line gets built thus:
MsTest.exe /testcontainer:blah /test:test1 /test:test2 [...] /resultsfile:"c:\blah.trx"
So when we go above a certain number of tests, the command-line ends up too long and becomes truncated, losing the /resultsfile switch and possibly some of the /test: switches also.
In this instance, TFS reports the suite as having passed, despite the fact that all tests did not run.
I see in the documentation for MSTest.exe that there is a /testlist: parameter that lets you specify which tests run by pointing at a file.
Can anyone help me work out how I switch over to using a testlist, or some other way of solving this issue?
I found the solution, it was to switch from using "Test Plan" to "Test Assembly"
Just as DaveShaw said in the comment, the limitation is not of MSTest or TFS, it's related to Windows Command Line AFAIK.
We can use both VSTEST and MSTEST to run automated unit and coded UI tests from a command line.
VSTest.Console.exe is optimized for performance and is used in place
of MSTest.exe in Visual Studio.
After finish running the test. It won’t save the results and there is no .trx file generated in the target directory.
MSTest.exe You can use the MSTest.exe program to run automated tests
in a test assembly from a command line.
MSTest is used for load tests and for compatibility with Visual Studio 2010 test projects.
MSTest can also be used to view the test results from these test runs, save the results to disk, and save your results to Team Foundation Server.
More detail info you could refer this link: How to choose between tcm.exe, mstest.exe and vstest.console.exe
Since you are using TFS2015, and if you don't have any VS2010 test projects,
we encourage you to use VStest instead of MStest.
Related
Question:
How do I run NUnit tests on TFS?
Attempts:
I've searched the web and am still struggling to figure out the exact steps required to accomplish (to what I believe is) a trivial task.
I added the NUnit3TestAdapter package to my test project.
However, my tests are still not being discovered.
TFS displays the following message:
No test runs are available for this build. Enable automated tests in
your build pipeline by running your test framework of choice, such as
JUnit, Visual Studio Test, or xUnit. If you choose to run tests using
a custom task or runner, you can publish results using the Publish
Test Results task.
Expectation:
I'm surprised that NUnit is not an option that I can just select from.Hence, it's the standard unit test framework for .Net.
How do I run NUnit tests on TFS?
There is a blog here describes clearly how to run NUnit tests on TFS: Running NUnit Tests in a TFS 2015 Build vNext
Simply summarized as follows:
Add Nuget Package for NUnit Test Adapter
Specify path of custom Test Adapter inside build definition
Copying adapters inside Visual Studio TestWindows folder
Specify Path to Custom Test Adapter with nunit packages
Note：
You do need to edit the Visual Studio Test task’s advanced settings
to point at the NuGet packages folder for your solution (which will be
populated via NuGet restore) so the custom nUnit test adaptor can be
found i.e. usually setting it to $(Build.SourcesDirectory)\packages
Some other tutorial for you reference:
xUnit or NUnit with Visual Studio Online Build
Running nUnit and Jasmine.JS unit tests in TFS/VSO vNext build
I am using a TFS Release to execute .loadtests written in Visual Studio.
I can execute the tests fine using MSTest, this produces the expected .trx file.
What I want is to be able to see the test results within TFS, I have tried using the 'Publish Test Results' task but have experienced issues.
The issue encountered was an Object not set to an instance of an object
Upon analysing the powershell scripts that 'Publish Test Results' task executes, from what I can see, the only results that are read in the TrxResultReader are from 'WebTest' and 'UnitTest'.
Has anyone managed to publish the results of a loadtest within TFS?
For TFS Publish Test Results task. This task supports popular test result formats including JUnit, NUnit 2, NUnit 3, Visual Studio Test (TRX), and xUnit 2. If you use the built-in tasks such as Visual Studio Test to run tests, results are automatically published and you do not need a separate publish test results task.
Based on the powershell script behind the build task, it use TestRunner. Since Vstest cannot run load test, it cannot publish load test result too. Take a look at below similar question:
Publish Test Results task fail when trying to publish a loadtest
It is not possible to use mstest to publish test results to build
made by a vNext. mstest can not identify the correct buildnummer /
buildid. Our solution will probably be that we store the *.trx file a
artifact. An other solution is to use a xaml build to perform
performance tests.
I am new to TFS and want to integrate OpenCover with TFS. If any has done this please help!
This question is rather old but maybe you are still interested. With current Version of TFS (2015 Update 2) this is now possible as a "vsts Extension".
See here for details: https://github.com/RobertK66/vsts-opencover
Since the answer doesn't specify the version of TFS, here is an answer for 2015/2017.
OpenCover can be run from TFS using the Powershell build step. You need to get the contents of the OpenCover NuGet package onto TFS and run OpenCover.console.exe from there.
Since TFS doesn't support the format produced by OpenCover, you need to take one additional step and convert the results to Cobertura format. It's possible using the OpenCoverToCoberturaConverter NuGet package.
I've described the whole process in much more detail on my blog:
http://codewithstyle.info/setting-up-coverage-reports-on-vsts-with-opencover/
OpenCover is just a console application so you can just modify your scripts to get OpenCover to run your unit tests.
I haven't used TFS for several years and it has changed since then however this blog post should help
To incorporate coverage measurement of OpenCover the build process of TFS (second half)
The original is in Japanese but if you are familiar with TFS then the screens will probably be obvious.
OpenCover also comes with an MSBuild task that may help you with your integration.
I've just integrated opencover with TFS Build, to generate a xml with the results that will be processed by sonar:
Created a RunCoverage.cmd at the root of my source folder
Installed / copied in TFS Servers the Opencover binaries and added into Path(must restart TFS Services to enable TFS to see it).
Created a new actitivity in the build definition, that is, editing the build template (before sonar execution activity) to run the cmd:
Running command line statements from TFS custom activity
http://msdn.microsoft.com/en-us/library/gg265783.aspx
There is the cmd contents:
#REM #echo suppresses command line. ^ is line continuation character
#ECHO Executing OpenCover...
OpenCover.Console.exe -register:user ^
-target:"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" ^
-targetargs:"/testcontainer:%~dp0test.dll /usestderr /resultsfile:%~dp0MSTestsResults.trx" ^
-output:%~dp0opencovertests.xml
But i'm facing three issues (that are related with my concrete implementation, but you may face them):
The Tests are runned twice (one for template itself, and other for OpenCover)
The MsTest.exe in the TFS Servers is not in the same path, so when the Controller asigns an agent (if the match is done by tags) then if the agent executing the build is in a TFS Server that does not have the MSTest in the right path it will fail.
How to inject in cmd the corresponding test runner (MsTest, XUnit, Nunit, etc...) depending on the test project
Hope that it helps! (and someone can help me ;-)
I am attempting to run UI tests (using Windows Automation API) on a WPF application, on a CI basis. I have associated my solution with a .testsettings file and linked my test .dll to the build process. All seems to be well.
When I queue a new build, in TFS's build log I can see the command line arguments passed to MSTest (2010) on the build machine - it appears that the .testsettings file is loaded correctly and that the correct test .dll is passed to MSTest.exe.
However, it seems to be "stuck" after the Executing tests message - both when running through the build and by running it directly through the build machine. In addition, I have specified a startup script to run before the tests, in the .testsettings file (it copies the application's binaries locally so the test can start the application and run the applicable tests). This script doesn't seem to be executed (the directory to which the files are supposed to be copied is empty).
Has anyone seen a similar problem and knows how to deal with it?
As Mike said in the comments, the issue was actually a bug in VS 2010 that made it impossible to run tests through the 2010 IDE if you have VS 2012 installed side by side. Unfortunately, I was unable to install VS 2010 SP1 on the build machine, but it is also possible to solve the issue by adding the /noisolation argument to the standard command line syntax (this runs the tests through the MSTest process, which somehow solves the issue).
However, since you cannot define additional arguments to be passed to MSTest through TFS's integrated automated testing feature, I wrote my own application that invokes as a scheduled task, runs the tests and sends an email containing an HTML report (I used trx2html for that -- note that if you're using VS2012, you'll need the beta 0.7 version, as there's been a change in the .trx format since VS 2010).
I have NUnit 2.5.10 and Specflow 1.8.1 installed with VS2010.
Right click on .feature file in VS, there is an item "Debug Specflow Scenarios" in context menu. But I always get "Could not find matching test runner". In Tools/Options/Specflow, there is no test runner option for NUnit.
Am I missing something?
Thanks in advance. :)
I don't know if NUnit has a seperate GUI for running tests as i use MSTests but this should be how u do it.
Setup a config file in your Test Project to tell Specflow what Test framework to use.
https://github.com/techtalk/SpecFlow/wiki/Configuration
The Tools > Options > Specflow > Test Runner tool is the tool that executes the Tests. The fact that nothing is showing here seems like the test runner is not installed. For example i have several, MSTest, Resharper, SpecRun, i normally leave it on Auto and just run the test from the test tool i want to use. I ran installers for these though which should add any VS extensions
You could try using the VS runner, Test > Windows > Test View
VS will just treat a Specflow test as a NUnit as the generated code behind is just an NUnit test (when configured that way). So i think the real question is how to debug an NUnit test in Visual Studio. A quick search suggested attaching the debugger to the NUnit test runner process.
Like Ryan I also have no test runner specifically for Nunit. However since I'm also running resharper I get additional resharper runners, and that is what Auto uses to run it.
Have you actually got nUnit installed on your machine? In which case you can test how successful the process is without VS and Specflow-addin getting in the way. If not, well installing may give you the runner you need.
Alternatively, I use nCrunch to automatically run all my tests. I've had it running nunit/specflow for the last few months on multiple solutions with no issues. No need to right click at all then. :-)
SpecFlow itself does not provide a test runner and if you want to right click on the .feature and run it, you'll have to re-configure SpecFlow to use MSTest, which should use built-in VS runner.
If you want to use NUnit, then additional tools are required. I would recommend Resharper, but it's not free. Check out this article for more options to run NUnit tests from VS.
Good day. To debug i do the following :
Put a break point on the scenario line that you are interested in
Make sure that you have NUnit running
Attach the NUnit process
Go to tools
Select attach process
In the available processes select NUnit agent.exe
Click attach
Run the test you want to debug in NUnit
Hope that helps