I am having some problems after enabling Code Coverage during our TFS builds. The solution under test contain three test projects and out of 209 tests three tests fail (one in one project and two in another), with the following error:
Test method MyNamespace.MyTest threw exception:
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: Unexpected error returned by SetDetourProvider in profiler library 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\IntelliTrace\14.0.0\Microsoft.IntelliTrace.Profiler.14.0.0.dll'.
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.Initialize()
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationRuntime.InitializeUnitTestIsolationInstrumentationProvider()
Microsoft.QualityTools.Testing.Fakes.Shims.ShimRuntime.CreateContext()
Microsoft.QualityTools.Testing.Fakes.ShimsContext.Create()
MyNamespace.MyTest () in MyFile.cs: line 224
The stack trace indicates that is connected to the use of Shims. All test pass locally even when running Code Coverage and disabling Code Coverage during build makes the tests pass again.
The issue seems slightly related to Microsoft Fakes generates UnitTestIsolationException, but Resharper is not installed on the build servers.
What I have tried:
Different build types: Build vNext, XAML Build with customized or Default Template
Explicitly disabled IntelliTrace using /p:EnableIntelliTrace=true and .runsetting
Added Diagnostic="true" to all .fakes files
Increased Logging Verbosity to get more error information
Run the tests locally on the server through command line
My build server setup:
Windows Server 2012 R2
Team Foundation Server 2015.3
Visual Studio Enterprise 2015.3
I struggled with the same error message, though in my case it was when running the tests themselves. When executed from the commandline, vstest.console.exe worked flawlessly, but when run from a Bamboo build task, it failed with the message above.
The problem was ultimately the user running the task - the build server used the SYSTEM user. When I ran the task as a normal privilege user, it succeeded.
According to Feedbacks UnitTestIsolationException when debugging tests using Fakes with IntelliTrace Enabled and Debugging a c# Unit Test with fakes and Intellitrace turned on throws UnitTestIsolationException, Fakes does not work if intellitrace is enabled in test setting.
Related
We are trying to integrate the Coded UI Test automation into TFS 2015. The build definition is created and is working correctly. But Release Definition is having problem.
Below are the tasks added for Release Definition:
1. Windows Machine File Copy
2. Visual Studio Test Agent Deployment
3. Run Functional Tests
Please find the attached screen shots for each task:
WindowsMachineFileCopy
Visual Studio Test Agent Deployment
Run Functional Tests
Please find the attachment for the automated test case which is associated in Run Functional Test Task.
Automated Test Case
The Release runs successfully till Visual Studio Test Agent Deployment. Even the Test Agent pop up is opened. After this, the browser is not getting opened i.e., while running Run functional Tests task, it throws below error:
"Unit Test Adapter threw exception:
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.."
I have placed all the dll's including the references in bin folder. Please find below screen shot:
DLL's
I am using Visual Studio Premium 2012, Test Agent 2015 Update 3 and TFS 2015.
Installed Test Agent 2013 and now getting attached error in Deploy Test Agent task:
Test Agent Deploy error
Please suggest the solution so that my coded ui starts running through TFS.
You need to install Test Agent for Visual Studio 2013 on the test machine based on the Compatible versions of TFS,and test agent
Also there is a blog introduce the related issue :Test Agents support for Visual Studio 2015
So, just try to use Test Agent for VS 2013 instead of Test Agent 2015 Update 3, then try it again.
We have a CI system using TFS 2015, and all was working well.
However, we added some new tests, and the "Gather Artifacts" stage stopped working.
I tracked this down to the fact that the MSTest command-line gets built thus:
MsTest.exe /testcontainer:blah /test:test1 /test:test2 [...] /resultsfile:"c:\blah.trx"
So when we go above a certain number of tests, the command-line ends up too long and becomes truncated, losing the /resultsfile switch and possibly some of the /test: switches also.
In this instance, TFS reports the suite as having passed, despite the fact that all tests did not run.
I see in the documentation for MSTest.exe that there is a /testlist: parameter that lets you specify which tests run by pointing at a file.
Can anyone help me work out how I switch over to using a testlist, or some other way of solving this issue?
I found the solution, it was to switch from using "Test Plan" to "Test Assembly"
Just as DaveShaw said in the comment, the limitation is not of MSTest or TFS, it's related to Windows Command Line AFAIK.
We can use both VSTEST and MSTEST to run automated unit and coded UI tests from a command line.
VSTest.Console.exe is optimized for performance and is used in place
of MSTest.exe in Visual Studio.
After finish running the test. It won’t save the results and there is no .trx file generated in the target directory.
MSTest.exe You can use the MSTest.exe program to run automated tests
in a test assembly from a command line.
MSTest is used for load tests and for compatibility with Visual Studio 2010 test projects.
MSTest can also be used to view the test results from these test runs, save the results to disk, and save your results to Team Foundation Server.
More detail info you could refer this link: How to choose between tcm.exe, mstest.exe and vstest.console.exe
Since you are using TFS2015, and if you don't have any VS2010 test projects,
we encourage you to use VStest instead of MStest.
We have come across an issue where TFS build fails to trigger coded UI tests, based on how the Test Settings file is specified in the build definition.
We have been unable to get CodedUI test project to run. The goal is to have it run on a separate machine (virtual machine). We have configured the test controller and agent on the VM. We have also set it to run as interactive process. When we run build it partially succeeds and the following test error is reported
The failure message returned is as below:
MsTestAdapter failed to discover tests in class ‘MVC_Test.MVCTest_CUI’ of assembly ‘C:\Builds\1\Test_COE\MVCAppFramework\bin\MVC_Test.dll’. Reason Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=12.0.0.0. Culture =neutral, PublickeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified..
Setup / Environment Details:
Visual studio 2015 ultimate
TFS server 2015
Agents for Visual studio 2015(Test Controller and Test Agents)
Test agent is installed as interactive process for Coded UI tests
Your CodedUI tests were written in the 2013 version of VS. I believe you have two options:
Install the Agents for Visual Studio 2013 on your test agent
Use binding redirects to automatically use the newer version as described in this TFS 2010 to 2015 Upgrade: CodedUI Feature Test Builds blog post
UPDATE
You could try upgrading your CodedUI DLLS to the VS2015 versions (14.0.0.0).
It cannot find the dll libraries for Version 12.0 which provided by VS2013. Install VS2013 on the test agent and then retry.
My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.
Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.
If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?
Here is our pattern matching for our SpecFlow test project:
And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests:
UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:
Test method
AccessTheNDCHomePage threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)
After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?
If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.
The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.
In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.
Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.
if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.
So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.
EDIT
It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.
Here were some of the issues and solutions we found to set up SpecFlow on our build server:
The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue
WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.
I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.
The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.
https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive
I am attempting to run UI tests (using Windows Automation API) on a WPF application, on a CI basis. I have associated my solution with a .testsettings file and linked my test .dll to the build process. All seems to be well.
When I queue a new build, in TFS's build log I can see the command line arguments passed to MSTest (2010) on the build machine - it appears that the .testsettings file is loaded correctly and that the correct test .dll is passed to MSTest.exe.
However, it seems to be "stuck" after the Executing tests message - both when running through the build and by running it directly through the build machine. In addition, I have specified a startup script to run before the tests, in the .testsettings file (it copies the application's binaries locally so the test can start the application and run the applicable tests). This script doesn't seem to be executed (the directory to which the files are supposed to be copied is empty).
Has anyone seen a similar problem and knows how to deal with it?
As Mike said in the comments, the issue was actually a bug in VS 2010 that made it impossible to run tests through the 2010 IDE if you have VS 2012 installed side by side. Unfortunately, I was unable to install VS 2010 SP1 on the build machine, but it is also possible to solve the issue by adding the /noisolation argument to the standard command line syntax (this runs the tests through the MSTest process, which somehow solves the issue).
However, since you cannot define additional arguments to be passed to MSTest through TFS's integrated automated testing feature, I wrote my own application that invokes as a scheduled task, runs the tests and sends an email containing an HTML report (I used trx2html for that -- note that if you're using VS2012, you'll need the beta 0.7 version, as there's been a change in the .trx format since VS 2010).