I am attempting to run UI tests (using Windows Automation API) on a WPF application, on a CI basis. I have associated my solution with a .testsettings file and linked my test .dll to the build process. All seems to be well.
When I queue a new build, in TFS's build log I can see the command line arguments passed to MSTest (2010) on the build machine - it appears that the .testsettings file is loaded correctly and that the correct test .dll is passed to MSTest.exe.
However, it seems to be "stuck" after the Executing tests message - both when running through the build and by running it directly through the build machine. In addition, I have specified a startup script to run before the tests, in the .testsettings file (it copies the application's binaries locally so the test can start the application and run the applicable tests). This script doesn't seem to be executed (the directory to which the files are supposed to be copied is empty).
Has anyone seen a similar problem and knows how to deal with it?
As Mike said in the comments, the issue was actually a bug in VS 2010 that made it impossible to run tests through the 2010 IDE if you have VS 2012 installed side by side. Unfortunately, I was unable to install VS 2010 SP1 on the build machine, but it is also possible to solve the issue by adding the /noisolation argument to the standard command line syntax (this runs the tests through the MSTest process, which somehow solves the issue).
However, since you cannot define additional arguments to be passed to MSTest through TFS's integrated automated testing feature, I wrote my own application that invokes as a scheduled task, runs the tests and sends an email containing an HTML report (I used trx2html for that -- note that if you're using VS2012, you'll need the beta 0.7 version, as there's been a change in the .trx format since VS 2010).
Related
We have a CI system using TFS 2015, and all was working well.
However, we added some new tests, and the "Gather Artifacts" stage stopped working.
I tracked this down to the fact that the MSTest command-line gets built thus:
MsTest.exe /testcontainer:blah /test:test1 /test:test2 [...] /resultsfile:"c:\blah.trx"
So when we go above a certain number of tests, the command-line ends up too long and becomes truncated, losing the /resultsfile switch and possibly some of the /test: switches also.
In this instance, TFS reports the suite as having passed, despite the fact that all tests did not run.
I see in the documentation for MSTest.exe that there is a /testlist: parameter that lets you specify which tests run by pointing at a file.
Can anyone help me work out how I switch over to using a testlist, or some other way of solving this issue?
I found the solution, it was to switch from using "Test Plan" to "Test Assembly"
Just as DaveShaw said in the comment, the limitation is not of MSTest or TFS, it's related to Windows Command Line AFAIK.
We can use both VSTEST and MSTEST to run automated unit and coded UI tests from a command line.
VSTest.Console.exe is optimized for performance and is used in place
of MSTest.exe in Visual Studio.
After finish running the test. It won’t save the results and there is no .trx file generated in the target directory.
MSTest.exe You can use the MSTest.exe program to run automated tests
in a test assembly from a command line.
MSTest is used for load tests and for compatibility with Visual Studio 2010 test projects.
MSTest can also be used to view the test results from these test runs, save the results to disk, and save your results to Team Foundation Server.
More detail info you could refer this link: How to choose between tcm.exe, mstest.exe and vstest.console.exe
Since you are using TFS2015, and if you don't have any VS2010 test projects,
we encourage you to use VStest instead of MStest.
I am having some problems after enabling Code Coverage during our TFS builds. The solution under test contain three test projects and out of 209 tests three tests fail (one in one project and two in another), with the following error:
Test method MyNamespace.MyTest threw exception:
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: Unexpected error returned by SetDetourProvider in profiler library 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\IntelliTrace\14.0.0\Microsoft.IntelliTrace.Profiler.14.0.0.dll'.
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.Initialize()
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationRuntime.InitializeUnitTestIsolationInstrumentationProvider()
Microsoft.QualityTools.Testing.Fakes.Shims.ShimRuntime.CreateContext()
Microsoft.QualityTools.Testing.Fakes.ShimsContext.Create()
MyNamespace.MyTest () in MyFile.cs: line 224
The stack trace indicates that is connected to the use of Shims. All test pass locally even when running Code Coverage and disabling Code Coverage during build makes the tests pass again.
The issue seems slightly related to Microsoft Fakes generates UnitTestIsolationException, but Resharper is not installed on the build servers.
What I have tried:
Different build types: Build vNext, XAML Build with customized or Default Template
Explicitly disabled IntelliTrace using /p:EnableIntelliTrace=true and .runsetting
Added Diagnostic="true" to all .fakes files
Increased Logging Verbosity to get more error information
Run the tests locally on the server through command line
My build server setup:
Windows Server 2012 R2
Team Foundation Server 2015.3
Visual Studio Enterprise 2015.3
I struggled with the same error message, though in my case it was when running the tests themselves. When executed from the commandline, vstest.console.exe worked flawlessly, but when run from a Bamboo build task, it failed with the message above.
The problem was ultimately the user running the task - the build server used the SYSTEM user. When I ran the task as a normal privilege user, it succeeded.
According to Feedbacks UnitTestIsolationException when debugging tests using Fakes with IntelliTrace Enabled and Debugging a c# Unit Test with fakes and Intellitrace turned on throws UnitTestIsolationException, Fakes does not work if intellitrace is enabled in test setting.
My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.
Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.
If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?
Here is our pattern matching for our SpecFlow test project:
And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests:
UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:
Test method
AccessTheNDCHomePage threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)
After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?
If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.
The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.
In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.
Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.
if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.
So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.
EDIT
It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.
Here were some of the issues and solutions we found to set up SpecFlow on our build server:
The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue
WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.
I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.
The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.
https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive
I’ve been reading some articles (this one in particular: http://speaksql.wordpress.com/2013/06/07/a-journey-to-db-deployment-automaton-ssis-build-using-msbuild/) about using msbuild.exe to build and deploy an SSIS package (.ispac). I had no problem with that from my computer which has all the required assemblies, and only using the msbuild.exe command.
Once I tried to use the TFS Build Server I had some problems. First I realized that using a project with msbuild.exe (SSIS.MSBuild.proj) was not recommended with TFS 2013 since it was used with TFS 2008 and 2010. Anyway, I just wanted to make it work and it sounded plausible, but I had to use the TFSBuild.proj. That was not a problem, but my next problem was that the project I was being using (Microsoft SQL Server Community Samples: Integration Services) references a SQL Server assembly (Microsoft.SqlServer.ManagedDTS) which is not installed on my build server.
Then I realized that even if I managed to install that assembly on the gac, or referenced it on a relative path I would have a bigger problem next, I am using custom activities on my packages which I need to install using gacutil.exe on the host server, and I was wondering how to install, remotely, those dll.
That’s when I started to lose faith, and here I am, is there an “easy” way to implement continuous integration for SSIS packages without installing third party tools (http://remotegacutil.codeplex.com/ for example), and adding missing assemblies to the gac of a build server?
Did any of you have a similar issue? Did you solve it? How?
Thanks!
Use devenv.exe to build the ispac. Add an invoke process task and call out to devenv.
Add an Invoke Process to the Build Template, the one above shows a Sequence container, which assigns the path to the correct version of DeVenv to call, VS2010 /12 /13 etc. this is shown as hard coded but can be put into an argument, so it can be populated in individual build definitions. the one you can't see clearly is "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\devenv.com"
Next is the invoke Process, this call out to the specified devenv.exe and passes arguments used to build the ISPAC file.
The arguments are passed in localProject would get you the actual project you wanted to build in previous versions, you may have to do something different for 2013, due to the changes to the new templates.
this gives a TFS2012 way of doing it, as i say you may have to do something different to get the project that you want to build under 2013, but the build will run and an .ISPAC file will be generated.
at this point i would deploy the ISPAC using powershell, you may want to add additional scripts to the powershell calls to handle creating the SSIS Catalog and scheduling of the job.
deploying with Powershell can be found here Deploying ISPAC's with Powershell
Another few TFS questions if anyone can help even remotely :-)
We currently have TFS 2012 installed. We have 2 servers (both VMs running Windows Server 2008 R2 x64). One server is TFS with the application layer and database (running SQL Server 2008 R2) installed. The other server is our BUILD server with the build controller and build agent installed on it.
We can connect to TFS 2012 OK from our local developer PCs using Visual Studio 2012 Ultimate.
We then have various complex build definitions that compiles our code, runs unit tests, generates real-time sandcastle documentation and installs to our multi tiered, multi server TEST or LIVE environments using InstallShield (takes SSL and performs pre-install conditions etc)
We are using the UpgradeTemplate.xaml build process definition along with TFSBuild.proj file. FYI it was originally developed in TFS 2008 and has served us well hence why we don't WANT to upgrade to a modified defaulttemplate.xaml
We now want to include "post deployment tests" (aka smoking testing or system testing or end-to-end testing I believe) in the form of Coded UI tests.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an internative process and not a service). Is this correct?
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
1.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
If the coded ui test is used as a stand alone unittest in the unittest dll, mstest should pick up the coded ui tests. If the coded ui test is however an automation of a test case, an alternative is to use the TCM command line tooling to kick of the test case and thereby executing the coded ui test.
2.
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an interactive process and not a service). Is this correct?
Yes: You would need an interactive build agent.
No: the current build agent could also be set to interactive.
3.
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Yes: It works as the articles imply
No: The intellitrace collectors and logs are usually more useful than a video recording. The articles also forget to mention that your TFS server might grow out of proportions if not handled with care. http://geekswithblogs.net/terje/archive/2011/11/15/guide-to-reduce-tfs-database-growth-using-the-test-attachment.aspx
4.
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
No, there not the same and don't try to pretend that (like blend). The focus of the tools are different; MTM is focused on testing by a tester (functional, manual, exploratory), testing in visual studio is focused on testing as a developer (unittest, webtesting, automating tests). Visual Studio ultimate therefore can't edit and create test cases and suites. MTM can't create develop (coded) unittest tests. Installing visual studio ultimate also installs MTM, so you can use both if needed.