We would like to have our TFS automated CI build and test process use a remote test agent to run our tests separate from our build agent. On normal unit tests that do not use moles or moles host, we have no problems. However, when we include a moles host in our .testsettings or have a moles hosted test (via the HostType("Moles") attribute), the test agent seems to try and get the path to the moled .dlls that the build agent uses. It is almost as if the build agent passes a full path to the test agent and the test agent uses it. We either get a "You do not have the appropriate permissions to perform this action" (when in fact each service is running with administrative rights) or a cryptic "MSTest failed returned code 1 expected 0" and no logging information.
My questions for the group:
Has anyone gotten MSTest, Moles, TFS Build, and Visual Studio 2010
Test Controller/Test Agent (not part of the Test Lab setup) to work?
If you have gotten it to work, what is your setup?
It doesn't work.
While my scenario WAS in TFS Lab Mgmt, I found the problem really didn't relate to lab mgmt, but instead in the fact the the Moles installer looks for registry keys just don't exist when you only install the remote Test Agent without VS 2010 installed. See the following question for my Question & Answer:
Microsoft Moles HostType errors in TFS 2010 Lab Management environment
I stress that while I was using TFS Lab Management, it didn't matter. The issue occurs any time you want to run remote tests with VS installed.
Related
We are trying to integrate the Coded UI Test automation into TFS 2015. The build definition is created and is working correctly. But Release Definition is having problem.
Below are the tasks added for Release Definition:
1. Windows Machine File Copy
2. Visual Studio Test Agent Deployment
3. Run Functional Tests
Please find the attached screen shots for each task:
WindowsMachineFileCopy
Visual Studio Test Agent Deployment
Run Functional Tests
Please find the attachment for the automated test case which is associated in Run Functional Test Task.
Automated Test Case
The Release runs successfully till Visual Studio Test Agent Deployment. Even the Test Agent pop up is opened. After this, the browser is not getting opened i.e., while running Run functional Tests task, it throws below error:
"Unit Test Adapter threw exception:
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.."
I have placed all the dll's including the references in bin folder. Please find below screen shot:
DLL's
I am using Visual Studio Premium 2012, Test Agent 2015 Update 3 and TFS 2015.
Installed Test Agent 2013 and now getting attached error in Deploy Test Agent task:
Test Agent Deploy error
Please suggest the solution so that my coded ui starts running through TFS.
You need to install Test Agent for Visual Studio 2013 on the test machine based on the Compatible versions of TFS,and test agent
Also there is a blog introduce the related issue :Test Agents support for Visual Studio 2015
So, just try to use Test Agent for VS 2013 instead of Test Agent 2015 Update 3, then try it again.
I am a newbie to TFS Build Pipeline. Is there any difference between Unified Build Agent vs Unified Test Agents in TFS 2018 Context?
What are Unified Agents and how they help to minimize effort and Build machines pool? Any Step by Step Guide.
PS: I have setup simple build definition and it is compiling the code. For testing, the build output gets a copy on multiple dedicated machines where the test runs.
I was doing some R & D and found that there is a concept Unified Agent where you are not required to copy Build output to multiple dedicated machines and run tests there.
Unified Agents do this itself without copying build output and run tests without dedicating machines. You just need to mention VMs (Test Machines) in a Machines Pool where it automatically pics it up.
It seems you have confusion about Visual Studio Test task. With this task, you can run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. In version 2.* of Visual Studio Test task, tests can be distributed on multiple agents.
What's new in version 2.*
Run tests using agent phase: Unified agent across Build, Release and Test allows for automation agents to be used for testing
purposes as well. You can distribute tests using the multi-agent
phase setting. Multi-config phase setting can be used to replicate
tests in different configurations. Check this blog for more information: https://blogs.msdn.microsoft.com/devops/2017/03/26/vstest-task-dons-a-new-avatar-testing-with-unified-agents-and-phases/
Test Impact Analysis: Automatically select and run only the tests needed to validate the code change.
Use the Visual Studio Test Platform Installer task to run tests without needing a full Visual Studio installation.
We have come across an issue where TFS build fails to trigger coded UI tests, based on how the Test Settings file is specified in the build definition.
We have been unable to get CodedUI test project to run. The goal is to have it run on a separate machine (virtual machine). We have configured the test controller and agent on the VM. We have also set it to run as interactive process. When we run build it partially succeeds and the following test error is reported
The failure message returned is as below:
MsTestAdapter failed to discover tests in class ‘MVC_Test.MVCTest_CUI’ of assembly ‘C:\Builds\1\Test_COE\MVCAppFramework\bin\MVC_Test.dll’. Reason Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=12.0.0.0. Culture =neutral, PublickeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified..
Setup / Environment Details:
Visual studio 2015 ultimate
TFS server 2015
Agents for Visual studio 2015(Test Controller and Test Agents)
Test agent is installed as interactive process for Coded UI tests
Your CodedUI tests were written in the 2013 version of VS. I believe you have two options:
Install the Agents for Visual Studio 2013 on your test agent
Use binding redirects to automatically use the newer version as described in this TFS 2010 to 2015 Upgrade: CodedUI Feature Test Builds blog post
UPDATE
You could try upgrading your CodedUI DLLS to the VS2015 versions (14.0.0.0).
It cannot find the dll libraries for Version 12.0 which provided by VS2013. Install VS2013 on the test agent and then retry.
I'm looking to get TFS to build and deploy to our manual and auto TEST environments.
manual TEST - overnight updates
auto TEST - continuous integration
Once deployed, I want TFS to deploy and run a set of web-integration tests (SpecFlow / Selenium / nUnit) on another server (acting pseudo client of the website).
I then want TFS to collate the results of these tests and report back.
What's the best approach to get this working?
You'd normally change the build process template to do what you want. Adding steps to deploy could be one of these customozations. Have a look at the Build Release Deploy build template from the Microsoft ALM Rangers which includes steps to publish the build to another machine.
You can also make use of WebDeploy and add a set of publish profiles to your solution. Team Build will happily execute those steps during build, if told to do so.
In TFS you can use a Test Agent to then execute tests from a remote machine and have the results added to the test results for your build.
Seeing that you'd like to use NUnit ad SpecFlow I suggest using TFS 2012 in combination with Visual Studio 2012, which have built-in support for executing 3rd party based tests. You'll need to install the NUnit adapter and put the assemblies for it in source control so that Team Build will pick them up.
Another few TFS questions if anyone can help even remotely :-)
We currently have TFS 2012 installed. We have 2 servers (both VMs running Windows Server 2008 R2 x64). One server is TFS with the application layer and database (running SQL Server 2008 R2) installed. The other server is our BUILD server with the build controller and build agent installed on it.
We can connect to TFS 2012 OK from our local developer PCs using Visual Studio 2012 Ultimate.
We then have various complex build definitions that compiles our code, runs unit tests, generates real-time sandcastle documentation and installs to our multi tiered, multi server TEST or LIVE environments using InstallShield (takes SSL and performs pre-install conditions etc)
We are using the UpgradeTemplate.xaml build process definition along with TFSBuild.proj file. FYI it was originally developed in TFS 2008 and has served us well hence why we don't WANT to upgrade to a modified defaulttemplate.xaml
We now want to include "post deployment tests" (aka smoking testing or system testing or end-to-end testing I believe) in the form of Coded UI tests.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an internative process and not a service). Is this correct?
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
1.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
If the coded ui test is used as a stand alone unittest in the unittest dll, mstest should pick up the coded ui tests. If the coded ui test is however an automation of a test case, an alternative is to use the TCM command line tooling to kick of the test case and thereby executing the coded ui test.
2.
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an interactive process and not a service). Is this correct?
Yes: You would need an interactive build agent.
No: the current build agent could also be set to interactive.
3.
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Yes: It works as the articles imply
No: The intellitrace collectors and logs are usually more useful than a video recording. The articles also forget to mention that your TFS server might grow out of proportions if not handled with care. http://geekswithblogs.net/terje/archive/2011/11/15/guide-to-reduce-tfs-database-growth-using-the-test-attachment.aspx
4.
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
No, there not the same and don't try to pretend that (like blend). The focus of the tools are different; MTM is focused on testing by a tester (functional, manual, exploratory), testing in visual studio is focused on testing as a developer (unittest, webtesting, automating tests). Visual Studio ultimate therefore can't edit and create test cases and suites. MTM can't create develop (coded) unittest tests. Installing visual studio ultimate also installs MTM, so you can use both if needed.