Another few TFS questions if anyone can help even remotely :-)
We currently have TFS 2012 installed. We have 2 servers (both VMs running Windows Server 2008 R2 x64). One server is TFS with the application layer and database (running SQL Server 2008 R2) installed. The other server is our BUILD server with the build controller and build agent installed on it.
We can connect to TFS 2012 OK from our local developer PCs using Visual Studio 2012 Ultimate.
We then have various complex build definitions that compiles our code, runs unit tests, generates real-time sandcastle documentation and installs to our multi tiered, multi server TEST or LIVE environments using InstallShield (takes SSL and performs pre-install conditions etc)
We are using the UpgradeTemplate.xaml build process definition along with TFSBuild.proj file. FYI it was originally developed in TFS 2008 and has served us well hence why we don't WANT to upgrade to a modified defaulttemplate.xaml
We now want to include "post deployment tests" (aka smoking testing or system testing or end-to-end testing I believe) in the form of Coded UI tests.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an internative process and not a service). Is this correct?
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
1.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
If the coded ui test is used as a stand alone unittest in the unittest dll, mstest should pick up the coded ui tests. If the coded ui test is however an automation of a test case, an alternative is to use the TCM command line tooling to kick of the test case and thereby executing the coded ui test.
2.
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an interactive process and not a service). Is this correct?
Yes: You would need an interactive build agent.
No: the current build agent could also be set to interactive.
3.
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Yes: It works as the articles imply
No: The intellitrace collectors and logs are usually more useful than a video recording. The articles also forget to mention that your TFS server might grow out of proportions if not handled with care. http://geekswithblogs.net/terje/archive/2011/11/15/guide-to-reduce-tfs-database-growth-using-the-test-attachment.aspx
4.
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
No, there not the same and don't try to pretend that (like blend). The focus of the tools are different; MTM is focused on testing by a tester (functional, manual, exploratory), testing in visual studio is focused on testing as a developer (unittest, webtesting, automating tests). Visual Studio ultimate therefore can't edit and create test cases and suites. MTM can't create develop (coded) unittest tests. Installing visual studio ultimate also installs MTM, so you can use both if needed.
Related
We were using MTM 2015 and TFS 2015. While trying to run the Coded UI automation with vNext build started encountering an issue
The build directory of the test run either does not exist or access
permission is required
Started digging deep and then came across this link msdn.
At the end if you see Q and A:
Q: Can I use vNext builds to run automated tests in lab environments?
A: Currently only XAML builds are supported with automated testing.
Microsoft Test Manager does not support Team Foundation 2015 build
(also called Build vNext). Microsoft Test Manager will allow you to
associate a vNext build with your Test Plan, but when you try to
trigger an automated test using a lab environment, your test run will
fail and the error log will show you the following message - which may
be misleading.
The build directory of the test run either does not exist or access permission is required
Question is does MTM 2017 supports Coded UI automation execution with vNext build? If not when can we expect this MTM + vNext build work without any issues?
No. That feature will not be added to MTM; MTM and Lab Management are deprecated in favor of the full web-based test hub experience. When you run MTM 2017, it even pops up a warning telling you this. It's also worth noting that Coded UI is deprecated in favor of Selenium for web browser-based testing and Appium for desktop testing. Although all of these tools are still supported, they are no longer receiving feature updates.
You can run test automation from release definitions in TFS 2017 and 2018.
I am a newbie to TFS Build Pipeline. Is there any difference between Unified Build Agent vs Unified Test Agents in TFS 2018 Context?
What are Unified Agents and how they help to minimize effort and Build machines pool? Any Step by Step Guide.
PS: I have setup simple build definition and it is compiling the code. For testing, the build output gets a copy on multiple dedicated machines where the test runs.
I was doing some R & D and found that there is a concept Unified Agent where you are not required to copy Build output to multiple dedicated machines and run tests there.
Unified Agents do this itself without copying build output and run tests without dedicating machines. You just need to mention VMs (Test Machines) in a Machines Pool where it automatically pics it up.
It seems you have confusion about Visual Studio Test task. With this task, you can run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. In version 2.* of Visual Studio Test task, tests can be distributed on multiple agents.
What's new in version 2.*
Run tests using agent phase: Unified agent across Build, Release and Test allows for automation agents to be used for testing
purposes as well. You can distribute tests using the multi-agent
phase setting. Multi-config phase setting can be used to replicate
tests in different configurations. Check this blog for more information: https://blogs.msdn.microsoft.com/devops/2017/03/26/vstest-task-dons-a-new-avatar-testing-with-unified-agents-and-phases/
Test Impact Analysis: Automatically select and run only the tests needed to validate the code change.
Use the Visual Studio Test Platform Installer task to run tests without needing a full Visual Studio installation.
Our plan is to use Team Foundation Build service to do automatic builds, then use the testing facility to automatically perform testing on the build server then release that build onto the application server.
So far we have
Team Foundation Server with TF Build Controller configured
Build server with win2012, Visual Studio 2013 and Build agent configured.
SQL Server with SQL 2013 installed
Application Server with Win2012 and .netframework installed
My question is what do I need to do to set up automatic builds, and to execute the unit test harness once compilation is successful.
Also the deployment target machine will initially be DEV, however we would like to quickly build for test env and prod etc.
This is what I got so far.
Build Controller (Already set up I believe)
Build Agent (Already installed on build server)
Build Process Template (Do I need to do anything with this. Is this what controls the whole lot)
Team Build Definition (I had a look at this, and it seems to use the build process template)
Drop Folder (I am assuming this is where the executables will be dropped into).
At the moment I have bits and pieces of info, what I would like to know is how this whole thing is hanging together. From the moment the developer wants to do the build to the moment that exe is placed into the DEVAPPSERV (Development application server).
Is anyone able to point me in the right direction or give a summary of what I need to make this happen?
Many thanks,
Dalibor
Install TFS Server (TFS Disk) Create a Team Project Collection and any desired Projects
Install TFS Controller + Agents onto a dedicated machine (TFS Disk) Configure only the build options if on a different machine to the TFS Server
Configure Build Controller to connect to a Specific Team Collection on your TFS Server
Install VS Premium or higher on build machine, if you want code coverage results for your tests
Add some code to TFS Source Control
Create a Build Definition using the default template.
Configure the build definition.
Set the working folder for the build, include only what you need as this will speed up the process
Point the definition to your .sln or proj file.
Ensure testing is enabled and that your test assembly names will match the regex used to identify test dll's i.e. name your test assemblies with the word test.
Set the trigger to be CI or what ever flavour of build you require i.e. gated build
Save the build definition
Trigger a manual build and debug any issues
you should have the basics done and a repeatable build created.
That should cover the basics, you may want to customise the build template (see Ewald Hoffman's guide for tips), you may want to narrow down your code coverage (look for runsettings file info).
If you follow these steps you should be able to get a basic build created and running from these, if you hit any issues you can come back and ask specific questions about a particular area
In order to do automatic builds you should check the CI build option ( under the trigger build option ) and third party automated testing can be run by executed by a post build script.
See the following TFS article about post build scripts.
http://msdn.microsoft.com/en-us/library/dn376353.aspx
I'm looking to get TFS to build and deploy to our manual and auto TEST environments.
manual TEST - overnight updates
auto TEST - continuous integration
Once deployed, I want TFS to deploy and run a set of web-integration tests (SpecFlow / Selenium / nUnit) on another server (acting pseudo client of the website).
I then want TFS to collate the results of these tests and report back.
What's the best approach to get this working?
You'd normally change the build process template to do what you want. Adding steps to deploy could be one of these customozations. Have a look at the Build Release Deploy build template from the Microsoft ALM Rangers which includes steps to publish the build to another machine.
You can also make use of WebDeploy and add a set of publish profiles to your solution. Team Build will happily execute those steps during build, if told to do so.
In TFS you can use a Test Agent to then execute tests from a remote machine and have the results added to the test results for your build.
Seeing that you'd like to use NUnit ad SpecFlow I suggest using TFS 2012 in combination with Visual Studio 2012, which have built-in support for executing 3rd party based tests. You'll need to install the NUnit adapter and put the assemblies for it in source control so that Team Build will pick them up.
We would like to have our TFS automated CI build and test process use a remote test agent to run our tests separate from our build agent. On normal unit tests that do not use moles or moles host, we have no problems. However, when we include a moles host in our .testsettings or have a moles hosted test (via the HostType("Moles") attribute), the test agent seems to try and get the path to the moled .dlls that the build agent uses. It is almost as if the build agent passes a full path to the test agent and the test agent uses it. We either get a "You do not have the appropriate permissions to perform this action" (when in fact each service is running with administrative rights) or a cryptic "MSTest failed returned code 1 expected 0" and no logging information.
My questions for the group:
Has anyone gotten MSTest, Moles, TFS Build, and Visual Studio 2010
Test Controller/Test Agent (not part of the Test Lab setup) to work?
If you have gotten it to work, what is your setup?
It doesn't work.
While my scenario WAS in TFS Lab Mgmt, I found the problem really didn't relate to lab mgmt, but instead in the fact the the Moles installer looks for registry keys just don't exist when you only install the remote Test Agent without VS 2010 installed. See the following question for my Question & Answer:
Microsoft Moles HostType errors in TFS 2010 Lab Management environment
I stress that while I was using TFS Lab Management, it didn't matter. The issue occurs any time you want to run remote tests with VS installed.