We were using MTM 2015 and TFS 2015. While trying to run the Coded UI automation with vNext build started encountering an issue
The build directory of the test run either does not exist or access
permission is required
Started digging deep and then came across this link msdn.
At the end if you see Q and A:
Q: Can I use vNext builds to run automated tests in lab environments?
A: Currently only XAML builds are supported with automated testing.
Microsoft Test Manager does not support Team Foundation 2015 build
(also called Build vNext). Microsoft Test Manager will allow you to
associate a vNext build with your Test Plan, but when you try to
trigger an automated test using a lab environment, your test run will
fail and the error log will show you the following message - which may
be misleading.
The build directory of the test run either does not exist or access permission is required
Question is does MTM 2017 supports Coded UI automation execution with vNext build? If not when can we expect this MTM + vNext build work without any issues?
No. That feature will not be added to MTM; MTM and Lab Management are deprecated in favor of the full web-based test hub experience. When you run MTM 2017, it even pops up a warning telling you this. It's also worth noting that Coded UI is deprecated in favor of Selenium for web browser-based testing and Appium for desktop testing. Although all of these tools are still supported, they are no longer receiving feature updates.
You can run test automation from release definitions in TFS 2017 and 2018.
Related
I recently started to work with TFS2018 instead of TFS2017.
I had a perfect build with my TFS2017 that used Test Agents.
From TFS2018 we don't have anymore Test Agents but Build Agents.
I managed to upload a Build Agent that run my tests but the tests which uses data source from the test in TFS Tests can't access the data and fail (It says that that the machine has a missing .Net Framework and to check in the machine.config but the machine.config has the .Net Framework provider. The coded test is connected to the test in TFS with Work Item and matching test number).
Tests that don't use data passes (the build works fine).
Running the tests on local - All the tests passes (even those with data).
And I wonder - does TFS2018 not supports MSTest V1 anymore?
Can't believe that the Visual Studio version matters (2015).
Can't find the reason why the test would fail like this and it takes me out of my mind.
Any suggestion will help.
Our development team uses Visual studio 2013 Pro along with TFS 2010. We need the following abilities for our continous integration / delivery initiative:
1. Automatically build our web project on every checkin.
2. Automatically deploy green builds to development and/or test environments.
3. Provide build/deployment notifications via email.
Extra Credit: Run unit tests with each build.
We don't have a budget, the reason Jenkins is appealing. The setup is new to me and not sure how the pieces fit and what other tools are needed to preform the tasks above?
If you have MSDN subscriptions, you can use TFS for free. TFS 2015 can provide all these features. TFS 2010 can provide most of this as well.
Use a gated checkin or CI build definition in TFS (2010 and 2015) to automatically build on checkin and run tests. The build definition has the option to run tests as part of the build.
Use MSDeploy to package up your components into an automated package. Most people don't know you can use it for more than web applications. (For example, databases)
TFS also provides email alerts on many events.
Regarding automating the deployments, we used separate TFS (in 2010) build definitions to trigger deployments to each environment. Now we use TFS 2015 which supports release definitions that adds many more options.
We have come across an issue where TFS build fails to trigger coded UI tests, based on how the Test Settings file is specified in the build definition.
We have been unable to get CodedUI test project to run. The goal is to have it run on a separate machine (virtual machine). We have configured the test controller and agent on the VM. We have also set it to run as interactive process. When we run build it partially succeeds and the following test error is reported
The failure message returned is as below:
MsTestAdapter failed to discover tests in class ‘MVC_Test.MVCTest_CUI’ of assembly ‘C:\Builds\1\Test_COE\MVCAppFramework\bin\MVC_Test.dll’. Reason Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=12.0.0.0. Culture =neutral, PublickeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified..
Setup / Environment Details:
Visual studio 2015 ultimate
TFS server 2015
Agents for Visual studio 2015(Test Controller and Test Agents)
Test agent is installed as interactive process for Coded UI tests
Your CodedUI tests were written in the 2013 version of VS. I believe you have two options:
Install the Agents for Visual Studio 2013 on your test agent
Use binding redirects to automatically use the newer version as described in this TFS 2010 to 2015 Upgrade: CodedUI Feature Test Builds blog post
UPDATE
You could try upgrading your CodedUI DLLS to the VS2015 versions (14.0.0.0).
It cannot find the dll libraries for Version 12.0 which provided by VS2013. Install VS2013 on the test agent and then retry.
I'm looking to get TFS to build and deploy to our manual and auto TEST environments.
manual TEST - overnight updates
auto TEST - continuous integration
Once deployed, I want TFS to deploy and run a set of web-integration tests (SpecFlow / Selenium / nUnit) on another server (acting pseudo client of the website).
I then want TFS to collate the results of these tests and report back.
What's the best approach to get this working?
You'd normally change the build process template to do what you want. Adding steps to deploy could be one of these customozations. Have a look at the Build Release Deploy build template from the Microsoft ALM Rangers which includes steps to publish the build to another machine.
You can also make use of WebDeploy and add a set of publish profiles to your solution. Team Build will happily execute those steps during build, if told to do so.
In TFS you can use a Test Agent to then execute tests from a remote machine and have the results added to the test results for your build.
Seeing that you'd like to use NUnit ad SpecFlow I suggest using TFS 2012 in combination with Visual Studio 2012, which have built-in support for executing 3rd party based tests. You'll need to install the NUnit adapter and put the assemblies for it in source control so that Team Build will pick them up.
Another few TFS questions if anyone can help even remotely :-)
We currently have TFS 2012 installed. We have 2 servers (both VMs running Windows Server 2008 R2 x64). One server is TFS with the application layer and database (running SQL Server 2008 R2) installed. The other server is our BUILD server with the build controller and build agent installed on it.
We can connect to TFS 2012 OK from our local developer PCs using Visual Studio 2012 Ultimate.
We then have various complex build definitions that compiles our code, runs unit tests, generates real-time sandcastle documentation and installs to our multi tiered, multi server TEST or LIVE environments using InstallShield (takes SSL and performs pre-install conditions etc)
We are using the UpgradeTemplate.xaml build process definition along with TFSBuild.proj file. FYI it was originally developed in TFS 2008 and has served us well hence why we don't WANT to upgrade to a modified defaulttemplate.xaml
We now want to include "post deployment tests" (aka smoking testing or system testing or end-to-end testing I believe) in the form of Coded UI tests.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an internative process and not a service). Is this correct?
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
1.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
If the coded ui test is used as a stand alone unittest in the unittest dll, mstest should pick up the coded ui tests. If the coded ui test is however an automation of a test case, an alternative is to use the TCM command line tooling to kick of the test case and thereby executing the coded ui test.
2.
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an interactive process and not a service). Is this correct?
Yes: You would need an interactive build agent.
No: the current build agent could also be set to interactive.
3.
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Yes: It works as the articles imply
No: The intellitrace collectors and logs are usually more useful than a video recording. The articles also forget to mention that your TFS server might grow out of proportions if not handled with care. http://geekswithblogs.net/terje/archive/2011/11/15/guide-to-reduce-tfs-database-growth-using-the-test-attachment.aspx
4.
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
No, there not the same and don't try to pretend that (like blend). The focus of the tools are different; MTM is focused on testing by a tester (functional, manual, exploratory), testing in visual studio is focused on testing as a developer (unittest, webtesting, automating tests). Visual Studio ultimate therefore can't edit and create test cases and suites. MTM can't create develop (coded) unittest tests. Installing visual studio ultimate also installs MTM, so you can use both if needed.