We have MTM tests running on Release build of our product (Desktop Application).
Now we want the same tests to run on two product builds: Beta and Release.
When a test run is initiated from MTM (or tcm), we need a way to pass a 'value' to the test run telling it which version/build of the product it needs to test. This 'value' will then be read in the test method and correct decision will be taken while the tests are executing (like installation path, test results file updates etc).
Is there any way to achieve this? in TFS or MTM?
Consider using Test Settings.
If you start an automated tests from MTM you can specify Test Settings to use when running this tests.
In "Advanced" part of Test Settings you can specify scripts to run on your environment before running the tests.
Create two scripts, one for Release and one for Beta version. These scripts could create a file with particular content, set an environment variable or do something else that can then be checked by your test, when it’s running.
Create two Test Settings, one for Release and one for Beta version and
set up appropriate script to run for each Test Settings.
Use one of these Test Settings when starting tests.
This way you could pass information to your test.
We also faced similar problem in our project. We decided to modify the build definition template to take product build type (Beta or RTM or Release) as an input parameter. Using this value during TFS build, we can either update the TFS build name to reflect the product build type or create a file (xml) as part of TFS build process to contain this type detail.
See here for more detail on how to add Arguments and Parameters to build definition: http://www.ewaldhofman.nl/post/2010/04/27/Customize-Team-Build-2010-e28093-Part-2-Add-arguments-and-variables.aspx
Pls take a look at the below link, if it can be used to suit your needs.
http://blogs.infosupport.com/switching-browser-in-codedui-or-selenium-tests-based-on-mtm-configuration/
one question: Are you using Build-Deploy-Test flow to install the product on the environment or doing it any other way?
So, when you select to run a set of automated tests and pick the build from the drop down list this tells MTM which drop folder to go look in. So if your configuration is code, as it should be, then you can set this up to be automatic.
It is not possible to pass additional variables when you start a test run in MTM.
You could setup your tests to run from the Release Management tool instead. You would then be able to configure the environment however you like based on passed in veriables.
http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
Related
I have 2 executables, an app and an automation test app who will perform actions on the app. My automation tests are basically NUNIT tests who calls Chrome Web Driver.
Everything is hosted at TFS. In my build definition, i run sanity checks for every PR. I wanted to expand that. The automation tests are divided into many different categories (being Sanity a category). I've been seeing some stuff related to TFS Test Suits, and my idea was to make in a way whenever when someone makes a PR, he could choose some test category to run on that PR using that build. So in a easyer way of sayng, if my PR changes how 'blue buttons work' ill run the 'blue buttons test suit' on my pr.
Would using Test Suits be the best solution for this ? Has any1 done this or have any nice information on how to achieve this ?
Thanks for any responses !! Best regards !
You could be able to use the Visual Studio Test Agent Deploy and Run Functional tests steps in your build definition to run auto tests on build agents.
Associated test methods with test cases in Visual Studio.
Create a build definition to build your project and add the 2 steps I mentioned above. In the Run Functional Test step, select the test suites which contains those test cases in step1.
More details please refer this blog: Executing Automated tests in Build vNext using Test Plan, Test Suites
I have a situation where my project needs to be tested in x64, but now I'm adding Coded UI tests which need to run in 32 bits.
I have previously used the multi-configuration option to spread work across multiple agents
The problem is that adding another variable for the test settings file will cause my build to run all categories in both x64 and x86 test settings and I want to be able to choose which categories use which settings, how can I do that?
No, you can't. Seems you want to achieve only build specifically configuration and platform such as Debug|x86 , Release|x64.
You cannot specify unique combinations of variables. For now, vNext Build will always build all permutations of the variables specified.
You could add a uservoice here, TFS admin and PM will kindly review your suggestion.
We recently added Coded UI Tests to our solution. The tests complete successfully when ran through Test Explorer but when the code is checked in and a build is triggered, all the CUIT's fail (error message is below).
I have gone to each of the links in the error message. The first one details how to set up a test agent to run the tests. We don't really need tests on all our environments as we are setting up a lab environment to run them and adding the test agent would require me submitting a ticket which would probably take months to get a response back, not to mention walking the person through what needs to be done.
The second link is dead and I'm pretty sure I don't want to change the build to be interactive.
I am hoping there is an easy way to change the build definition (which I have access to) so that it can ignore all Coded UI Tests but still run the Unit Tests. Is this possible? Is there an easier way of going about this? Each set of tests have their own project file, all Coded UI Tests in one project and all Unit Tests in another project.
Thanks in advance.
Here is the error that is on all of the coded UI tests:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
If you are running the tests as part of your team build, you must also set up the build agent to run as an interactive process. For more information, see "How to: Configure and Run Scheduled Tests After Building Your Application" (http://go.microsoft.com/fwlink/?LinkId=254735).
One thing you can do is to group all the coded UI tests into a particular test category and exclude running those in the build. We do this with our "integration" tests and only choose to run our true unit tests for speed reasons.
See below link for how to use test categories:
https://msdn.microsoft.com/en-us/library/dd286683.aspx
Also see the below for how to edit your build definition to only run the tests you want:
Excluding tests from tfs build
You can use use some naming patterns and combine that with the Test case Filter option in the execute test step:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/
Something like:
FullyQualifiedName ~ .Ui.
This would require the Ui tests to have .Ui in the namespace
Using Visual Studio Online I created a test plan for a program that was written by a different company that my company uses. We have a specific set of tests that need to be tested before we accept a new version of this program. So when I edit the test plan I would like to be able to manually select a build by typing in say version "1.0.1.195". Then when a newer version comes out I can just type in a newer version and retest using those same tests. However when I go to select a build TFS is filtering against my builds for my code. Is it possible to do what I'm asking using TFS?
EDIT
To answer a few of the questions in the comments I'll be a bit more descriptive of what I am doing. A 3rd party company made a program we use to test some hardware. Every now and then there is an update to that software. Since a few of us use this program to test out the hardware we need to know that the software can be installed with little to no down time while upgrading. So we came up with a small set of tests that we run the program through to make sure that we can test reliably. Those tests were written in a Word document, so I put them into MTM. Although I make some software that is related to this their software depends on mine. I've not had to update my code for some time now. My overall intention is to use MTM to document my testing of this program.
Do you want to store the version of the 3rd party component along with the test result of the test run it was tested with on TFS?
That would be nice. My ultimate end game is to put the results of said test back into that Word Document and make that available to those who don't have MTM installed (which is everyone). This way when a new version of the software is updated I can just go into MTM reset all my tests back to active update the version number and retest.
The build you set up in Microsoft Test Manager (MTM) defines where is the drop location containing your tests, not the application under test (it can be different if you build your tests using another build).
That's why you only can select one of your builds for your code.
What you are talking about is deployment.
That means you have to make sure the right version of the 3rd party program is deployed to the environment the tests are running on.
EDIT
What you need is a Test Configuration
Here you can find a is a good explanation how to create one: Test Configurations - specifying test platforms
The idea in your use case would be as following
(below I'm using terms described in the article mentioned above):
Create a Configuration Variable where you will store the current version of the 3rd party program
Create a Test Configuration and add this variable to it.
Set this Test Configuration as default test configuration for your test plan.
Be aware, if your test plan already contains test cases you will have to add this Test Configuration to each Test Case manually since only new added Test Cases get it assigned automatically
If you get a new version of the 3rd party program you will:
Add the new version of the program to the values allowed for the Configuration Variable
Open the Test Configuration you are using and update the program's version to the new one.
Reset your tests and run them.
Doing so you:
store all versions you have tested so far in the Configuration Variable since you add the new one instead of overwrite the old one, so you get a kind of history.
store the last version you have tested in the Test Configuration.
That should meet you needs.
Additional suggestion
(has nothing to do with your question but with your use case)
Consider describing tests within your Test Cases instead of creating a Word document.
Here is a good place to start reading: How to: Create a Manual Test Case
The benefits would be:
You can run your manual tests using Test Runner provided by MTM
Doing so you will have all steps you have done stored by the Test Result, you can add comments to each step when executing that, etc.
You can still export the test description to a Word document using this MTM add-on: Test Scribe.
Using this add-on you can also create a report of your test execution.
Additionally, if you are going to use MTM more in your daily job I would recommend you this free e-book Testing for Continuous Delivery with Visual Studio 2012
I am new to TFSBuild but I have been able to create a build definition for my solution. I have a couple questions that help would be great with.
I have created 3 definitions - I wonder if this is the correct way to this.
A definition that fires for every check in, builds the code and runs unit tests only
A definition runs every night, builds everything, runs all unit and integration tests
A definition that I specifically use for deployments - so I specify the environment via a parameter and it builds the code, runs unit and integration tests and ms deploys it to specified environment, again via parameter
When I branch my code etc, I will have to create 3 definitions for each and this could become unmanageable. Feedback on this please?
Is it true that each definition has its own set of build numbers? Can they be shared?
My application is an MVC4 app with VS2012 IDE.
Sadly TFS Build doesn't have very good support for branches, yes this typically means you duplicate your build definitions for each branch. There are a few custom build process template that I've seen in the past which try to get around this, but nothing built in.
You could replace #2 with a windows scheduler task to run #1 with custom parameters, not the nicest solution, but could be extended to queue every build definition at midnight with the integration test flag.
For #3 instead of using a build definition to deploy I use an external tool called TFS Deployer, this allows me to use any build to deploy simply by changing the build quality of the build. Reducing the number of builds that need to be setup by 1 for each branch.
Each build definition has its own build numbers, there's no built in way to share however I believe this is set by the build definition, so you may be able to hack around it somehow.