I recently realized that not all of my tests are being run in the build server. On the TFS Build config I have set to run all dlls and check for folders recursively.
Test Sources Spec: **\**.dll
I want to TFS build output folder and it seems all the libraries that has test are there. I am not sure what I am doing wrong.
I would like to know why it is not firing all my tests?
UPDATE:
Here is my build settings.
Related
I currently have an on-premises TFS 2017, a TFS build and a test that fails on the build agent but not when running locally. I heard about historical debugging but I couldn't find instructions on how to enable it on a TFS build, if that's even possible.
Is it possible to configure my build so that it generates an intellitrace file for me to debug the test run afterwards?
For intellitrace which related to the old testsettings file and could add some variables to configure this.
You could also manually enable such option in the VS on your build agent.
Also find a link mentioned it seems also work with Msbuild arguments. Even not documented anywhere in Msbuild command doc. I wonder whether IntelliTrace is still running even though it's not explicitly specified to do so
However, looks like you just want to find the root cause why your tests failed on tfs build even it passed locally.
To narrow down the issue, suggest you directly remote to your build agent. Instead of through TFS build, manually build your project, run the tests in the Visual Studio on build agent.
It will come out whether it's an environment issue. You must make sure it's the same environment for build agent and your local.
TFS is just using build service account to invoke msbuild/vstest command in build agent to run build/test.
Also make sure build service account have appropriately permission if your tests required some permission.
We have the following setup for unit and functional testing in TFS (under Build Definitions in TFS)
Install our program into a clean VM.
Deploy test agent (default vstf_testagent.exe)
Run tests
Some of the tests are written in NUnit3 and the C++ test in Google Tests. So, they are not seen by the default test runner.
In Visual Studio, the test adapters work great to run the tests.
But, I can't seem to figure out how to install these test adapters to vstf_testagent.exe on the virtual machine so that it sees and runs all the tests.
I found this page on how to install agents but it is with using Visual Studio.
You can install the test adapters to your project via Nuget and add a Nuget restore task in your build definition to restore the packages. "Run Functional Tests" task will search the test adapters in "packages" folder automatically.
Test Adapters: There is no explicit field to specify Test Adapter path in the task. The task automatically searches for "packages"
directory that exists in the same folder as the .sln file (nuget
restored directory structure). If your adapters are in a different
directory or you did not copy over the source files, use a runsettings
file with TestAdaptersPaths as described at
https://msdn.microsoft.com/en-us/library/jj635153.aspx
If the test adapters is in other folder, you can create a runsettings file and run the test with it.
Refer to this link for details: Visual Studio Test using Test Agent Task.
First, you need to install Visual Studio on your VM.
After that the directly way to make NUnit adapters available is downloading them from Visual Studio Gallery, unzip the .vsix extension file, then copy all the extension file in the folder just as the link mentioned in your question.
This operation should be done in every machine where you deployed Test Agents that will be used to run build with nunit tests. This can be annoying if you have lots of agents, if you prefer, Test Runner Task has an options to specify the path where the agent can find needed test runners.
You can also automate all the process by adding Nunit3 Test Adapter nuget package just as Eddie suggest to your test project, adding a standard Nuget Reference. Detail ways you can refer this blog: nUnit and Jasmine.JS unit tests in TFS/VSO vNext build
Nuget Package
NUnit 3 Test Adapter for Visual Studio 2012/13/15
It is also possible to copy the .dlls of your testadapter that you want to use directly from the nuget-package to the folder C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensionsof your computer where you want to run your tests.
Also see:
http://www.codewrecks.com/blog/index.php/2016/06/04/running-unit-tests-on-different-machine-during-tfs-2015-build/
My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.
Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.
If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?
Here is our pattern matching for our SpecFlow test project:
And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests:
UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:
Test method
AccessTheNDCHomePage threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)
After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?
If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.
The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.
In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.
Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.
if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.
So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.
EDIT
It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.
Here were some of the issues and solutions we found to set up SpecFlow on our build server:
The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue
WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.
I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.
The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.
https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive
We're using TFS Build and are having the TFS Build process run tests through a Test Controller on a Test Agent on a remote machine. We do this via a testsettings file in which we specify the test controller. This testsetting file is specified as an argument in the default TFS 2012 Build process template.
Everything works fine. However, our build takes about an hour. In the event that a test should fail because of an environmental issue (so in the event where it fails only on the test agent and not on our Dev machines), I'd like to be able to re-run the tests and diagnose the environmental issue that happened. However I can't seem to figure out how to do that WITHOUT re-running the entire build. Once the environmental problem is fixed, I'd like to be able to re-run the tests again to make sure everything is ok. I'd like to do all this without re-running the entire build. Ideally when I re-run my tests they should run in as similar a configuration as possible as to when TFS Build runs the tests.
Is there a way to do this? Is using MSTest command line an option to re-run the tests? If so, how do I know which arguments to pass to it? Ideally i'd want the same arguments TFS passed to the QTAgent process which ran the tests?
Create build specific, based on sequential workflow specific whose you delete from designer WWF all activities that don't need.
I suggest theses steps : 1. Select link 'Create Build Definition' 2.Create Process Template by copy pasting on Source Control, 3.Open your copied Workflow on designer 4.Delete all activities that you don't need. 5.Finish by associating this workflow to build definition (Select Process Tab)
The easiest way to prevent the whole build from running is to set "Clean Workspace" to None.
That means if you want to re-run the tests you can just queue the existing build definition manually, change to the Parameters tab and set Clean Workspace to None before click the "Queue" button.
In this case existing build outputs remain as well as source code files that have not changed since the most recent build (by performing a tf get without the /all switch).
It's probably not exactly what you want because if there were some check-ins since the most recent build the changed source files will be fetched and built.
But if it is not important for you to re-run the tests using exactly the same build output each time it could be a good solution for you since it usually reduces build time significantly.
One way for you to possibly do this is:
Open the build that had failed tests in Visual Studio
Scroll down to the test runs section.
Click on the failed test run. The Test Results window should open.
Select the failed tests you want to rerun.
click the "Run checked tests" button
Supposedly this should work: vstest to tfs logger. I can verify that the test are run and published to our tfs, but I could not see the test result on the tfs build page only using the link that is produced (the one with mtm as protocol)
We are working on fine tuning the automated build process using TFS 2010. During development, we use a special configuration to run our unit tests. During the build, does it make sense/possible to do the following:
Compile the application in UnitTest configuration and run unit tests. If all pass, run the build in release configuration and deploy.
The reasoning behind the above suggestion is that we are using config file transformation for some settings. However, I can make the build server match those, without the need for a different set up. I also wonder if the above approach is supported in TFS build. I.E: How do you run two compilations in different configurations.
Or is the following approach better:
Compile the application in Release configuration and run unit tests. If all pass, deploy.
If you edit your build definition in Team Explorer and navigate to the Process page, you'll see an "Items to Build" parameter in the Required section. If you expand that parameter, you'll see a child parameter called "Configurations to Build." Clicking the ellipsis button for that property invokes a dialog where you can specify the platforms and configuration you want to build. By default, TFS will build the default platform and configuration only. You can, however, specify as many configurations as you like.
Regarding which configuration to test and deploy, I would personally lean towards the release version since. Make sure you're still generating symbols for that configuration and you should still get full stack trace information for any test failures. If you intend to deploy the release build, then that's probabaly the configuration you should be running your tests on.