I've just literally hit a brick wall trying to configure F# 4.5.0.0 under VS 15.8.1 to run xUnit tests. Here is the repo with the issue: https://github.com/kkkmail/fSharp-xUnit .
When I try to run all tests, VS comes back with this message: No test is available in C:\GitHub\fSharp-xUnit\DummyApp\DummyApp\bin\Debug\DummyApp.exe C:\GitHub\fSharp-xUnit\DummyApp\Tests\bin\Debug\Tests.dll C:\GitHub\fSharp-xUnit\DummyApp\DummyApp\DummyApp.fsproj C:\GitHub\fSharp-xUnit\DummyApp\Tests\Tests.fsproj. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Both the "DummyApp" and tests were created from scratch using VS and I cleared all temp files and all nuget caches prior to creating the solution as advised in various places on the web.
TL;DR Add xunit.runner.visualstudio to your test project.
For discovering tests Visual Studio relies on its test runner. This test runner can discover / run all tests for testing frameworks which implement the corresponding adaptor interfaces ITestDiscoverer and ITestExecutor. For xUnit, these interfaces are implemented in VsTestRunner which is published in xunit.runner.visualstudio.
See also the xUnit docs about running tests in VS
Related
I recently started to work with TFS2018 instead of TFS2017.
I had a perfect build with my TFS2017 that used Test Agents.
From TFS2018 we don't have anymore Test Agents but Build Agents.
I managed to upload a Build Agent that run my tests but the tests which uses data source from the test in TFS Tests can't access the data and fail (It says that that the machine has a missing .Net Framework and to check in the machine.config but the machine.config has the .Net Framework provider. The coded test is connected to the test in TFS with Work Item and matching test number).
Tests that don't use data passes (the build works fine).
Running the tests on local - All the tests passes (even those with data).
And I wonder - does TFS2018 not supports MSTest V1 anymore?
Can't believe that the Visual Studio version matters (2015).
Can't find the reason why the test would fail like this and it takes me out of my mind.
Any suggestion will help.
I am a newbie to TFS Build Pipeline. Is there any difference between Unified Build Agent vs Unified Test Agents in TFS 2018 Context?
What are Unified Agents and how they help to minimize effort and Build machines pool? Any Step by Step Guide.
PS: I have setup simple build definition and it is compiling the code. For testing, the build output gets a copy on multiple dedicated machines where the test runs.
I was doing some R & D and found that there is a concept Unified Agent where you are not required to copy Build output to multiple dedicated machines and run tests there.
Unified Agents do this itself without copying build output and run tests without dedicating machines. You just need to mention VMs (Test Machines) in a Machines Pool where it automatically pics it up.
It seems you have confusion about Visual Studio Test task. With this task, you can run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. In version 2.* of Visual Studio Test task, tests can be distributed on multiple agents.
What's new in version 2.*
Run tests using agent phase: Unified agent across Build, Release and Test allows for automation agents to be used for testing
purposes as well. You can distribute tests using the multi-agent
phase setting. Multi-config phase setting can be used to replicate
tests in different configurations. Check this blog for more information: https://blogs.msdn.microsoft.com/devops/2017/03/26/vstest-task-dons-a-new-avatar-testing-with-unified-agents-and-phases/
Test Impact Analysis: Automatically select and run only the tests needed to validate the code change.
Use the Visual Studio Test Platform Installer task to run tests without needing a full Visual Studio installation.
My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.
Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.
If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?
Here is our pattern matching for our SpecFlow test project:
And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests:
UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:
Test method
AccessTheNDCHomePage threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)
After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?
If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.
The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.
In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.
Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.
if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.
So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.
EDIT
It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.
Here were some of the issues and solutions we found to set up SpecFlow on our build server:
The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue
WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.
I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.
The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.
https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive
Another few TFS questions if anyone can help even remotely :-)
We currently have TFS 2012 installed. We have 2 servers (both VMs running Windows Server 2008 R2 x64). One server is TFS with the application layer and database (running SQL Server 2008 R2) installed. The other server is our BUILD server with the build controller and build agent installed on it.
We can connect to TFS 2012 OK from our local developer PCs using Visual Studio 2012 Ultimate.
We then have various complex build definitions that compiles our code, runs unit tests, generates real-time sandcastle documentation and installs to our multi tiered, multi server TEST or LIVE environments using InstallShield (takes SSL and performs pre-install conditions etc)
We are using the UpgradeTemplate.xaml build process definition along with TFSBuild.proj file. FYI it was originally developed in TFS 2008 and has served us well hence why we don't WANT to upgrade to a modified defaulttemplate.xaml
We now want to include "post deployment tests" (aka smoking testing or system testing or end-to-end testing I believe) in the form of Coded UI tests.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an internative process and not a service). Is this correct?
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
1.
We can record these tests locally, add asserts etc and check them into TFS etc but my problem is how to invoke them from TFS Build at the end of our build sequence without upgrading to a defaulttemplate.xaml? We use the true build property to invoke our unit tests but this is run prior to deployment. What other property(s) do i need to set? is this at all possible?
If the coded ui test is used as a stand alone unittest in the unittest dll, mstest should pick up the coded ui tests. If the coded ui test is however an automation of a test case, an alternative is to use the TCM command line tooling to kick of the test case and thereby executing the coded ui test.
2.
I have googled/MSDN'd an believie we may need a third server with a test controller and test agent on it (that isnt locked with a screen saver, build service account has permissions and the build service is configured to run as an interactive process and not a service). Is this correct?
Yes: You would need an interactive build agent.
No: the current build agent could also be set to interactive.
3.
If we want to video record the tests as they are run have people found this useful? Does it function/configure as easy as articles seem to imply?
Yes: It works as the articles imply
No: The intellitrace collectors and logs are usually more useful than a video recording. The articles also forget to mention that your TFS server might grow out of proportions if not handled with care. http://geekswithblogs.net/terje/archive/2011/11/15/guide-to-reduce-tfs-database-growth-using-the-test-attachment.aspx
4.
Can visual studio 2012 Ultimate do everything that Microsoft Test Manager 2012 does? and vice-versus? I haven't used this and wonder if this a testers tool only really?
No, there not the same and don't try to pretend that (like blend). The focus of the tools are different; MTM is focused on testing by a tester (functional, manual, exploratory), testing in visual studio is focused on testing as a developer (unittest, webtesting, automating tests). Visual Studio ultimate therefore can't edit and create test cases and suites. MTM can't create develop (coded) unittest tests. Installing visual studio ultimate also installs MTM, so you can use both if needed.
I have NUnit 2.5.10 and Specflow 1.8.1 installed with VS2010.
Right click on .feature file in VS, there is an item "Debug Specflow Scenarios" in context menu. But I always get "Could not find matching test runner". In Tools/Options/Specflow, there is no test runner option for NUnit.
Am I missing something?
Thanks in advance. :)
I don't know if NUnit has a seperate GUI for running tests as i use MSTests but this should be how u do it.
Setup a config file in your Test Project to tell Specflow what Test framework to use.
https://github.com/techtalk/SpecFlow/wiki/Configuration
The Tools > Options > Specflow > Test Runner tool is the tool that executes the Tests. The fact that nothing is showing here seems like the test runner is not installed. For example i have several, MSTest, Resharper, SpecRun, i normally leave it on Auto and just run the test from the test tool i want to use. I ran installers for these though which should add any VS extensions
You could try using the VS runner, Test > Windows > Test View
VS will just treat a Specflow test as a NUnit as the generated code behind is just an NUnit test (when configured that way). So i think the real question is how to debug an NUnit test in Visual Studio. A quick search suggested attaching the debugger to the NUnit test runner process.
Like Ryan I also have no test runner specifically for Nunit. However since I'm also running resharper I get additional resharper runners, and that is what Auto uses to run it.
Have you actually got nUnit installed on your machine? In which case you can test how successful the process is without VS and Specflow-addin getting in the way. If not, well installing may give you the runner you need.
Alternatively, I use nCrunch to automatically run all my tests. I've had it running nunit/specflow for the last few months on multiple solutions with no issues. No need to right click at all then. :-)
SpecFlow itself does not provide a test runner and if you want to right click on the .feature and run it, you'll have to re-configure SpecFlow to use MSTest, which should use built-in VS runner.
If you want to use NUnit, then additional tools are required. I would recommend Resharper, but it's not free. Check out this article for more options to run NUnit tests from VS.
Good day. To debug i do the following :
Put a break point on the scenario line that you are interested in
Make sure that you have NUnit running
Attach the NUnit process
Go to tools
Select attach process
In the available processes select NUnit agent.exe
Click attach
Run the test you want to debug in NUnit
Hope that helps