I am using VS2017 with NUnit Test Adapter 3.12.0 and
nuget NUnit 3.10.1 framework: >= net45
nuget NUnit.ApplicationDomain 11.1.0
nuget NUnit.Runners 3.9.0
I want to run each test in a separate domain but when applying [RunInApplicationDomain] attribute to a test, the test can't be found in the VS Test Explorer. Removing the attribute then it can be found. Is there a workaround?
For instance, the test below can not be found.
namespace Foo
open NUnit.ApplicationDomain
open NUnit.Framework
[<TestFixture>]
type ThreeNodeTests() =
[<Test; RunInApplicationDomain>]
member __.Test1() =
Assert.Pass()
Related
I changed the project type I am packaging from .net framework v4.6 to .net standard 2.0, now the build definition is failing in Nuget packager step and I am getting this error message.
[error]The default XML namespace of the project must be the MSBuild
XML namespace. If the project is authored in the MSBuild 2003 format,
please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
to the element. If the project has been authored in the old
1.0 or 1.2 format, please convert it to MSBuild 2003 format.
After researching about this error I understand that the NuGet packer step doesn't work on SDK-based csproj formats.
What is the best alternative available? I found the same issue here, but I can not find the command pack within the available commands.
While the pack command does not appear in the dropdown, you can enter it manually into the field.
This is how I resolved this issue:
1- Add package metadata to .csproj file.
2- Go to project properties -> package and check "Generate NuGet package on build".
3- In Build definition add the following tasks:
a- NuGet Restore:
Set path to solution.
Select Feeds in my NuGet.config as Feeds to use.
Set path to Nuget.config (Usually project root "src\nuget.config").
b- Visual Studio Build:
Set path to solution.
Platform: something like $(BuildPlatform).
Configuration: something like $(BuildConfiguration).
c- Copy and Publish Build Artifacts:
In contents enter *.nupkg.
Set Copy Root, Artifact name, and Artifact type.
I have code base in which the unit tests are written with Machine Specifications leveraging the nuget based test runner Machine.Specifications.Runner.VisualStudio, v2.10 to execute the tests. It works fine from the context of Visual Studio (2015 & 2017) and filtering by Trait works as expected. However, when using the Test Assemblies build step it does not seem to honor the category filter. Is there something special with how the TFS build agent runs the test adapter compared to visual studio?
Example Test
[Subject(typeof(RetrieveConfiguration))]
[Tags(Categories.Manual)]
public class When_hitting_the_general_services_uri : SpecificationContext
{
private static TestResult result;
Establish context = () =>
{
IServiceInfo serviceInfo = Substitute.For<IServiceInfo>();
serviceInfo.Url = "";
environment.GetService("Confiugration", "Retrieve").Returns(serviceInfo);
x509Manager.LoadFromSignature(ValidSignature).Returns(LoadFromMachine(ValidSignature));
};
Because of = () => error = Catch.Exception(() => result = sut.Execute(new Uri("https://myproductionuri/retrieve"), environment));
It should_have_the_succeeded = () => result.Result.ShouldEqual(StepResult.Success);
}
Build Step Configuration
Build Log
...
2017-08-10T20:49:44.8717852Z ##[debug]Calling Invoke-VSTest for all test assemblies
2017-08-10T20:49:44.9655216Z Working folder: E:\B39\BA78\WS\18
2017-08-10T20:49:44.9655216Z Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "E:\B39\BA78\WS\18\s\Src\Test\Verifier.Reporting.Azure.Storage.Spec\bin\Release\Verifier.Reporting.Azure.Store.Spec.dll" /TestCaseFilter:"TestCategory=ContinuousIntegration" /EnableCodeCoverage /logger:trx /TestAdapterPath:"E:\B39\BA78\WS\18\s"
2017-08-10T20:49:45.1999042Z Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
2017-08-10T20:49:45.1999042Z Copyright (c) Microsoft Corporation. All rights reserved.
2017-08-10T20:49:45.5592884Z Starting test execution, please wait...
2017-08-10T20:49:56.8721150Z Information: Machine Specifications Visual Studio Test Adapter - Executing tests in E:\B39\BA78\WS\18\s\Src\Test\Verifier.Reporting.Azure.Storage.Spec\bin\Release\Verifier.Reporting.Azure.Store.Spec.dll
2017-08-10T20:50:01.5285749Z Passed Verifier.Reporting.Azure.Store.Spec.When_publishing_a_report.should_have_succeeded
...
Update 8/25 - added the requested screen shots and feedback
Test Explorer without filtering
Notice there are 16 total tests, the indicates ones starting with when hitting are integration tests which are not expected to run within the context of the build agent.
Test Explorer with filtering on Category
The total number of tests has decreased from 16 to 14. Since the test did not have the requested tag it was dropped from the test run.
Running vs2015 vstest.console.exe
As for running the test outside of visual studio, it would appear that the test runner is experiencing issues loading the test adapter on my dev machine, whereas the adapter runs fine in Visual Studio and on the build agent.
The vstest task is just using vstest.console.exe to execute tests. The test Filter Criteria in TFS VStest task works the same way as the console option /TestCaseFilter of vstest.console.exe.
"TestCategory=ContinuousIntegration"
By above, I did not see such TestCategory name in your code, if we run the test by using command line(vstest.console.exe), we have to specify the matched name, which means at least we name a TestCategory attribute above your test method code
i.e, my test method code :
[TestCategory("nine"), TestMethod()]
public void TestMethod1()
{
Assert.AreEqual(1, 1);
}
I need run it by using code below in command line:
Vstest.console.exe UnitTestvstsada.dll /TestCaseFilter:TestCategory=nine
It will filter the test successfully and will get the same result in TFS build.
As for filter in Test Explorer, there is no such option to filter the test. Only a configure continuous integration which actually is not a filter. If you don't mind, please kindly show how did you using the filter trait:ContinuousIntegration in the test of test run via visual studio.
Afraid using the filter trait:ContinuousIntegration that is not equivalent to TestCategory=ContinuousIntegration in the TFS2015 build agent.
I am running my test in TFS (Nunit plus Visual Studio with Adapter) and I have set the build definition as below
Build succeeds but no test result file was generated
Does TFS writes this Xml file ?
Log
2017-02-08T08:08:40.8151428Z Executing the powershell script: D:\A1\agent\tasks\PublishTestResults\1.0.20\PublishTestResults.ps1
2017-02-08T08:08:41.0963795Z ##[warning]No test result files were found using search pattern 'D:\A1_work\1\s**\TEST-*.xml'.
If the Nunit plus Visual Studio with Adapter means you have two test steps: one for unit tests and the other one for vs tests.
Please also add two "Publish Test Results" step one for Nunit format.
Also run your test manually on the build server to see if test result file .trx generated on the machine.
I'm possibly in over my head here, but I've been asked to set up a scheduled Team Foundation Build for our team's branch and then after the build completes for our automated tests to be executed using NUnit.
I've had a look at a few online tutorials on setting up the build definition in TFS, but I can't seem to figure out how to call NUnit after the build is successful. I was expecting to see or find some kind of "run this command line on success" option somewhere; the best I could find is "Pre/Post-test script path", but that's related to tests like **\*test*.dll;**\*test*.appx and I'm not sure what that is.
Just knowing what to Google for would be a help, as I am at a loss now.
If you use XAML build:
You can either install the NUnit Test Adapter NuGet package in the unit test project. Or you can check the assemblies into the Build Controller's Custom Assemblies Path.
Useful article:
https://blogs.msdn.microsoft.com/visualstudioalm/2013/06/11/part-3-unit-testing-with-traits-and-code-coverage-in-visual-studio-2012-using-the-tfs-build-and-the-new-nuget-adapter-approach/
https://www.codit.eu/blog/2015/03/18/continuous-integration-with-javascript-nunit-on-tfsbuild-part-3-of-3-/
If you use new task based build:
You can add the NUnit Test Adapter NuGet package to your solution, and specify the path of NUnit Test Adapter NuGet package in the Path to Custom Test Adapters field in VSTest task. Check the screenshot below:
Useful article:
http://bartwullems.blogspot.sg/2015/10/team-foundation-server-2015enable-nunit.html
I had this working on a previous project and now on a new project I've setup SpecFlow, got it generating tests from my feature file but I can't run the tests from the feature file and instead have to go to the code behind to run the tests. I've also installed the VS extension "Spec Flow for Visual Studio". What can I try?
as Greg suggested the first thing to check is that your config is set up correctly for ms test. you basically need this:
<specflow>
<unitTestProvider name="MSTest"/>
</specflow>
Also worth checking your generated feature.cs tests to see what unit test language they are in