I've configured a DevOps pipeline to execute my SpecFlow Selenium tests. Everything appears to work fine, except that all my test scenarios are skipped on the DevOps agent.
When I run the tests locally, using VS2019 test explorer, they run without issues.
I've tried enabling system diagnostics to see why they are being skipped but couldn't see anything that could be the cause.
Here's the YAML for the VSTest task:
steps:
- task: VSTest#2
displayName: 'VsTest - RetroSpectro.Tests.UI.Specflow.dll'
inputs:
testAssemblyVer2: |
**\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
!**\*TestAdapter.dll
!**\obj\**
!**\bin\**\ref\**
uiTests: true
runSettingsFile: RetroSpectro.Tests.UI.Specflow/Serial.runsettings
pathtoCustomTestAdapters: packages
runTestsInIsolation: true
testRunTitle: 'SpecFlow Tests'
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
failOnMinTestsNotRun: true
rerunFailedTests: false
... and the output from task execution:
2021-06-25T09:02:15.0841271Z ##[section]Starting: VsTest - RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:15.1180446Z ==============================================================================
2021-06-25T09:02:15.1180820Z Task : Visual Studio Test
2021-06-25T09:02:15.1181621Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2021-06-25T09:02:15.1182585Z Version : 2.170.1
2021-06-25T09:02:15.1183892Z Author : Microsoft Corporation
2021-06-25T09:02:15.1184298Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/test/vstest
2021-06-25T09:02:15.1184712Z ==============================================================================
2021-06-25T09:02:16.6157598Z SystemVssConnection exists true
2021-06-25T09:02:16.7413804Z SystemVssConnection exists true
2021-06-25T09:02:16.9135626Z Running tests using vstest.console.exe runner.
2021-06-25T09:02:16.9136452Z ======================================================
2021-06-25T09:02:16.9138232Z Test selector : Test assemblies
2021-06-25T09:02:16.9139517Z Test filter criteria : null
2021-06-25T09:02:16.9140779Z Search folder : D:\a\1\s
2021-06-25T09:02:16.9937367Z Action when minimum tests threshold not met : fail
2021-06-25T09:02:16.9938318Z Minimum tests expected to be run: 1
2021-06-25T09:02:16.9939834Z VisualStudio version selected for test execution : latest
2021-06-25T09:02:16.9943536Z Attempting to find vstest.console from a visual studio installation with version [16.0,17.0).
2021-06-25T09:02:16.9973962Z Run settings file : D:\a\1\s\RetroSpectro.Tests.UI.Specflow\Serial.runsettings
2021-06-25T09:02:16.9981237Z Run in parallel : false
2021-06-25T09:02:16.9984189Z Run in isolation : true
2021-06-25T09:02:16.9986763Z Path to custom adapters : D:\a\1\s\packages
2021-06-25T09:02:16.9991903Z Other console options : null
2021-06-25T09:02:16.9994735Z Code coverage enabled : false
2021-06-25T09:02:17.0001246Z Diagnostics enabled : false
2021-06-25T09:02:17.0008374Z SystemVssConnection exists true
2021-06-25T09:02:17.0368729Z Run the tests locally using vstest.console.exe
2021-06-25T09:02:17.0369728Z ========================================================
2021-06-25T09:02:17.0464264Z Source filter: **\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll,!**\*TestAdapter.dll,!**\obj\**,!**\bin\**\ref\**
2021-06-25T09:02:22.7152869Z SystemVssConnection exists true
2021-06-25T09:02:22.7157229Z [command]D:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.170.1\Modules\DTAExecutionHost.exe --inputFile D:\a\_temp\input_0dad9d50-d594-11eb-bac6-a972b6c3989e.json
2021-06-25T09:02:22.7158487Z ======================================================
2021-06-25T09:02:22.8592004Z ##########################################################################
2021-06-25T09:02:22.8592761Z DtaExecutionHost version 18.170.30112.1.
2021-06-25T09:02:23.7498265Z Starting TestExecution Model...
2021-06-25T09:02:26.9047814Z Result Attachments will be stored in LogStore
2021-06-25T09:02:26.9297128Z Run Attachments will be stored in LogStore
2021-06-25T09:02:26.9433340Z Result Attachments will be stored in LogStore
2021-06-25T09:02:27.1235011Z Result Attachments will be stored in LogStore
2021-06-25T09:02:27.1235834Z Run Attachments will be stored in LogStore
2021-06-25T09:02:27.1470266Z Provided settings file:
2021-06-25T09:02:27.1485829Z <?xml version="1.0" encoding="utf-8"?>
2021-06-25T09:02:27.1486598Z <RunSettings>
2021-06-25T09:02:27.1487139Z <SpecRun>
2021-06-25T09:02:27.1488252Z <Profile>Serial.srprofile</Profile>
2021-06-25T09:02:27.1488944Z <GenerateFeatureTrait>true</GenerateFeatureTrait>
2021-06-25T09:02:27.1490018Z </SpecRun>
2021-06-25T09:02:27.1490506Z </RunSettings>
2021-06-25T09:02:27.1785620Z Updated Run Settings:
2021-06-25T09:02:27.1788682Z <RunSettings>
2021-06-25T09:02:27.1788992Z <SpecRun>
2021-06-25T09:02:27.1789483Z <Profile>Serial.srprofile</Profile>
2021-06-25T09:02:27.1789897Z <GenerateFeatureTrait>true</GenerateFeatureTrait>
2021-06-25T09:02:27.1790256Z </SpecRun>
2021-06-25T09:02:27.1790560Z <RunConfiguration>
2021-06-25T09:02:27.1790961Z <BatchSize>1000</BatchSize>
2021-06-25T09:02:27.1791385Z <ResultsDirectory>D:\a\_temp\TestResults</ResultsDirectory>
2021-06-25T09:02:27.1791975Z </RunConfiguration>
2021-06-25T09:02:27.1792294Z </RunSettings>
2021-06-25T09:02:27.1979593Z **************** Starting test execution *********************
2021-06-25T09:02:27.2197434Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "#D:\a\_temp\y3jxhopis4l.tmp"
2021-06-25T09:02:27.4233968Z Microsoft (R) Test Execution Command Line Tool Version 16.10.0
2021-06-25T09:02:27.4235225Z Copyright (c) Microsoft Corporation. All rights reserved.
2021-06-25T09:02:27.4413296Z vstest.console.exe "D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll"
2021-06-25T09:02:27.4414588Z /Settings:"D:\a\_temp\kpbetfmovlq.tmp.runsettings"
2021-06-25T09:02:27.4415655Z /InIsolation
2021-06-25T09:02:27.4416622Z /Logger:"trx"
2021-06-25T09:02:27.4417670Z /TestAdapterPath:"D:\a\1\s\packages"
2021-06-25T09:02:29.1247592Z Starting test execution, please wait...
2021-06-25T09:02:30.0040388Z A total of 1 test files matched the specified pattern.
2021-06-25T09:02:33.0429462Z SpecFlow+Runner execution started
2021-06-25T09:02:33.0431615Z SpecFlow+Runner 3.9.7 in Framework net50 in x64 mode execution started
2021-06-25T09:02:33.0433131Z TestAdapter Location: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\TechTalk.SpecRun.VisualStudio.TestAdapter.dll
2021-06-25T09:02:33.0433910Z Current UserName: 'VssAdministrator', MachineName: 'WIN-RNFTU0DQEPC'
2021-06-25T09:02:33.3384985Z SpecRun: running tests in D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.3386699Z Output folder configured to "D:\a\_temp\TestResults" (default)
2021-06-25T09:02:33.3387477Z Profile: RetroSpectro.Tests.UI.Specflow
2021-06-25T09:02:33.4638846Z Log file path: D:\a\_temp\TestResults\RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.log
2021-06-25T09:02:33.4749622Z Checking activation
2021-06-25T09:02:33.4854960Z Activation check completed
2021-06-25T09:02:33.4878329Z Starting test run
2021-06-25T09:02:33.7105584Z Discovering target: Chrome
2021-06-25T09:02:33.7106665Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.8841939Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.8843042Z Discovering target: Edge
2021-06-25T09:02:33.8843914Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9067409Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9068593Z Discovering target: Firefox
2021-06-25T09:02:33.9069400Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9193883Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9195007Z Discovering target: IE
2021-06-25T09:02:33.9196501Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9272484Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9277522Z Discovering target: Safari
2021-06-25T09:02:33.9278613Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9352988Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:56.8250888Z Scenario: Follow button is not available in header section on Add Item page in AddEditItemFeature (target: Chrome) -> Skipped on thread #-1
...
<!-- other chrome tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8376857Z Scenario: Group description is available in header section on Edit Item page in AddEditItemFeature (target: Edge) -> Skipped on thread #-1
...
<!-- other edge tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8518611Z Scenario: Group status is available in header section on Add Item page in AddEditItemFeature (target: Firefox) -> Skipped on thread #-1
...
<!-- other firefox tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8587604Z Scenario: Group description is available in header section on Add Item page in AddEditItemFeature (target: IE) -> Skipped on thread #-1
...
<!-- other IE tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8648632Z Scenario: Title label is displayed on Add Item page in AddEditItemFeature (target: Safari) -> Skipped on thread #-1
...
<!-- other Safari tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8764035Z test run finished
2021-06-25T09:02:56.8765441Z publishing test results
2021-06-25T09:02:56.8766131Z Not publishing results.
2021-06-25T09:02:56.8766729Z test results published
2021-06-25T09:02:56.8767274Z generating reports
2021-06-25T09:02:56.9558142Z Starting external report generation process
2021-06-25T09:02:56.9590180Z Starting external program: "D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\SpecFlowPlusRunner\net461\SpecFlow.Plus.Runner.Reporting.exe" "C:\Users\VssAdministrator\AppData\Local\Temp\ca95d93f-da8f-41ce-9966-62cb15b332e9" in
2021-06-25T09:02:58.4457850Z Skipped Follow button is not available in header section on Add Item page [1 ms]
...
<!-- all tests were skipped in [< 1 ms] --!>
...
2021-06-25T09:03:00.2990896Z 'D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\SpecFlowPlusRunner\net461\SpecFlow.Plus.Runner.Reporting.exe "C:\Users\VssAdministrator\AppData\Local\Temp\ca95d93f-da8f-41ce-9966-62cb15b332e9"' took 3337.3233ms
2021-06-25T09:03:00.2992974Z RAA6AFwAYQBcAF8AdABlAG0AcABcAFQAZQBzAHQAUgBlAHMAdQBsAHQAcwBcAFIAZQB0AHIAbwBTAHAAZQBjAHQAcgBvAC4AVABlAHMAdABzAC4AVQBJAC4AUwBwAGUAYwBmAGwAbwB3AF8AUgBlAHQAcgBvAFMAcABlAGMAdAByAG8ALgBUAGUAcwB0AHMALgBVAEkALgBTAHAAZQBjAGYAbABvAHcAXwAyADAAMgAxAC0AMAA2AC0AMgA1AFQAMAA5ADAAMgAzADMALgBoAHQAbQBsAA==
2021-06-25T09:03:00.2994487Z Successfully generated reports.
2021-06-25T09:03:00.3024068Z Successfully generated reports
2021-06-25T09:03:00.3157341Z Result: test framework error: At least one test thread aborted.
2021-06-25T09:03:00.3159973Z Total: 185
2021-06-25T09:03:00.3160575Z Succeeded: 0
2021-06-25T09:03:00.3161111Z Ignored: 0
2021-06-25T09:03:00.3161627Z Pending: 0
2021-06-25T09:03:00.3163981Z Skipped: 185
2021-06-25T09:03:00.3164566Z Failed: 0
2021-06-25T09:03:00.3165688Z Execution Time: 00:00:22.9059275
2021-06-25T09:03:00.3166798Z Report file: file:///D:/a/_temp/TestResults/RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.html
2021-06-25T09:03:00.3168312Z Adding attachments to VSTest
2021-06-25T09:03:00.3169047Z adding Test Execution Report 'D:\a\_temp\TestResults\RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.html'
2021-06-25T09:03:00.3283768Z Log file: file:///D:/a/_temp/TestResults/RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.log
2021-06-25T09:03:00.3294371Z SpecFlow+Runner execution finished
2021-06-25T09:03:00.6411255Z Results File: D:\a\_temp\TestResults\VssAdministrator_WIN-RNFTU0DQEPC_2021-06-25_09_02_58.trx
2021-06-25T09:03:00.6488918Z Attachments:
2021-06-25T09:03:00.6490475Z C:\Users\VssAdministrator\AppData\Local\Temp\0erzmwe3.ry2\RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.log
2021-06-25T09:03:00.6494999Z C:\Users\VssAdministrator\AppData\Local\Temp\0erzmwe3.ry2\RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.html
2021-06-25T09:03:00.6813251Z Test Run Successful.
2021-06-25T09:03:00.6814110Z Total tests: 185
2021-06-25T09:03:00.6814725Z Skipped: 185
2021-06-25T09:03:00.6826934Z Total time: 30.4837 Seconds
2021-06-25T09:03:00.7194885Z Vstest.console.exe exited with code 0.
2021-06-25T09:03:00.7196364Z **************** Completed test execution *********************
2021-06-25T09:03:00.7398059Z Test results files: D:\a\_temp\TestResults\VssAdministrator_WIN-RNFTU0DQEPC_2021-06-25_09_02_58.trx
2021-06-25T09:03:01.3258001Z Created test run: 1014788
2021-06-25T09:03:01.3261282Z Publishing test results: 185
2021-06-25T09:03:01.6120517Z Publishing test results to test run '1014788'.
2021-06-25T09:03:01.6165436Z TestResults To Publish 185, Test run id:1014788
2021-06-25T09:03:01.6272324Z Test results publishing 185, remaining: 0. Test run id: 1014788
2021-06-25T09:03:02.2543932Z Published test results: 185
2021-06-25T09:03:02.2556259Z Publishing Attachments: 3
2021-06-25T09:03:03.5514586Z ##[error]The specified minimum number of tests 1 were not executed in the test run.
2021-06-25T09:03:04.2147578Z Execution Result Code 1 is non zero, checking for failed results
2021-06-25T09:03:04.3270302Z Completed TestExecution Model...
2021-06-25T09:03:04.3691332Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2021-06-25T09:03:04.3697400Z ##[error]Error: The process 'D:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.170.1\Modules\DTAExecutionHost.exe' failed with exit code 1
2021-06-25T09:03:04.3703872Z ##[error]Vstest failed with error. Check logs for failures. There might be failed tests.
2021-06-25T09:03:04.3933532Z ##[section]Finishing: VsTest - RetroSpectro.Tests.UI.Specflow.dll
It looks like the test thread is being aborted but I have no idea why.
The .runsettings is relatively simple:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<SpecRun>
<Profile>Serial.srprofile</Profile>
<GenerateFeatureTrait>true</GenerateFeatureTrait>
</SpecRun>
</RunSettings>
The specflow profile is:
<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
<Settings projectName="RetroSpectro.Tests.UI.Specflow" />
<Execution retryFor="Failing" stopAfterFailures="0" testThreadCount="1" testSchedulingMode="Random" retryCount="2" />
<Environment framework="Net5.0" platform="x86" testThreadIsolation="AppDomain" apartmentState="STA"/>
<TestAssemblyPaths>
<TestAssemblyPath>RetroSpectro.Tests.UI.Specflow.dll</TestAssemblyPath>
</TestAssemblyPaths>
<VSTest testRetryResults="Unified" passRateAbsolute="1"/>
<Targets>
<Target name="Chrome">
<Filter>!#Parallel</Filter>
<DeploymentTransformationSteps>
<EnvironmentVariable variable="RUNNER_TARGET" value="chrome" />
</DeploymentTransformationSteps>
</Target>
<Target name="Edge">
<Filter>!#Parallel</Filter>
<DeploymentTransformationSteps>
<EnvironmentVariable variable="RUNNER_TARGET" value="edge" />
</DeploymentTransformationSteps>
</Target>
<Target name="Firefox">
<Filter>!#Parallel</Filter>
<DeploymentTransformationSteps>
<EnvironmentVariable variable="RUNNER_TARGET" value="firefox" />
</DeploymentTransformationSteps>
</Target>
<Target name="IE">
<Filter>!#Parallel</Filter>
<DeploymentTransformationSteps>
<EnvironmentVariable variable="RUNNER_TARGET" value="ie" />
</DeploymentTransformationSteps>
</Target>
<Target name="Safari">
<Filter>!#Parallel</Filter>
<DeploymentTransformationSteps>
<EnvironmentVariable variable="RUNNER_TARGET" value="safari" />
</DeploymentTransformationSteps>
</Target>
</Targets>
</TestProfile>
Any ideas why the tests are being skipped or what I can try to find out?
Edit:
I noticed that in a previous (working) run, VSTest was running scenarios in thread #-0. After a crash it started logging the "skipped in thread #-1" messages.
Related
I have some xUnit tests running as part of a build and I need to post the results back to TFS 2010 so they show up in the build summary/log like normal tests would.
I execute the tests with the -nunit argument so that the output is an NUnit xml and not a xUnit one. I'm trying to use NUnit4TeamBuild to publish the results xml back to TFS and it looks like it's working but there's no test results anywhere at the end.
This is the detailed log for that step:
Publish xUnit Results
NUnitTfs.exe -n results.xml -t Project -b "Build_20171221.8" -v 2010
Loading NUnit Output from results.xml...
Transforming NUnit output to .trx file...
Adding build information to .trx file...
Publishing .trx file...
Checking test run does not exist...
Reading users identity...
Creating test run...
Creating test results...
Updating test run with details...
Uploading test run details...
Uploading assemblies...
Updating test results with details...
Uploading test result details...
The build finishes with 0 errors and 0 warnings but also says "No Test Results" in the build summary and there are no test results anywhere.
I have no idea why the tests aren't publishing.
Here's an example of the XML that it's trying to publish (note that I've changed the file paths and shortened the stack trace):
<?xml version="1.0" encoding="utf-8"?>
<test-results name="Test results" errors="0" inconclusive="0" ignored="0" invalid="0" not-run="0" date="2017-12-21" time="08:39:57" total="1" failures="1" skipped="0">
<environment os-version="unknown" platform="unknown" cwd="unknown" machine-name="unknown" user="unknown" user-domain="unknown" nunit-version="xUnit.net 2.3.1.3858" clr-version="64-bit .NET 4.0.30319.34209 [collection-per-class, non-parallel]" />
<culture-info current-culture="unknown" current-uiculture="unknown" />
<test-suite type="Assemblies" name="C:\Tests.DLL" executed="True" success="False" result="Failure" time="57.41">
<results>
<test-suite type="Assembly" executed="True" name="C:\Tests.DLL" result="Failure" success="False" time="57.410">
<results>
<test-suite type="TestCollection" executed="True" name="Tests" result="Failure" success="False" time="48.992">
<results>
<test-case name="Google_Search_Test" executed="True" result="Failure" success="False" time="48.9917024">
<failure>
<message>System.Exception : Test Failed. Exception has been thrown by the target of an invocation.</message>
<stack-trace>at Tests.Fixtures.Fixture.FormatException(Exception e) in... etc. </stack-trace>
</failure>
</test-case>
</results>
</test-suite>
</results>
</test-suite>
</results>
</test-suite>
</test-results>
TFS 2010 uses a proprietary results format with a “.trx” extension to publish results so we need to transpose the NUnit results before sending back through web services.
You can use NUnitTFS for xUnit, just try the workflow template NUnitAndPublishTemplate.xaml which provided by Ian Battersby.
Refer to Ian Battersby's blog for details : Running and publishing NUnit test results into TFS 2010
I have built a simple UITesting framework for one of my apps built in Xcode. 2 of the tests pass and one purposefully fails:
XCTAssertTrue(false)
I am using the Jenkins Xcode plugin and am using the post-built action: Publish JUnit test result report.
Jenkins successfully launches the simulator and runs all 3 tests. It also successfully picks up the failure in the logs:
Failing tests:
-[LightAlarmUITests testFailingTest()]
** TEST FAILED **
However, the Test Results Analyzer (plugin installed) shows all 3 tests are passing. When I inspect the test-results/*.xml file I see the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<testsuite failures="1" errors="0" hostname="Charlies-MacBook-Pro.local" name="LightAlarmUITests" tests="3" time="21.0" timestamp="2016-12-20T16:24:33.125Z">
<testcase classname="LightAlarmUITests" name="testFailingTest" time="6.372"/>
<testcase classname="LightAlarmUITests" name="testShowSettingsPage" time="7.167"/>
<testcase classname="LightAlarmUITests" name="testShowSoundAlarmsPage" time="7.594"/>
</testsuite>
As you can see in the xml, it is not marking individual test cases as pass or fail, but instead marking a failure against the entire test suite.
Does anyone know how to mark pass/fails against individual test cases?
There is a pull request open to fix this issue.
https://github.com/jenkinsci/xcode-plugin/pull/75
The reason behind this is Xcode 8 changed the way of XML of UI test report. JUnit plugin is not able to parse this XML to generate a test report. I have used XCPretty mediator that converts Xcode 8 XML report into JUnit supported XML. You can find more details at https://blog.talentica.com/2017/04/04/use-xcode-8-with-jenkins/
When I run TFS 2013 with sonar I get the following error: No ProjectInfo.xml files were found. Check that the analysis targets are referenced by the MSBuild projects being built.
Message: TF270015: 'SonarQube.MSBuild.Runner.exe' returned an unexpected exit code. Expected '0'; actual '1'.
I use :
sonarqube-5.1.zip
sonar-csharp-plugin-4.0.jar
SonarQube.MSBuild.Runner-0.9.zip
sonar-runner-dist-2.4.zip
I've followed instructions found in "SonarQube Installation Guide for Existing TFS Environment.pdf".
Any help ?
Thank you.
Log :
SonarQube Analysis Summary
Analysis failed for SonarQube project "WpfApplication2", version 1.0
Product projects: 0, test projects: 0
Invalid projects: 0, skipped projects: 0, excluded projects: 0
Résumé
Debug | x86
0 erreur(s), 1 avertissement(s)
$/Essai2/WpfApplication2/WpfApplication2.sln - 0 erreur(s), 1 avertissement(s),
Afficher le fichier journal
C:\Builds\1\Essai2\WpfApplication2\src\documents\visual studio 2013\Projects\WpfApplication2\WpfApplication2.sln.metaproj : The specified solution configuration "Debug|x86" is invalid. Please specify a valid solution configuration using the Configuration and Platform properties (e.g. MSBuild.exe Solution.sln /p:Configuration=Debug /p:Platform="Any CPU") or leave those properties blank to use the default solution configuration.
$/Essai2/WpfApplication2/WpfApplication2.sln compilé
Aucun résultat des tests
Aucun résultat de couverture du code
Autres erreurs et avertissements
2 erreur(s), 0 avertissement(s)
01:45:52: No ProjectInfo.xml files were found. Check that the analysis targets are referenced by the MSBuild projects being built.
Exception Message: TF270015: 'SonarQube.MSBuild.Runner.exe' returned an unexpected exit code. Expected '0'; actual '1'. See the build logs for more details. (type UnexpectedExitCodeException)
Exception Stack Trace: at System.Activities.Statements.Throw.Execute(CodeActivityContext context)
at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)
Here is the Diagnostic (I truncated the source check out part) :
Compile, Test and Publish00:00:46
Run optional script before MSBuild00:00:08
InputsEnvironmentVariables:
Enabled: True
Arguments: /key:WpfApplication2 /name:WpfApplication2 /version:1.0
FilePath: C:\sonarqube\bin\SonarQube.MSBuild.Runner.exe
OutputsResult: 0
C:\sonarqube\bin\SonarQube.MSBuild.Runner.exe /key:WpfApplication2 /name:WpfApplication2 /version:1.0
Pre-processing (3 arguments passed)
Using environment variables to determine the download directory...
Using environment variable 'TF_BUILD_BUILDDIRECTORY', value 'C:\Builds\1\Essai2\WpfApplication2'
Creating the analysis bin directory: C:\Builds\1\Essai2\WpfApplication2\sqtemp\bin
SonarQube server url: http://localhost:9000
Downloading SonarQube.MSBuild.Runner.Implementation.zip from http://localhost:9000/static/csharp/SonarQube.MSBuild.Runner.Implementation.zip to C:\Builds\1\Essai2\WpfApplication2\sqtemp\bin\SonarQube.MSBuild.Runner.Implementation.zip
Executing file C:\Builds\1\Essai2\WpfApplication2\sqtemp\bin\SonarQube.MSBuild.PreProcessor.exe
Args: "/key:WpfApplication2" "/name:WpfApplication2" "/version:1.0"
Working directory: C:\Builds\1\Essai2\WpfApplication2\sqtemp\bin
Timeout (ms):300000
Process id: 4168
01:45:11: The path to the sonar-runner.properties file was not supplied on the command line. Attempting to locate the file...
01:45:11: Located the runner properties file: C:\sonar-runner-2.4\conf\sonar-runner.properties
01:45:11: Legacy TeamBuild environment detected
01:45:11: Creating config and output folders...
01:45:11: Removing the existing directory: C:\Builds\1\Essai2\WpfApplication2\sqtemp\conf
01:45:11: Creating directory: C:\Builds\1\Essai2\WpfApplication2\sqtemp\conf
01:45:11: Removing the existing directory: C:\Builds\1\Essai2\WpfApplication2\sqtemp\out
01:45:11: Creating directory: C:\Builds\1\Essai2\WpfApplication2\sqtemp\out
Generating the FxCop ruleset...
01:45:14: Saving the config file to C:\Builds\1\Essai2\WpfApplication2\sqtemp\conf\SonarQubeAnalysisConfig.xml
Process returned exit code 0
Run MSBuild00:00:12
InputsProjectsToBuild: String[] Array
MSBuildMultiProc: True
Verbosity: Normal
ToolPlatform: Auto
Targets:
RunCodeAnalysis: AsConfigured
CommandLineArguments: /p:SkipInvalidConfigurations=true /p:Configuration=Debug /p:Platform="Any CPU"
ConfigurationsToBuild: String[] Array
OutputLocation: SingleFolder
Enabled: True
ToolVersion:
CleanBuild: True
OutDir:
RestoreNuGetPackages: True
C:\Program Files\Microsoft Team Foundation Server 12.0\Tools\nuget.exe restore "C:\Builds\1\Essai2\WpfApplication2\src\documents\visual studio 2013\Projects\WpfApplication2\WpfApplication2.sln" -NonInteractive
C:\Program Files (x86)\MSBuild\12.0\bin\amd64\MSBuild.exe /nologo /noconsolelogger "C:\Builds\1\Essai2\WpfApplication2\src\documents\visual studio 2013\Projects\WpfApplication2\WpfApplication2.sln" /nr:False /fl /flp:"logfile=C:\Builds\1\Essai2\WpfApplication2\src\documents\visual studio 2013\Projects\WpfApplication2\WpfApplication2.log;encoding=Unicode;verbosity=normal" /p:SkipInvalidConfigurations=true /p:Configuration=Debug /p:Platform="Any CPU" /m /p:OutDir="C:\Builds\1\Essai2\WpfApplication2\bin\\" /p:Configuration="Debug" /p:Platform="x86" /p:VCBuildOverride="C:\Builds\1\Essai2\WpfApplication2\src\documents\visual studio 2013\Projects\WpfApplication2\WpfApplication2.sln.x86.Debug.vsprops" /dl:WorkflowCentralLogger,"C:\Program Files\Microsoft Team Foundation Server 12.0\Tools\Microsoft.TeamFoundation.Build.Server.Logger.dll";"Verbosity=Normal;BuildUri=vstfs:///Build/Build/52;IgnoreDuplicateProjects=False;InformationNodeId=12;TargetsNotLogged=GetNativeManifest,GetCopyToOutputDirectoryItems,GetTargetPath;LogProjectNodes=True;LogWarnings=True;TFSUrl=http://localhost:8080/tfs/DefaultCollection;"*WorkflowForwardingLogger,"C:\Program Files\Microsoft Team Foundation Server 12.0\Tools\Microsoft.TeamFoundation.Build.Server.Logger.dll";"Verbosity=Normal;" /p:BuildId="dbda6e4d-d5bc-4eed-8b3e-6cc79e3721cc,vstfs:///Build/Build/52" /p:BuildLabel="WpfApplication2_20150615.3" /p:BuildTimestamp="Mon, 15 Jun 2015 11:44:59 GMT" /p:BuildSourceVersion="LWpfApplication2_20150615.3#$/Essai2" /p:BuildDefinition="WpfApplication2"
Run optional script after MSBuild00:00:00
InputsEnvironmentVariables:
Enabled: True
Arguments:
FilePath:
OutputsResult: 0
Run optional script before Test Runner00:00:00
InputsEnvironmentVariables:
Enabled: True
Arguments:
FilePath:
OutputsResult: 0
Run VS Test Runner00:00:00
InputsTestSpecs: BuildParameter[] Array
Enabled: False
ConfigurationsToTest: String[] Array
OutDir:
This activity was not run since its Enabled property was set to False.
Run optional script after Test Runner00:00:25
InputsEnvironmentVariables:
Enabled: True
Arguments:
FilePath: C:\sonarqube\bin\SonarQube.MSBuild.Runner.exe
OutputsResult: 0
C:\sonarqube\bin\SonarQube.MSBuild.Runner.exe
Post-processing (no arguments passed)
Using environment variables to determine the download directory...
Using environment variable 'TF_BUILD_BUILDDIRECTORY', value 'C:\Builds\1\Essai2\WpfApplication2'
Executing file C:\Builds\1\Essai2\WpfApplication2\sqtemp\bin\SonarQube.MSBuild.PostProcessor.exe
Args:
Working directory: C:\Builds\1\Essai2\WpfApplication2\sqtemp\bin
Timeout (ms):3600000
Process id: 6548
01:45:28: Legacy TeamBuild environment detected
01:45:28: Loading the SonarQube analysis config from C:\Builds\1\Essai2\WpfApplication2\sqtemp\conf\SonarQubeAnalysisConfig.xml
01:45:28: Legacy TeamBuild environment detected
01:45:28: Attempting to locate the CodeCoverage.exe tool...
01:45:28: Code coverage command line tool: C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe
01:45:28: Fetching code coverage report information from TFS...
01:45:28: Connecting to TFS...
01:45:29: Fetching build information...
01:45:30: Fetch code coverage report info...
01:45:52: Operation timed out, Elapsed time (ms): 20358
01:45:52: ...done.
01:45:52: No code coverage reports were found for the current build.
01:45:52: Generating SonarQube project properties file to C:\Builds\1\Essai2\WpfApplication2\sqtemp\out\sonar-project.properties
01:45:52: No ProjectInfo.xml files were found. Check that the analysis targets are referenced by the MSBuild projects being built.
01:45:52: Writing processing summary to C:\Builds\1\Essai2\WpfApplication2\sqtemp\out\ProjectInfo.log
01:45:52: Generation of the sonar-properties file failed. Unable to complete SonarQube analysis.
01:45:52: Updating the TeamBuild summary...
Process returned exit code 1
Exception Message: TF270015: 'SonarQube.MSBuild.Runner.exe' returned an unexpected exit code. Expected '0'; actual '1'. See the build logs for more details. (type UnexpectedExitCodeException) Exception Stack Trace: at System.Activities.Statements.Throw.Execute(CodeActivityContext context) at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager) at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)
Handle Exception
Troubleshooting issues with the MSBuild.Runner v0.9
The following information is provided to help troubleshooting issues with version 0.9 of the MSBuild.Runner. The behaviour may change in later versions.
Most serious configuration issues will result in warnings or errors that will appear on the Build Summary page. In cases in which there are no errors or warnings reported, check the TFS TeamBuild diagnostic output (click on "Diagnostics" link at the top of the build summary report in Visual Studio). Both the pre- and post-processor steps log quite a lot of information to help diagnose issues.
1. Check the SonarQube Analysis Summary section appears in the Build Summary Report
The section should look like this:
SonarQube Analysis Summary
Analysis failed for SonarQube project "Simple console app", version 1.0
Product projects: 4, test projects: 2
Invalid projects: 0, skipped projects: 0, excluded projects: 0
If the section does not appear on the build summary page then there are configuration issues with the build agent, build definition, or both. Read the steps below to diagnose these further.
1.1 Check the number of projects that are reported in the summary
If no projects were found then it suggests that the integration targets are not being imported and used during the build.
If all of the projects are marked as invalid, it is likely that you are building more than one configuration e.g. [release or debug] | [x86 or x64]. You should also get an error saying no ProjectInfo.xml files could be found.
You can only analyse one configuration at a time. If you want to analyze both release and debug builds, then you will need to create a separate build definition for each, using different SonarQube project ids.
The diagnostic log will contain more infomation about why a project was invalid, which is normally because the project doesn't have a guid, or the guid is not unique. The normal reason for a non-unique guid is that you are building multiple configurations e.g. release and debug.
2. Check that the pre-build step is being executed
The diagnostic output should contain output similar to the following:
Run optional script before MSBuild
C:\SonarQube\bin\SonarQube.MSBuild.Runner.exe /key:MyProject /name:"My Project" /version:1.0
Pre-processing (3 arguments passed)
If it doesn't, check that the you are correctly calling SonarQube.MSBuild.Runner.exe in the "Pre-build script" step in the build definition. An easy mistake is to put the script parameters in the wrong box by setting the "Post-build" properties instead (unhelpfully, the Post-build fields in the UI appear before the Pre-build fields).
2. Check that the post-build step is being executed
The diagnostic output should contain output similar to the following:
Run optional script after Test Runner
C:\SonarQube\bin\SonarQube.MSBuild.Runner.exe
Post-processing (no arguments passed)
If it doesn't, check that the you are correctly calling SonarQube.MSBuild.Runner.exe in the "Post-test script" step in the build definition.
3. Check the rest of the diagnostic log
If the pre- and post- build steps are being executed correctly, check the rest of the diagnostic output for any clues as to what isn't working. Pay particular attention to messages from the pre- and post-processor about the directories and files they are writing to.
4. Check the files that have been created on the build agent
The analysis process creates temporary files under the build directory under a folder called SQTemp in v0.9.
Examining the contents of the folders can help determine the stage at which the analysis is failing.
The expected folders are as follows:
bin: this folder contains the analysis targets and binaries. It is created by the SonarQube.MSBuild.Runner.exe which downloads the files from the SonarQube server. If this folder does not exist or is empty then the analysis failed at a very early stage and there should warnings or errors. Check that the C# plugin v4 is installed correctly on the SonarQube server, and that the SonarQube has been restarted since the plugin was installed.
conf: this folder is created during pre-processing and contains the settings downloaded from the SonarQube server. If there are any FxCop rules active in the SonarQube the folder should contain an FxCop ruleset.
out: this folder is populated during the MSBuild phase. It should contain one folder per project being built. It will also contain the generated sonar-project.properties file that is passed to the sonar-runner. If the folder is empty then it is possible the SonarQube.Integration.targets are not being imported correctly.
The error message "No ProjectInfo.xml files were found" will normally appear in this case. The ProjectInfo.xml files are generated by the target "WriteSonarQubeData" in the SonarQube.Integration.targets, so the next thing to check is that SonarQube.Integration.targets are being imported correctly and the expected targets are being executed.
5. Checking the SonarQube.Integration.targets are imported and called
Firstly, double-check the installation of the SonarQube.Integration.ImportBefore.targets file:
check it is in the correct folder for the version of MSBuild you are using i.e. %ProgramFiles(x86)%\MSBuild\[12.0 or 14.0]\Microsoft.Common.Targets\ImportBefore
check for typos in the folder name "ImportBefore" (should be singular, not plural)
(Thanks to Richard from BlackMarble for this tip).
If the targets are installed correctly then you will need to increase the verbosity of the MSBuild logging to diagnostic and build again so you can check the detailed MSBuild output. Unfortunately, there doesn't seem to be a simple way to increase the verbosity of the MSBuild logs when running under TeamBuild 2013. You can produce an additional log file with more detail, but you will need access to the build agent machine to pick up the log file.
To create an MSBuild log with more detailed info:
select "Queue New Build" from the BUILD menu in Visual Studio
click on the "Parameters" tab.
Expand the "2. Build" section and the "5. Advanced" sections
Set the "MSBuild arguments" property to the following:
/m:1 /fl2 /flp2:"verbosity=diagnostic"
The "/m:1" tells MSBuild not to build in parallel which can make the log easier to read. The other two parameters tell MSBuild to log to a file. See the MSBuild command line help for more info.
Click "Queue" and wait for the build to finish.
Find and open the log on the build agent. It should be called msbuild2.log and be under the source directory for your build definition.
If the SonarQube.Intergration.ImportBefore.targets are being executed, there will be a log entry saying the "SonarQubeImportBeforeInfo" target was executed. There should be some output message saying whether the file "SonarQube.Integration.targets" was located or not.
The ProjectInfo.xml files are written by the target "WriteSonarQubeProjectData". The log should show that this target has been executed, and the "WriteProjectInfoFile" task has executed. If not, the log should give an idea of why the targets were skipped.
I've been trying to get Jenkins to display a JUnit report of a sample js project which i am testing with QUnit. I have literally scoured the internet for bits and pieces and so far, Running QUnit tests with Jenkins and Apache Ant? is the most helpful post that i have found.
I can confirm that:
The user has sufficient privileges to write to disk
PhantomJS works headless from the shell when i write something along the lines of:
[user#myserver PhantomJS]$ phantomjs phantomjs-runner/runner.js web/index.html
And shows:
Took 8ms to run 6 tests. 6 passed, 0 failed.
Qunit does work and provides test results when executed in a browser
Still i cannot get the report.xml to generate in order to feed it into Jenkins. Below is the target i have added to my build.xml file:
<target name="qunit" description="runs QUnit tests using PhantomJS">
<!-- QUnit Javascript Unit Tests -->
<echo message="Executing QUnit Javascript Unit Tests..."/>
<apply executable="/usr/local/CI/phantomjs/bin/phantomjs" >
<arg value="/usr/local/CI/phantomjs-runner/runner.js" />
<arg line="--qunit /usr/local/CI/jenkins/workspace/PhantomJS/web/js/qunit-1.17.1.js --tests /usr/local/CI/jenkins/workspace/PhantomJS/web/index.html --junit /usr/local/CI/jenkins/workspace/PhantomJS/test-results/report.xml" />
<fileset dir="${basedir}/web/" includes="/js/prettydate.js" />
<srcfile/>
</apply>
<echo message="Tests complete..."/>
</target>
Compiling the project in Jenkins gives me the following output:
? PhantomJS/result.xml
? PhantomJS/test-results
Using locally configured password for connection to :pserver:user#server:/cvsroot
cvs rlog -S -d18 Feb 2015 15:49:54 +0000<18 Feb 2015 15:51:40 +0000 QUnit_Jenkins
[PhantomJS] $ /usr/local/CI/ant/bin/ant qunit
Buildfile: /usr/local/CI/jenkins/workspace/PhantomJS/build.xml
qunit:
[echo] Executing QUnit Javascript Unit Tests...
[echo] Tests complete...
BUILD SUCCESSFUL Total time: 0 seconds Recording test results Test
reports were found but none of them are new. Did tests run? For
example,
/usr/local/CI/jenkins/workspace/PhantomJS/test-results/report.xml is 4
hr 18 min old
Build step 'Publish JUnit test result report' changed build result to
FAILURE Finished: FAILURE
As you may notice, jenkins can't find an updated report.xml file because there simply isn't one getting generated.
Can you observe any mistakes in my build.xml? If not, any ideas, hints that would assist me in getting the result.xml file generated?
I have found a solution to my answer by taking the following steps:
1) added <script src="js/qunit-reporter-junit.js"></script> as it is required to generate the report. Ensure you also have the qunit.js library included also. I used qunit-1.17.1.js
2) I placed the following code in the html file that tests my js code:
<script>
QUnit.jUnitReport = function(report) {
console.log(report.xml)
};
</script>
3) I added the Ant code in my build.xml file:
<target name="build" description="runs QUnit tests using PhantomJS">
<!-- Clean up output directory -->
<delete dir="./build/qunit"/>
<mkdir dir="./build/qunit"/>
<!-- QUnit Javascript Unit Tests -->
<echo message="Executing QUnit Javascript Unit Tests..."/>
<exec executable="/usr/local/CI/phantomjs/bin/phantomjs" output="./build/qunit/qunit-results.xml">
<arg value="/usr/local/CI/phantomjs-runner/runner-muted.js"/>
<arg value="./web/index.html"/>
</exec>
</target>
You will observe that i changed the name of runner.js to runner-muted.js This is so, because i have made changes to runner.js to not include its output to the xml file, as this makes it unreadable by jenkins. To do so:
cp /usr/local/CI/phantomjs-runner/runner.js /usr/local/CI/phantomjs-runner/runner-muted.js
Find and comment out console.log occurrences found under QUnit.done and QUnit.testDone to mute the runner from displaying its own test run results.
Ensure that in Jenkins you have selected the correct path to the generated xml file.
I hope this helps any of you trying to get it to work
What I'm trying to achieve is to have log4j as the only output from my Ant build file.
Getting log4j works pretty much perfectly except that it extends Ants output. So I'm getting Ant's DefaultLogger interspersed with log4j output. Take a VERY simple Ant build file:
<project name="Maintenance_to_Delivery" default="main" basedir=".">
<target name="main">
<echo message="test">
</target>
</project>
And log4j.properties
log4j.rootLogger=ERROR, stdout
log4j.logger.org.apache.tools.ant.Project=INFO
log4j.logger.org.apache.tools.ant.Target=INFO
log4j.logger.org.apache.tools.ant.taskdefs=INFO
log4j.logger.org.apache.tools.ant.taskdefs.Echo=INFO
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{HH:mm:ss} %-5p %m%n
log4j.appender.stdout.Threshold=INFO
And running this plus the output:
# ant -lib /app/apache-sf/log4j/1.2.16/log4j-1.2.16.jar:. -listener org.apache.tools.ant.listener.Log4jListener
Buildfile: /home/sysswb/working/qa/sep/M2D-new/build.xml
16:36:59 INFO Build started.
main:
16:37:00 INFO Target "main" started.
[echo] TEST
16:37:00 WARN TEST
16:37:00 INFO Target "main" finished.
BUILD SUCCESSFUL
Total time: 1 second
16:37:00 INFO Build finished
so - is it possible to get log4j to completely replace the DefaultLogger output? ...without writing a custom logger? Since the -listener argument registers extra listeners, can whatever listener the DefaultLogger uses be removed/disabled?
I'm just Java 1.6.0_18 and Ant 1.8.0.
TIA.
You can pass log4j.properties in the class path while running the ant.
Check out this, it has much detailed explanation, follow comment by "Kevin", here is the excerpt:
Log4j loads its configuration by
looking for the system property
log4j.configuration. Failing that, it
looks for log4j.properties or
log4j.xml on the classpath.
try using the -e switch
ant -lib . -listener org.apache.tools.ant.listener.Log4jListener -e main
Buildfile: D:\install\apache-ant-1.8.0RC1-bin\apache-ant-1.8.0RC1\bin\build.xml
14:29:20 INFO Build started.
main:
14:29:20 INFO Target "main" started.
test
14:29:20 WARN test
14:29:20 INFO Target "main" finished.
BUILD SUCCESSFUL
Total time: 0 seconds
14:29:20 INFO Build finished.
[echo] test is gone