Is there any way to execute code before and after each specific thread when executing tests in parallel with SharedAppDomain thread isolation in Specflow?
I was using BeforeTestRun/AfterTestRun Hooks for this purpose, but it seems that since last Specflow versions this is only possible when running tests "AppDomain" thread isolation, and in SharedAppDomain isolation it runs only once per test.
If you are running your tests in parallel, each thread will represent a running scenario. To run something after each scenario you can use the AfterScenario hook.
BeforeTestRun/AfterTestRun are for the entire test run where multiple threads may be running. They are designed to run once for all threads but I have found AfterTestRun to be patchy. Sometimes it appears unable to determine that the last running thread has finished before entering AfterTestRun.
I have Jenkins building my C# .NET Core api project. I added some xUnit tests and included a powershell script inside of my Jenkins build with the "dotnet test" command to execute the tests.
That all works well and the tests are run and i can see the output in the Jenkins console.
The problem is that if i have failing tests nothing happens - jenkins goes merrily along and finished up the build process and reports it as a success.
How can i get it to fail the build?
Is there a response from the 'dotnet test' command?
I know there are xUnit Jenkins plugins but they all seem to revolve around "display the results of xUnit tests". Which is not really what i am after. I want to ACT on the results of the tests, not just see them in fancy html.
You should check for the return code from dotnet test command. It returns 0 if all tests were successful and 1 if any of the tests failed. Unfortunately it's not documented but was confirmed in this issue
I want to run dart unit tests on a Windows Jenkins server. The problem is when the tests complete, it prints
`pub serve` was not terminated
and waits for user input before (Ctrl-C x2) returning to the console. This causes Jenkins to not know the tests completed, causing an eventual timeout build failure. Is there a way to make pub run test/angular_test return to the console when the tests are completed?
Using Protractor/Jasmine conjunction automation framework I want to run test suite. When my Jenkins job runs I do not want to fail my job if any test case fails.
Pytest in python provides #pytest.mark.xfail feature to mark expected failures and this does not impact the jenkins job.
Is there any such feature in Protractor which can mark test cases as expected to fail?
I saw xit and xdescribe features but it skips the test case rather then expected failure
I have a pipeline job running on Jenkins that runs iOS unit tests.
I'm seeing some really strange behaviour when our tests fail, fastlane shows the failure in it's summary, but then begins to rebuild the code and re-test.
This happens continually, and I can not figure out if it's a fastlane/scan setting I am missing, or something with running this in a Jenkins Pipeline.
bundle exec fastlane test runs locally just fine, and only executes once per device defined in my scan file.
My console output looks like so:
[11:26:36]: fastlane finished with errors
[!] Test execution failed. Exit status: 65
#######################################################################
# fastlane 2.19.3 is available. You are on 2.17.0.
# It is recommended to use the latest version.
# Please update using `bundle update fastlane`.
#######################################################################
2.19.3 sigh Hotfix
* fix sigh undefined variable regression (#8457)
2.19.2 Snapshot improvements and more
* Add tests for ReportsGenerator available_devices
* Fix HTML report generation for snapshot
* Prioritize options passed to supply
* Collect logs for iOS 10+ devices
* Add row for Fabric.app-installed fastlane to env command
* Sigh reports better errors for team members
* Expand the notes_path parameter
* Update xcov intialization
2.19.1 Hot fix for provisioning profile creation/deletion
- Re-implement csrf token retrieval for provisioning profiles (#8410)
- don't attempt to create apps on the dev center and itc if mac app (#8404)
To see all new releases, open https://github.com/fastlane/fastlane/releases
Please update using `bundle update fastlane`
/Users/hudson/build/workspace/MNELastWord-PR-Pipeline#tmp/durable-ba2bbebe/script.sh: line 2: shell_session_update: command not found
Sending interrupt signal to process
▸ Cleaning Pods/CocoaLumberjack-iOS [Debug]
▸ Check Dependencies
▸ Cleaning Pods/Bento-iOS-Lib-iOS-BentoResources [Debug]
▸ Check Dependencies
▸ Cleaning Pods/AFNetworking-iOS [Debug]
▸ Check Dependencies
▸ Cleaning Pods/JSONModel-iOS [Debug]
You can see after fastlane completes, it just goes on it's merry way and starts to rebuild and re-test instead of exiting the job.
Based on this output, I think it's something in Fastlane causing this to re-execute, because there's no output from Jenkins that indicates the stage is getting called again.
That said, I see nothing in fastlane issues on Github about this, other than Snapshot having a retry option (which I am not using snapshot)
for completeness, here's my Scanfile, Jenkinsfile, and Fastfile
and screens from relevant jenkins config:
Can someone help out here? I've tried the latest fastlane, older fastlanes, and still see the behaviour.
My DevOps and SAs are getting pretty upset with us because our jobs never end, and are hanging our jenkins nodes :) (aborting/terminating them is really problematic)
So ultimately I think I've gotten to the bottom of the issue here.
If I wrap my Test stage in a try/catch block, then I get appropriate behaviour from the job.
stage 'Tests'
try {
env.FASTLANE_EXPLICIT_OPEN_SIMULATOR = 1
bash 'bundle exec fastlane test'
} catch (e) {
currentBuild.result = 'FAILURE'
throw e
}
This doesn't really make sense since my entire section is wrapped in a try/catch - so if someone has insight as to WHY this works, i'm all ears.
But at least I am able to work around the issue right now.
Looks like test case is failing but Jenkins job is not stopping.
Could be two possibilities here:
Try exiting the job once the test case fails, so it won't continue after, hence added explicit exit 1 if test case execution fails
Example
stage('Run Tests') {
// Shell build step
sh """
#!/bin/bash
echo "Executing Fastlane test lane..."
$HOME/.fastlane/bin/fastlane tests || exit 1;
"""
}
Check if test case changing if source code file and doing the commit back to the SCM, and on commit/push Jenkins would be polling hence resulting in a loop.
To solve this use polling ignore on commit/user/message basis. Adding screenshot for reference