I would like all my scenarios to run, but I'd like to tag some scenarios so they are only excluded when running in certain environments. For example, when a scenario has no tags I want it to run in all environments, but if I tag it with #dev I want it to be excluded from all non-dev environments.
Is there a way to use scope binding to achieve this or is it better implemented with execution flags on the test runner?
Other than flags passed to the test runner, I was thinking maybe a scenario hook would be possible, but not sure how to implement the exclude condition because once the scenario has started I can't find a way to abort it.
When using xUnit, Tags are translated into Traits.
With them, you can filter which scenarios you want to execute.
This runs all tests with the #dev tag:
xunit.console.exe ... -trait "Category=dev"
Brendan Connolly wrote a nice blog post about xUnit traits: http://www.brendanconnolly.net/organizing-tests-with-xunit-traits/
About aborting an already started scenario: This is not possible.
Related
I ask you to consult on the following question about allure: I use jenkins + pytest to run the tests. The same tests run on several virtual machines, these machines differ in operating systems (different linux distributions) and test environment. After running the tests, I want to combine the results from all the machines into one report. - here the question arises - if I put all the reports in one directory and generate a report, then the results from different machines will be considered as rerun of the same test and combined into one. How can I get around this? so as not to be combined and so that it was possible to somehow sort out which result from which machine. Thanks.
i have solve this by override the names of tests/suites.
Meaning you have to make some code implementation, work with the before listeners, there you can get the current test name and override it. Set the test name by OS + Browser or something unique.
When you combine reports, they will be unique and properly displayed.
I ran into a similar issue with behave where Allure was treating each parallel build as a retry of the first build. I realize this isn't the same as pytest, but perhaps it'll help.
I was inspired by the previous answer and started experimenting. By changing the scenario name(s) within the feature, I was able to make Allure recognize each parallel build as separate tests. I accomplished this by adding a before_feature method to my environment.py file that simply added the hostname to each scenario name within that feature:
def before_feature(context, feature):
for scenario in feature.scenarios:
scenario.name = f'[{socket.gethostname()}] {scenario.name}'
Originally, I tried to directly change scenario.name in before_scenario but that seemed to have no effect in Allure.
We have a project with more than 100k regression tests and therefore we'd like to have the alternative to manually trigger and check if a Test Suite(s) have been broken due to a specific change instead of running all tests.
I have a build project configured in a way that everything is in place and it expects only parameter value to be appended i.e. -DwildcardSuites=<Suite Name Pattern>.
Is there a Jenkins plugin that would allow entering an input text to be appended to the Goals and options and that corresponds to the specific Suite pattern we'd like to run?
Extensible Choice? https://wiki.jenkins-ci.org/display/JENKINS/Extensible+Choice+Parameter+plugin
There are a few similar plugins, they tend to have one or more of "Choice", "Active" and "Dynamic" in the name.
how to add a single test page to multiple test suites?
Please note that I want to get the suite name in SuiteSetUp during the test run. Is this possible in FitNesse?
how to add a single test page to multiple test suites
You simply create a page under your suite and include the test page.
Not sure about the second questions, sounds like you're trying to do dynamic include and that's not possible.
I.e.
!include $mySuite
where $mySuite is defined in SuiteSetUp. At least I couldn't find a solution.
You might want to consider generating the data in the SuiteSetUp and use something like parametrized includes.
Personally I found them to work better with symbols.
My Jenkins job runs many tests that create log files. In case of failure, I want to look at the log of the failed test. I'd rather use Jenkins web-server to do it, even have a link in the email it sends me.
Is there any plugin that can do it? Or maybe another way?
You provide few details in your question, so it is impossible to give specific advice. In a general level: this is already possible. When your test framework creates JUnit XML files with test results, the test output can be included between the <failure> and </failure> tags. Usually test frameworks should take care of this automatically, so you are probably not using a test framework and are manually generating the XML files containing test results?
I recommend you adopt some test framework. It is usually well worth the effort.
Is there a way to mark a fitnesse test such that it will not be run as part of a suite, but it can still be run manually?
We have our FitNesse tests running as part of our continuous integration, so new tests that are not yet implemented cause the build to fail. We'd like a way to allow our testers and BAs to be able to add new tests that will fail while still continuing to validate the existing tests as part of continuous integration.
Any suggestions?
The best way to do this is with suite tags. You can mark tests with a tag from the properties page and then you can filter for the or filter to exclude them.
In this case I would exclude with "NotOnCI" tag. Then add the following argument to the URL:
ExcludeSuiteFilter=NotOnCI
This might look like this then as the full URL:
Http://localhost:8080/FrontPage?test&ExcludeSuiteFilter=NotOnCI
You can select multiple tags by splitting with commas, but they act as "or",Not "and".
Check the FitNesse user guide for more details. http://fitnesse.org/FitNesse.UserGuide.TestSuites.TagsAndFilters
Would it make sense to have multiple Suites, one for regression tests that should always pass, and another one for the tests that are not yet implemented?
Testers and BAs can add tests/suites to the latter suite and the CI server only runs tests in the former suite.
Once a developer believes he has implemented the behavior they can move the test/suite relating to that functionality to the 'regression' suite so that it will be checked in continuous integration.
This might make the status of a test/suite a bit more explicit/obvious than just having a tag. It would also provide a clear handover from development to test/BA to indicate the implementation is finished.
If you just want to have a test/suite not run during an overall run of a suite that contains the particular test/suite you could also just tick 'Skip (Recursive)' in the properties page of that test/suite (below 'Page Type').