Is there a proper way to create a build matrix specific for Pull Requests?
The idea is:
In normal builds, I want to test a few things only (code style/standards, some unit tests, some general validation). Mostly one item only in the build matrix.
In pull requests, I want to run the tests with several different environments, including different databases and versions. This is what I currently have but it demands a lot from travis (and it is slow).
I know I can achieve that in the script by checking TRAVIS_PULL_REQUEST and skipping the tests, but that will misleadingly show some environments as "passed" when they were actually not tested.
Thank you for any help / guidance,
Daniel
Interesting wish!
This is not possible at the moment. You might want to chime in on https://github.com/travis-ci/beta-features/issues/11 to bring it to the attention of the relevant people.
Related
When I am running my test solution, single browser is getting launched but it is running two feature file simultaneously due to which test cases are failing. One step it is taking from one feature file and other from other feature file.
Contrary to the comment left on your question, I think I may have enough context to answer you.
You describe feature files that are sharing steps and concerns about multiple browser instances. This tells me that your various step files might each be containing a browser instance.
What you're likely looking to do instead is to use a SpecFlow Context -- SpecFlow provides it own ScenarioContext object you can use, or you can create your own context and inject it.
Some links that might help:
SpecFlow docs on sharing data between bindings, which explains about ScenarioContext and FeatureContext:
https://docs.specflow.org/projects/specflow/en/latest/Bindings/Sharing-Data-between-Bindings.html
Here's an article on using SpecFlow with Selenium and the Page Object Model: https://docs.specflow.org/projects/specflow/en/latest/ui-automation/Selenium-with-Page-Object-Pattern.html
The SpecFlow YouTube channel will likely be helpful as it's full of experts walking through these sort of examples: https://www.youtube.com/c/SpecFlowBDD/videos
Here's the first video in a 5 part series on how to automate a web application with Selenium and SpecFlow: https://www.youtube.com/watch?v=y1dAogvWVh8
Based on your question, it's also possible that your issue could be that you want things to run in parallel, but have the problem of your tests being dependent on one another or running in a certain order. This will be a bit more complex to solve.
I strongly suggest you treat your tests so that they can be run in isolation. You may need to add separate data to a database, or operate your tests so that they're not touching the same thing. This takes more work, but is more than worth it, because it will enable better maintainability and reliability of your tests and also ensure they can run in parallel successfully.
I hope this helps!
I'm developing an application in rails with cucumber.
The application includes a workflow that have multiple steps.
Some steps are
A user import files (3 different files),
Other user make make some checks to date that was imported,
Other user input some parameter,
Other user apply the parameter to the data that was imported,
etc.
The steps must be executed in the correct order, and I is necessary to run all the previous steps in order to execute each one, for example to apply the parameter its necessary to have the data imported and the parameters defined.
My problem is how to build cucumber scenarios/features in this situation.
I know that a scenario is not suppose to call all the previous scenario.But the only other idea that I have is to create a very long scenario performing all this steps, and that make sense because it will be a scenario with more than 2 hundred steps.
Any thought on a pragmatical way of implementing cucumber in this kind of situation?
Many Tks
It sounds as if you have to perform every thing every time.
Will every usage of your system include importing three files? Are there any cases where the user may only need to import two files? If the case is that there will always be three files imported, then you might abstract that step as
given the files are imported
Things that always have to be done may be combined into some generic setup. As the setup never changes, the details may not be necessary to mention explicit.
My experience though, is that at the beginning it is hard to separate scenarios and try to do too much in a few scenarios with many steps. If you don't see any other way, start there. Look at your scenario and see if they possible to separate into perhaps two independent scenarios. It may be possible to separate it into two scenarios that are independent. Next step would be to see if these two new scenarios are possible to divide into two smaller, independent scenarios. It happens that it is possible.
It is obviously always possible that Cucumber is not the tool you need. It is possible that you would be better off with a unit test framework.
I have number of different manual test cases which needs to be automated with the Specflow .
There are multiple test cases multiple scenarios. SO there will be multiple feature files ?
We are following Sprint system. each sprint has 100+ test cases which are going to be automated.
Which will be the best practice for managing the test cases and scenarios using the feature files ? Theres no point in creating same set of functions everytime for different test cases.
you would manage this the same as you would manage any other code files. Make changes, if the changes conflict with others changes then merge the changes before you check in.
The best way to avoid merge issues is to try and work in different areas. Create many feature files as then multiple people can work on different features at one time and you won't have conflicts.
Communication between the testers is important in avoiding conflicts as well, and in the case of scenarios in specflow it will be important in ensuring that you use consistent step names. Also checking in often will ensure that you minimise the number of merge issues, even after each scenario has been created.
EDIT
based on your edited question in specflow all steps are global, so if feature1 has a scenario with a step Given a user 'Bob' is logged in and Feature32 also has a scenarion with the step Given a user 'Tom' is logged in then they will both share the same step source code and so the same function will be reused.
As long as you write your steps in a consistent manner (ie use the same text) then you should get excellent reuse of functions across all of your hundreds of features and scenarios.
I have an ASP.Net MVC project and I thought I could use a tool like MS Test or NUnit to perform regression testing from the controller layer down to the database, however I hit an issue where tests are not designed to run in order (You can use ordered tests in MS Test, but the tests still run concurrently) and the other problem is how to allow the data created from one test accessible to another?
I have looked at Selenium and WatiN but I just wanted to write something that is not dependent on the UI layer which is most likely going to change an increase the amount of work to maintain the tests.
Any suggestions? Is it just the wrong tool for the job? Should I just use Selenium/WatiN?
Tests should always be independent of each other, so that running order doesn't matter. If your tests depend on other tests you are losing control of what you are testing.
WatiN, and I'm assuming Selenium, won't solve your ordering problem. I use WatiN and NUnit for UI automation and the running order is not guaranteed, which initially posed similar problems to what you're seeing.
In the vein of what dskh answered, you want independent tests, and I've done this in two ways for Integration / Regression black-ish box testing.
First: In your test setup, have any precondition data values setup so you're at a known "good state". For system regression test automation, I've got a number of database scripts that get called to reset data to a known state; this adds some dependencies so be conscious of the design. Note: In straight unit testing, look at using mock objects to take out dependencies and get your test to be "testing one thing". Mock objects, stubbing method calls, etc is the way to go if you can, which based on your question sounds likely.
Second: For cases where certain things absolutely had to be setup in a certain way, and scripting them to test setup added a ridiculous amount of necessary system internal knowledge (eg: all users setup + all permissions setup + etc etc etc) a small number of "bootstrap" tests were setup, in their own namespace to allow easy running via NUnit, to bootstrap the system. Keeping the number of tests small and making sure the tests were very stable was paramount. Now, on a fresh install, the bootstrap tests are run first and serve as advanced smoke tests; no further tests are run if any of the bootstrap tests fail. It is clunky in some ways, but the alternatives were clunkier or more time/resource/whatever consuming.
Update
The link below (and I assume the project) is dead.
Best option maybe using Selenium and the Page Object Model.
See here: http://code.tutsplus.com/articles/maintainable-automated-ui-tests--net-35089
Old Answer
The simplest solution I have found is Rob Conery's Qixote:
https://github.com/robconery/Quixote
It works by firing http requests and consuming responses.
Simple to set up and use and provides integration testing. This tool also allows a series of tests to be executed in order to create test dependencies.
Has anyone used BDD for driving their domain Layer?
Yes, we have found this process to have worked very well and used specflow to deliver this approach fairly easy. We have over 2000+ scenarios implemented in our domain layer alone and we have also used this approach for testing our controllers in our UI layer too(another 2000+ tests).
It's a good idea if working on a large project to put some thought into how to organise the steps before you start as you will quickly start to collect a large amount of steps and finding a step can become a bit of a challenge.
The biggest issue we had was having multiple people on a team writing scenarios, they would often write the same step but with a slightly different wording resulting in the same step being added twice.
Yes, although lately we've been looking at Cuke and Specification by Example as a higher level to start driving from. See http://specificationbyexample.com/
Yes, that's what it's for!
I've found BDD main benefits is how it, in a natural way, lets you;
Drive the design (plan then do)
Discover and emphasize the domains ubiquitous language
Document project progress and current state (specs maps to stories and sprint plans)
If it also results in acceptance or unit tests that's great but I think most value comes out of the above. It also helps new team members get a grip on things and it's very easy to return to projects domain mentally after being away for a while.
I also agree on before mentioned "step-duplication" problems, it's well spent time refactoring and consolidating steps trying to keep them well-structured.