We are just getting started with BDD in our company, and we are currently trying to write our first features.
We have created something like the following:
Feature: There can only be one
It is important that there is only one Highlander in the system for a given time period.
Scenario: Inserting a new Higlander for a time period already taken by another
Given there is a Highlander Adam in the system for time period 1820-1900
When I insert a Higlander Bert for time period 1800-1900
Then the System raises Error Code "ONLY1"
Scenario: Inserting a new Higlander for a free time period
Given there is a Highlander Adam in the system for time period 1820-1900
When I insert a Higlander Bert for time period 1901 - 1902
Then the System raises no Error
My question ist about the error code (ONLY1) in the first scenario. Currently, our system raises many errors, but we do not have
a simple way to identify a specific error condition.
Our System ist more than 15 years old. Isn't it funny that we havn't found this to be an issue for so many years?
I find it great that BDD makes it so obvious that we need a clear, shared language to identify our error messages.
Some of my colleagues argue that there is no need for an error code. This means that we have to write the assertion like this:
Then the System shows the error "There can only be 1 Highlander for a given time period."
This makes me cringe, because
The test will break if we later decide to change the text to something else
The test will only be successful on the english version of the software (we support several languages)
An error Code is great for finding information in a knowledge base or in suppoert tickets.
Of course, we should show the error code and a readable description of the error in the users language, but in the test spec, I would prefer to only mention the error code.
How do you test for errors? do you use codes or exceptions or just plain text?
Best Regards
Mat
Speaking strictly from a BDD persepective, you want to stay focused on the behavior that is expected. Keep your tests devoid of technical details. That being said, error codes are technical details. When trying to adhere to BDD, avoid error codes in your feature files. Put the error codes in your step definitions instead.
User friendly error messages are a little different though. An error message shown to an end user is not a technical detail in the same sense as an error code or enum. It is something that an end user should understand. If they do not understand the user friendly error message, then it is not user friendly and should be rewritten until it is. So your example below is perfectly valid BDD:
Then the System shows the error "There can only be 1 Highlander for a given time period."
You should also understand the drawback to this approach. Any change to the message terminology means you need to update all tests that reference this message. While putting the message in your features might not violate BDD, it does make your test suite more labor intensive to maintain. A better assertion captures the essence of the behavior without requiring the scenario author to know the exact details of the user interface:
Then the system says there can only be 1 highlander
The step definition is then free to use error codes, enums or UI error messages as part of the assertion. This provides a layer of abstraction between the implementation and the description of the behavior. A functional change to the UI or code that does not affect the overall behavior can be fixed in one spot.
Check out BDD 101: Writing Good Gherkin for some good guidance.
Related
I plan to use https://github.com/risoflora/brookframework as an embedded static server in a Delphi application. It requires https://github.com/risoflora/libsagui and seems very fast. But I can't find an option to suppress messages coming from the libsagui DLL.
A common scenario is when the desired port can't be bound, I prefer to get an exception or some error handler callback, rather than get a MessageBox which can't be controlled by my application.
Any info or suggestion?
I'm going to take a flying leap here and just toss out what occurs to me at the moment. First, DLLs are not supposed to have ANY sort of UI activity. If someone does that deliberately and not because they just don't know, then it usually signals there's a serious problem afoot in the run-time activity.
Second, assuming that this signals a serious problem, then there's likely to be some mechanism in place to avoid it. This is for an embedded situation, so one might ask "why can't a desired port be bound" if the person doing the embedding is also responsible for configuring the overall embedded environment? One does not leave things undefined or unconfigured in embedded situations or the application just blows up at some point. So the obvious question is, why are YOU (as the person who configured the embedded environment) writing code that's trying to connect to unavailable ports? And why are YOU also complaining that the underlying library isn't throwing an exception rather than throwing up its virtual hands saying it doesn't know how to handle a request YOUR CODE should have known better than to make?
In other words, if you feel the need to add a layer to your code between your app and the platform you configured to support it, in order to validate your own requests, then you're free to do so. Otherwise, I'm guessing the embedded library designer expects you to know the limits you configured into it. (Or there may be some he did not anticipate.)
I've done a lot of embedded programming in my past, and if I was using a configurable kernal that I configured to only support 10 tasks (because I knew my application didn't need more than that), then in my client code I'd do a check to ensure that said limit wasn't reached BEFORE creating a new task and expecting to get an exception back if I was too stupid to follow my own design guidelines. In embedded situations, "exceptions" can lead to auto-shutdown or even self-destruct sequences. There is no effective way to handle "exceptions" in real-time situations, and often even just embedded situations.
Imagine pushing on a brake pedal in a car and an exception is thrown -- the user is saying "slow down" and the library says, "Sorry, I can't do that right now". It's shades of "2001 A Space Odyssy" with HAL telling Dave "No". That's what "exceptions" in embedded systems can result in.
What's different between that and your app saying, "Connect to this URL via this port" and the underlying library saying, "Sorry I can't do that right now". What is your exception handling code going to do? You need to detect that stuff FIRST rather than wait for an exception to be thrown by the embedded library.
Finally, this is more of a rant than anything else ... but one thing I've found sorely lacking in the vast majority of open-source projects on github I've looked at is any clear explanation of one or more USE-CASE MODELS. In this situation, WHY did the author choose to design this embedded DLL so it has situations where it simply throws up its hands by displaying an error message in a low-level MessageBox rather than providing some mechanism for handling the issue more elegantly? Obviously, he had SOMETHING in mind when he did that. Unfortunately for us, he did not explain it anywhere. Here we are trying to figure out how to deal with something that, on its face, makes no sense -- a DLL throwing up its hands when faced with an exceptional situation.
So my answer is, in the "spirit" of open-source code repos, you're free to dig into the code and try to figure out the USE-CASE MODEL that the author had in mind when he created this thing, and adapt your code to fit that model. Maybe start by looking at examples of code he posted that use this library, although if they're your typical trival examples people often post, then they won't offer a lot of insight into the overall USE-CASE MODEL that is leading to your situation.
Said another way, assume the library is behaving "as designed". Then WHAT IS THIS LIBRARY EXPECTING YOU TO DO IN YOUR CODE IN ORDER TO AVOID THAT SITUATION FROM ARISING?
Hopefully someone will see this who has a much shorter definitive answer to that.
I have a fairly large MVC3 application, of which I have developed a small first phase without writing any unit tests, targeted especially detecting things like regressions caused by refactoring. I know it's a bit irresponsible to say this, but it hasn't really been necessary so far, with very simple CRUD operations, but I would like to move toward a TDD approach going forward.
I have basically completed phase 1, where I have written actions and views where members can register as authors and create course modules. Now I have more complex phases to implement where consumers of the courses and their trainees must register and complete courses, with academic progress tracking, author feedback and financial implications. I feel it would be unwise to proceed without a solid unit testing strategy, and based on past experience I feel TDD would be quite suitable to my future development efforts here.
Are there any known procedures for 'converting' a development effort to TDD, and for introducing unit tests to already written code? I don't need kindergarten level step by step stuff, but general strategic guidance.
BTW, I have included the web-development and MVC tags on this question as I believe these fields of development can significant influence on the unit testing requirements of project artefacts. If you disagree and wish to remove any of them, please be so kind as to leave a comment saying why.
I don't know of any existing procedures, but I can highlight what I usually do.
My approach for an existing system would be to attempt writing tests first to reproduce defects and then modify the code to fix it. I say attempt, because not everything is reproducible in a cost effective manner. For example, trying to write a test to reproduce an issue related to CSS3 transitions on a very specific version of IE may be cool, but not a good use of your time. I usually give myself a deadline to write such tests. The only exception may be features that are highly valued or difficult to manually test (like an API).
For any new features you add, first write the test (as if the class under test is an API), verify the test fails and implement the feature to satisfy the test. Repeat. When you done with the feature, run it through something like PEX. It will often highlight things you never thought of. Be sensible about which issues to fix.
For existing code, I'll use code coverage to help me find features I do not have tests for. I comment out the code, write the test (which fails), uncomment the code, verify test passes and repeat. If needed, I'll refactor the code to simplify testing. PEX can also help.
Pay close attention to pain points, as it highlights areas that should be refactored. For example, if you have a controller that uses ObjectContext/IDbCommand/IDbConnection directly for data access, you may find that you require a database to be configured etc just to test business conditions. That is my hint that I need an interface to a data access layer so I can mock it and simulate those business conditions in my controller. The same goes for registry access and so forth.
But be sensible about what you write tests for. The value of TDD diminishes at some point and it may actually cost more to write those tests than it is to give it to someone in India to manually test.
I'm starting to learn acceptance testing, and I want to test an ASP.NET MVC app using SpecFlow. Currently I'm writing tests for CRUD scenario
I know it's a completely noobie question, but is it practical to test every edge case in my SpecFlow scenarios? I have a form to fill, it has about 15 fields, do I need to test that, for example, "if field A is valid and field B is not, I should see this validation message on the screen"? Or it should be sufficient to write "if form is correctly filled, I should see "Task added" message"? Should I unit-test model validation separately in this case?
Thanks ahead
I tend to look for higher-level behavior in scenarios or acceptance tests than I do with unit tests. Particularly, I'm looking for behavior that's valuable to users or other stakeholders.
In this case, the high-level behavior might be phrased as "the user gets feedback on how to fill in the form". You can then just use one or two examples of how the user gets that feedback, and check that the validation message appears.
You can then put the logic around the specific messages in a unit test.
Acceptance tests, especially if you're using BDD scenarios, aren't really tests. They're examples of how to use the system so that you can use those examples to drive conversation with your business stakeholders, and I would tend to phrase them in the same granularity and terms that the business stakeholder is interested in. BDD isn't a substitute for manual testing, though it can certainly help to reduce the burden.
We are running a pretty big website, we have some critical legacy code there we want to have well covered.
At the same time we would like to have a report of the features we are currently supporting and covered. And also we want to be sure we really cover every possible corner case. Some code paths are critical and will need many more tests even after achieving 100% coverage.
As we are already using rspec and rspec has "feature" and "scenario" keywords, we tried to make a list using rspec rather than going for cucumber but I think this question can be applied to any testing tool.
We want something like this:
feature "each advertisement will be shown a specified % of impressions"
scenario "As ..."
This feature is minimal from the point of view of managers but huge in the code. It involves a backend tool, a periodic task, logic in the models and views in backend and front end.
We tried to divide it like this:
feature "each creative will be shown a specified % of impressions"
context "configuration"
context "display"
scenario "..."
context "models"
it "should ..."
context "frontend"
context "display"
scenario "..."
context "models"
it "should ..."
Configuration takes place in another tool, display would contain integration tests and models would contain unit test.
I repeat myself but the idea is sto assure that the feature is really finished(including building the configuration tool) and 100% tested.
But looking at this file, it is not integration, nor unit test not even belong to any particular project.
Definitely there should be a better way of managing this.
Any experiences, resources, ideas you can share to guide us ?
The scenario you're describing is a huge reason why BDD is so popular. It forces you to write code in a way that's easy to test. Having said that, you're obviously not going to go back and rewrite the entire legacy application. There are a few things you should consider though:
As you go through each section of the application, you should ask yourself 'Will it be harder to refactor than to write tests for this?'. Sometimes refactoring before writing tests just cannot be avoided.
Testing isn't about 100% coverage, it's about 100% confidence. As you mentioned, you plan on writing more tests even when you have 100% coverage. This is because you're going for confidence. Once you're confident in a piece of code, move on. You can always come back to it at a later time.
From my experience, Cucumber is easier for tests that cover a large portion of the application. I think the reason for this is that writing out the tests in plain english makes you think of things you wouldn't have otherwise. It also allows you to focus on the behavior instead of the code and can make refactoring a less daunting task.
You don't really get much out of adding tests to existing code if you never touch that code again. Start with testing the code you want to make changes (i.e. refactor) to first.
I also recommend the book Rails Test Prescriptions, specifically one of the last chapters called "Testing a Legacy Application".
So, I've been working on a new project at work, and today had a coworker bring up the idea to me that my exceptions and even returned error messages should be completely localized. I thought maybe that was a good idea, but he said that I should only error return error codes. I personally don't like the error code idea a lot as it tends to make other programmers either
To reuse error codes where they don't fit because they don't want to add another one
They tend to use the wrong error codes as there can get to be so many defined.
So my question is what doe everyone else do to handle this situation? I'm open for all sorts of suggestions including those that think error codes are the way to go.
There may be cultural differences, according to your coding language ?
In Java for example, numerical errors codes are not used much ...
Concerning exceptions, I believe it is just a technical tool.
What is important is wether your message is targeted at a user, or a developper.
For a user, localizing messages is important, if several languages appears, or to be able to change the messages without recompiling (to customize between clients, to adapt to changing user needs ..).
In my projects, our culture is to use (java) enums to handle all collections of fixed values.
Errors are no different.
Enums for errors could provide :
strong typing (you can't pass something else to a method that expect an error code)
simple localisation (a simple utility method can find automatically the message corresponding to each one, using for example "SimpleClassName"."INSTANCE_NAME" pattern ; you could also expose the getMessage() method on each enum, that delegates the implementation to your utility method)
verification of your localized files (your unit tests could loop for each language on the code and the files, and find all unmatched values)
error level functionnality (we use the same levels as for logging : fatal, error, warn ; the logging decisions are very easily implemented then !).
to allow for easy finding of the appropriate error by other developpers, we use several enums (possibly in the same package), classifying the errors according to their technical or functionnal domain.
To adress your two concerns :
Adding one only requires adding an instance to an enum, and a message in the localisation file (but the tests can catch the later if forgotten).
With the classification in several enums, and possibly the javadoc, they are guided to use the correct error.
I wouldn't be using error codes for localization. There may be good reasons to use error codes (e.g. to be able to test which specific kind of error occurred), but localization is not one of those reasons. Instead, use the same framework that you use for the rest of the message localization also for exceptions. E.g. if you use gettext everywhere else, also use it in exceptions. That will make life easier for the translator.
You can include an error code in an exception, thereby getting the best of both.
One frequent cause of error with old-style function-return error codes was failure to check the error code before continuing with subsequent code. An exception cannot be implicitly ignored. Eliminating a source of error is a good thing.
An error code allows:
Calling code to distinguish between different kinds of errors.
Messages to be constructed by UI components when errors occur in non-UI, non-localized code.
A list of errors in the code that may be useful when writing user documentation or troubleshooting guides.
A few guidelines I have found useful:
Only UI architectural layers construct and localize messages to the user.
When non-UI layers communicate errors via exceptions, those may optionally carry error codes and additional fields useful to the UI when constructing a message.
In Java, when error codes are defined in a UI layer Enum, the error message format strings can be accessed through the enumerators themselves. They may contain symbols for additional data carried in the error.
In Java, include argument index in format specifiers in the original language's format strings, so that translators can move dynamic data around in localized messages.