I'm building a test harness for my Delphi 2009 app. Testing the logic is fairly simple. Making sure the forms work properly is proving a bit more complicated. I'd like a way to simulate real user input, to open a form, make it think there's a user typing certain things and clicking in certain places, and make sure it reacts correctly. I'm sure there's a way to do this, I just don't know what it is. Does anyone know how to do it?
DUnit has GUITesting.pas whicih extends testing so you can send clicks, keys and text to controls on form, but that is about it.
Last year there where mentions of Zombie GUI testing framework that was used internaly by CodeGear developers, but nothing since Steve left for Falafel.
TestComplete is a good choice. Another commercial option for GUI testing is SmarteScript:
well for .net there's NUnitForms for GUI testing a win application.
don't know any open source for delphi though.
Test Complete can test delphi forms but it's not free.
There's 2 parts to this, firstly how do you automate the GUI, and secondly how do I then 'test' whether its working/not working.
Firtsly: To automate the GUI on windows try using AutoIT. Its a free tool for controlling windows interfaces, sending keyboard input events etc. http://www.autoitscript.com/autoit3/
Secondly: Testing is a big field, and I won't try and give you a whirlwind tour. But the mechanics of driving the GUI and testing the results could be handled using AutoIT's built in Basic like language or by using it in conjunction with a language like Ruby and TestUnit (rubys built-in unit testing framework).
If there is nothing Deliphi-specic out there and you need a quick solution try some easy to learn scripting solutions like AutoIt.
For a bit more sophisticated scripting, you might have a look on Scripted GUI Testing with Ruby.
But be aware, that you should not test too much functionality via the GUI, because such tests are very likely to break. If you end up with too much GUI testing you may need to rethink the design: Decouple logic from the GUI and test the logic directly with some xUnit framework.
Also have a look on a similar question about windows forms test automation.
It seems like DUnit has some gui-testing functionality: delphiextreme.com
Not exactly an answer to your question, but there is a very good page (IMHO of course) about GUI Architectures by Martin Fowler, featuring the "Humble View" architecture as the last entry, which is geared specifically towards test-driven software development. Worth looking into.
This will of course not help you with the task of testing whether all controls are wired correctly and handle all necessary events, but it should help to minimize the amount of GUI code that does need testing.
OpenCTF is good for you.
Quote:
OpenCTF is a test framework add-on for Embarcadero Delphi® which
performs automatic checks of all components in Forms (or DataModules).
It provides an easy way to build automatic quality checks for large
projects where many components have to pass repeated tests.
Adding OpenCTF tests to a DUnit test suite requires only a few lines of code.
Writing your own custom component tests needs only a few seconds.
OpenCTF is based on the DUnit open source test framework and extends
it by specialized test classes and helper functions.
Please head here to download.
Related
We are looking to initiate a bdd-style approach, inspired by Gojko Adzic's specification by example. Implementation is in java and devs are already writing junit tests.
Key requirement is that specifications (acceptance tests) can be written, read and maintained by non-developers. The project will run as an agile team - so it's fine if devs have to instrument the specs. However, I don't want developers, testers or domain experts having to read or write something that looks like code.
So far I've looked at FitNesse, Concordion and various others (e.g. Spock). I've rejected spock and similar tools because they target developers as the primary audience. FitNesse seems to meet most of the requirements.
Concordion is probably current favourite however: specs looks cleaner and simpler.
So my question (actually three):
Any suggestions for other tools I should look at?
Has anyone been successful with using concordion (or another tool) in this way?
Is concordion still actively developed/supported? Difficult to tell from website and most related SO questions are several years old.
Thanks.
I've worked with a number of teams implementing Specification by Example with Concordion. We train our whole team up to write Concordion specifications in HTML. Only a small subset of HTML is required, so we can train a newbie in about 30 minutes. Typically we have the testers writing the HTML specification, with the BA or Scrum Master sometimes writing them.
We've used Eclipse (Web Page Editor) for editing the HTML. This works well, except that Concordion requires valid XHTML, and Eclipse does not allow HTML to be validated as XHTML. This mostly shows with <br> tags being used rather than <br/>. We cover this off in training. We also train the whole team in the use of source control. By using Eclipse, we have a single user interface for editing and source control. We also find that having the team using the same IDE is a step on the journey to a cross-functional team.
I know of another team where the BA is writing the specifications using a Mac-based HTML editor.
Concordion is actively maintained, with rapid responses on the mailing list (Yahoo) and the issues list. The Concordion codebase is stable. The active development over the last year or so has focussed on the extension mechanism, allowing users to add commands and listeners (eg. for capturing screenshots on test failure).
Also take a look at Cucumber and JBehave, both of which allow specifications to be written in plain text.
If you choose to use FitNesse, it may also be worth looking at Slim, which sits behind FitNesse in place of Fit. It provides slightly different table formats to Fit, and I've found it suits BDD much better.
Just to update the topic, you can also consider jnario.
When unit-testing a component, the following question occured to me:
There are a number of mouse-related event handlers. I see two possibilties to test these handlers:
Simulate mouse events using Windows API calls.
Use the protected hack to access protected event handlers and call them directly.
I know that unit testing is normally restricted to the interface of a class (which also means that tests don't have to be changed when class internals change), but is this scenario worth an exception?
How do you usually handle mouse events when unit-testing components?
Personally, I think you need an architectural change to facilitate automated user interface testing. Reasons why are well formulated in this article: http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil
The Delphi Magazine once had an interesting article on automated testing of user interfaces from code (without a specific gui testing tool that is). Taking a bit longer to find than I expected and may not be available online. Will update my answer when/if I find it.
The article is "Creating Easily Testable User Interfaces" by Julian Bucknall (DevExpress) and was published in issue 120 of "The Delphi Magazine". Unfortunately the article is no longer online. You would have to buy the total collection of The Delphi Magazine: a 1 GB USB stick with all issues and all codes ever published in The Delphi Magazine. Well worth the 36 GBP! (And no, I do not get a commission).
Usually you are supposed to write your code in a way you can test it, so you test the methods you call by the mouseevent without simulating the mouse event.
In order to do that you need a good separated GUI and logical middleware.
Can you maby provide some Code or more information on the funcions you call on the mousevents in your app.
I think you are looking for some automated GUI testing. You could try Test Complete from Smart Bear, for example.
DUnit does offer some basic GUI testing facilities (in the GUITesting unit) but I don't think it may not be comprehensive enough for your needs.
I am looking at SpecFlow examples, and it's MVC sample contains several alternatives for testing:
Acceptance tests based on validating results generated by controllers;
Integration tests using MvcIntegrationTestFramework;
Automated acceptance tests using Selenium;
Manual acceptance tests when tester is prompted to manually validate results.
I must say I am quite impressed with how well SpecFlow examples are written (and I managed to run them within minutes after download, just had to configure a database and install Selenium Remote Control server). Looking at the test alternatives I can see that most of them complement each other rather than being an alternative. I can think of the following combinations of these tests:
Controllers are tested in TDD style rather than using SpecFlow (I believe Given/When/Then type of tests should be applied on higher, end-to-end level; they should provide good code coverage for respective components;
MvcIntegrationTestFramework is useful when running integration tests during development sessions, these tests are also part of daily builds;
Although Selenium-based tests are automated, they are slow and are mainly to be started during QA sessions, to quickly validate that there are no broken logic in pages and site workflow;
Manual acceptance tests when tester is prompted to confirm result validity are mainly to verify page look and feel.
If you use SpecFlow, Cucumber or other BDD acceptance test framework in you Web development, can you please share your practices regarding choosing between different test types.
Thanks in advance.
It's all behaviour.
Given a particular context, when an event occurs (within a particular scope), then some outcome should happen.
The scope can be a whole application, a part of a system or a single class. Even a function behaves this way, with inputs as context and the output as outcome (you can use BDD for functional language as well!)
I tend to use Unit frameworks (NUnit, JUnit, RSpec, etc.) at a class or integration level, because the audience is technical. Sometimes I document the Given / When / Then in comments.
At a scenario level, I try to find out who actually wants to help read or write the scenarios. Even business stakeholders can read text containing a few dots and brackets, so the main reason for having a natural language framework like MSpec or JBehave is if they want to write scenarios themselves, or show them to people who will really be put off by the dots and brackets.
After that, I look at how the framework will play with the build system, and how we'll give the ability to read or write as appropriate to the interested stakeholders.
Here's an example I wrote to show the kind of thing you can do with scenarios using simple DSLs. This is just written in NUnit.
Here's an example in the same codebase showing Given, When, Then in class-level example comments.
I abstract the steps behind, then I put screens or pages behind those, then in the screens and pages I call whatever automation framework I'm using - which could be Selenium, Watir, WebRat, Microsoft UI Automation, etc.
The example I provided is itself an automation tool, so the scenarios are demonstrating the behaviour of the automation tool through demonstrating the behaviour of a fake gui, just in case that gets confusing. Hope it helps anyway!
Since acceptance tests are a kind of functional tests, the general goal is to test your application with them end-to-end. On the other hand, you might need to consider efficiency (how much effort is to implement the test automation), maintainability, performance and reliability of the test automation. It is also important that the test automation can easily fit into the development process, so that it supports a kind of "test first" approach (to support outside-in development).
So this is a trade off, that can be different for each situation (that's why we provided the alternatives).
I'm pretty sure, that today the most widely fitting option is to test at the controller layer. (Maybe later as UI and UI automation frameworks will evolve, this will change.)
I admit that I have almost none experience of unittesting. I did a try with DUnit a while ago but gave up because there was so many dependencies between classes in my application.
It is a rather big (about 1.5 million source lines) Delphi application and we are a team that maintain it.
The testing for now is done by one person that use it before release and report bugs. I have also set up some GUI-tests in TestComplete 6, but it often fails because of changes in the application.
Bold for Delphi is used as persistance framework against the database.
We all agree that unittesting is the way to go and we plan to write a new application in DotNet with ECO as persistance framework.
I just don't know where to start with unittesting...
Any good books, URL, best practice etc ?
Well, the challenge in unit testing is not the testing itself, but in writing testable code. If the code was written not thinking about testing, then you'll probably have a really hard time.
Anyway, if you can refactor, do refactor to make it testable. Don't mix object creation with logic whenever possible (I don't know delphi, but there might be some dependency injection framework to help in this).
This blog has lots of good insight about testing. Check this article for instance (my first suggestion was based on it).
As for a suggestion, try testing the leaf nodes of your code first, those classes that don't depend on others. They should be easier to test, as they don't require mocks.
Writing unit tests for legacy code usually requires a lot of refactoring.
Excellent book that covers this is Michael Feather's "Working Effectively with Legacy Code"
One additional suggestion: use a unit test coverage tool to indicate your progress in this work. I'm not sure about what the good coverage tools for Delphi code are though. I guess this would be a different question/topic.
Working Effectively with Legacy Code
One of the more popular approaches is to write the unit-tests as you modify the code. All new codes gets unit tests, and for any code you modify you first write its test, verify it, modify it, re-verify it, and then write/fix any tests that you need due to your modifications.
One of the big advantages of having good unit test coverage is being able to verify that the changes you make don't inadvertently break something else. This approach allows you to do that, while focusing your efforts on your immediate needs.
The alternate approach I've employed is to develop my unit tests via Co-Ops :)
When you work with legacy code, mock objetcs are really usefull to build unit tests.
Take a look at this question regarding Delphi and mocks: What is your favorite Delphi mocking library?
For .Net unittesting read this : "The Art of Unit Testing: with Examples in .NET"
About best pratices :
What you said is right : Sometimes, it's difficult to write unit tests because of the dependancy between classes...
So write unit tests just after or just before ;-) the implementation of the classes. Like this, if you have some difficulties to write the tests, maybe it means you have a design problem !
Is there a proper way yet to unit test views with the aspx view engine?
I've been playing with various ways that will let me get a parseable string as a result like :
view.RenderView(viewContext);
But I'm not having any luck so far.
Most of what I've read strays into integration test territory. Integration test overlap a fair bit but don't allow me to use dummy data to check the output, which is the main thing I'm trying to achieve by unit testing views.
Have you checked out Selenium? It is directed at Web UI testing and is discussed in Hunt and Thomas' Pragmatic Unit Testing in C# with NUnit. It uses NUnit although it does so in a way that is really more of a system test than a Unit test (simply because you have to bring up the web app and navigate to your page to test it). Sorry that I didn't go down this route in my earlier answer but I was in a bit of a rush yesterday morning!
There is also NUnitAsp but that project has been essentially abandoned. It works and was widely praised in its day but I'm loathe to recommend anything that is no longer under active development.
What I have done is to create objects to "drive" the user interface and placed them all into a "BusinessObjects" dll. The interface essentially pulls its results from these objects when it is time to create the interface. I then Unit test against the Business Objects using standard NUnit unit tests. I'm afraid that your question might be a bit more specific to the MVC framework but I hope that this gives you some food for thought.
My advice is to not bother. Stick to testing the model and the controller. Checking for specific HTML is only going to lead to a fragile test base, that will break the moment you make the slightest change to the layout.