I need to load test a Vaadin application.
I found this link explaining how to use JMeter with Vaadin however I can't get it to work. I suspect the problem are my MenuItems since Vaadin 6.4 ChangeLog mentions that some change was made to MenuItems in order to make Vaadin MenuItems HTTP-Level tester friendly. However upgrading the Vaadin 6.4 and 6.5 yielded no better results.
Here is my problem, I need to load test my Vaadin application to prove it will work with at least 50 concurrent users. However the async nature of Vaadin means that most tools have no idea when a response has taken place or whether it is correct. How can I script a test and emulate 50 users? Any help will be much appreciated I've tried a lot with Selenium and other tools but their scripts never execute reliably. Has anyone load test a vaadin app? If so how?
Selenium does not work out of the box with Vaadin apps, that partially is why TestBench exists.
The easiest solution to your problem is to add a sleep to each request in jMeter. It's ugly and needs some experimenting to get the sleep times good enough. And it's not completely fool-proof. It might however be enough for you to prove that 50 concurrent users works.
Another thing is to add some kind of client-side rendering and use JS to check if there is still an existing connection from client to server. TestBench uses something like this for implementing a waitForVaadin() function that checks if the request is complete. Here's some discussion: http://vaadin.com/forum/-/message_boards/message/236584#_19_message_236823
Vaadin uses many technical ids for the components, so in order to automate a realistic navigation you need to correlate those ids.
You can read this blog which illustrates the use of a commercial plugin for Open source Apache JMeter that does auto correlation for you.
You just record and you can replay, all correlation is done by plugin.
Disclaimer : We are provider of this solution
Related
This is sort of an open-ended question/request (hope that's allowed).
On my team we are using Karate API testing for our project, which we love. The tests are easy to write and fairly understandable to people without coding backgrounds. The biggest problem we're facing is that these API tests have some inherent degree of flakiness (since the code we're testing makes calls to other systems). When running the tests locally on my machine, it's easy to see where the test failed. However, we're also using a Jenkins pipeline, and when the tests fail in Jenkins it's difficult to see why/how they failed. By default we get a message like this:
com.company.api.OurKarateTests > [crossdock] Find Crossdock Location.[1:7] LPN is invalid FAILED
com.intuit.karate.exception.KarateException
Basically all this tells us is the file name and starting line of the scenario that failed. We do have our pipeline set up so that we can pass in a debug flag and get more information. There are two problems with this however; one is that you have to remember to put in this flag in every commit you want to see info on; the other is that we go from having not enough information to too much (reading through a 24MB file of the whole build).
What I'm looking for is suggestions on how to improve this process, preferably without making changes to the Jenkins pipeline (another team manages this, and it will likely take a long time). Though if changing the pipeline is the only way to do this, I'd like to know that. I'm willing to "think outside the box" and entertain unorthodox solutions (like, posting to a slack integration).
We're currently on Karate version 0.9.3, but I will probably plan to upgrade to 0.9.5 as part of this effort. I've read a bit about the changes. Would the "ExecutionHook" thing be a good way to do this? I will be experimenting with this on my own a bit.
Have other teams/devs faced this issue? What were your solutions? Again we really love Karate, just struggling with the integration of it to Jenkins.
Aren't you using the Cucumber Reporting library as described here: https://github.com/intuit/karate/tree/master/karate-demo#example-report
If you do - you will get an HTML report with all traffic (and anything you print) in-line with the test-steps, and of-course error traces, and most teams find this sufficient for build troubleshooting, there is no need to dig through logs.
Do try upgrade as well, because we keep trying to improve the usefulness of the logs, and you may see improvements if you had failed in a JS block or karate-config.js.
Else, yes the ExecutionHook would be a good thing to explore - but I would be really surprised if the HTML report does not give you what you need.
Vaadin and Apache Click seem to be equally good, which one should I choose for developing my web application. Or rather, what are the Pros and Cons of each framework.
I'm a committer of Apache Click but hopefully you will find my opinion objective.
I don't know Vaadin at all, just had a cursory look at their website and examples and mission statement.
Apache Click and Vaadin are meant for different problem spaces. Apache Click is targeted at traditional enterprise web applications while Vaadin targets the Rich Internet Application (RIA) space.
Apache Click is a traditional share nothing, stateless framework. Unlike traditional action based web frameworks Click provides a slightly higher level of abstraction by using Pages and Components. At the of the day Click isn't a revolution. Its just one of many approaches of doing web application development. In my opinion its a very good approach, both from a developer and maintainer point of view. (In enterprise environments the developer and maintainer are often not the same individual, so I differentiate between the two roles).
However developers sometimes have requirements that cannot be satisfied by traditional web applications, which is where RIA comes in. They provide a richer user experience (think desktop) but this does not come for free. RIA comes at a cost in terms of complexity, productivity and time-to-market. It makes sense though, if the requirements are for a richer experience, you need to do more in order to deliver that experience.
My advice here is: think carefully about your requirements. Don't simply assume that RIA is better, there is a price to pay, so make sure you get return for your investment.
Lastly, if your requirement is for RIA, then you should compare Vaadin with Flex. If your requirement is for a more traditional web application then compare Click with Struts, Stripes, Wicket, Tapestry etc.
Kind regards
Bob
I use both ;-)
Click is the best for classic web applications (pure html with no/ small/ handmade javascript).
It is very lightweight (small size and stateless), less verbose (you can do your html in html with velocity/freemarker power ;-).
A real php-killer.
Vaadin is the best for desktop like enterprise applications.
It is statefull, heavyweight, verbose, but amazing.
Grails, Play!, Spring MVC is a real step back after Click/Vaadin.
So my advice: use both!
For most of your pages you will use Click, for complicated tasks - Vaadin.
I've been using Click since 2006. I've built various web applications with it in the last 4 years. All apps (B2B) are still going strong and continue to be maintained/added to now. The best things about Click IMO include:
Very little magic and almost no plumbing. It's all just plain java (and jsp/velocity/freemarker take your pick; or alternatively use all 3 if you so choose; I have for some scenarios because it's so easy to).
It's super fast (It doesn't mean your app won't be slow, it just means Click will never be the reason why your app is slow)
It is so easy to integrate your favourite java lib eg. Ibatis SQLMAP, Spring, displaytag, DWR etc.etc.
You don't need to buy a book to understand/use it (although you may need books for all the other 3rd part libs you plan to use).
Most of my code written 4 years ago still works with the current release. That is just awesome.
Click fits in your head. The whole framework that is. It is so simple you don't have too many questions even as a beginner.
The guys that maintain Click are super responsive and very noob friendly.
It's simplicity helps my apps adapt to new/old tech. i.e I can easily do AJAXy stuff if I want or not.
The form features make handling html forms very easy/fast.
The table features make displaying data super easy/fast
Click's features help me write apps very quickly. Makes the need for scaffolding unnecessary imo.
If you want a light weight, fast, easy to understand framework which helps you build applications as fast as you or your team are personally able to, check out click, it will be worth your while.
Before making your choice you should evaluate all options by taking them out on a test drive.
I'm not familiar with Click, and someone should correct me if I'm wrong, but I understand Click basically a technique to link your server side code to web pages. The UI is page-based and you define the UI with components corresponding to HTML elements. Ajax is not (yet) supported, I gather.
Vaadin operates at a higher level of abstraction. With Vaadin you get to concentrate on your business logic, and build your UI naturally with views, layouts and other components (check out the Sampler). You don't need to care about RPC, page transitions, HTML or templates. Vaadin gives you a fully cross-browser compliant Ajax UI that looks and works great out of the box. It's very easy to get started, and the forum is very active (>700 messages last month compared to 96 on the Click mailing list).
Disclaimer: I'm a member of the Vaadin team and have not used Click.
I would choose, and I already did in a few projects Click (over all other frameworks).
The main selling point for me was that I was able to learn Click and be quite productive in one week-end (Vaadin - but other frameworks too took me much more).
The free and available documentation for Click is much better - you can compare the sites yourself (even if the Vaadin site looks cooler, the Click site is simply more useful - at least it was for me).
The live examples are much better - I always look at them as a snippet repository, and just copy what I need from there (since I'm a lazy programmer :) ) - there's a "Page Java" and "Page HTML" link for every example that shows the source code for every available example.
Regarding the message traffic, I don't think that should be a criteria, since having so good free online docs, and the framework being so much simpler, the users simply don't need to ask that much.
Disclaimer: I'm just a simple user that have used both frameworks.
Is Apache wicket an option as well ?
My experience is with jboss/ejb/jquery, but looking at technologies for starting a new project. I've done a quick bit of research myself abd Vaadin seems to be very well regarded.
Surely Apache Wicket is something you can not ignore if you are comparing UI frameworks. ( I am doing same.) I dropped Vaadin because of poor documentation ( maybe I am not a good Googler.)
Currently comparing Wicket, Click, and ZK.
Click and Wicket are ahead so far for the following reasons:
Generates pure HTML instead of rendering UI using JavaScript which happens in GWT, Vaadin, and ZK. You can use HTML5 features then.
Even with native HTML, server-side binding of UI forms is possible.
You don't need to worry about communication with the server. ZK also does this but with their own language ZUML.
I have found server-side memory footprint higher in ZK in the case of components like the ZK Grid. (Memory footprint for Click is not known to me yet.)
Clarification/summary for the question -- we're looking for:
a hosted bug tracking system,
that is as convenient to use as lighthouse/github/launchpad,
can deal with attachments,
integrates email notifications and operations (implies operations in commit messages),
has a script-friendly API,
allows anonymous bug reports, or ones with an email but that do not require setting up an account for submission.
Lighthouse is close but fails on the last point, launchpad is similar, github also doesn't handle attachments. Tender is great for the last point, but fails as a general bug tracking system (and it looks like its open-source version will be limited to basically being a forum).
We looked into a number of applications to install and setup -- but with this range of requirements, they are always coming with a huge cost in terms of investing time in setting up and maintaining a working system.
In our (open-source) project we have been using Gnats for a really long time. It doing what it was designed to do fine, but that's getting to be pretty inconvenient: it's no longer maintained, has features that we never use, and lack features that we'd want to use... It doesn't deal with attachments, has no easy way to perform actions via emails, no integration with commit messages, and a web interface that was designed for 90s browsers. So I've been looking around in an attempt to find something that could replace it, hopefully some hosted solution to avoid the setup/maintenance hassle.
Probably the most impressive tool that I've seen is lighthouse: it has a very nice and practical interface, properly deals with attachments, controllable via emails, and can respond to commands in commit messages. But... It doesn't have any sane way to submit a bug anonymously -- and that's a major requirement, since we need any random user to be able to submit bugs through our IDE. (It seems that there is a possible hack to forward an email faking the From field, but that doesn't work very well -- specifically, the reporter should be included in the followup email exchange.) On the other side, there is the related tender tool, which is very good in that area, but is very basic otherwise -- too basic to serve as a bug tracking system.
There's a whole bunch of other sites that I've tried -- it seems that all of them require submitters to have an account, so they don't work well for our needs; as well as being limited in various other ways (don't deal with attachments, no good email integration, etc etc). It doesn't help that the meta-descriptions of these sites is usually pretty obscure: it took me hours to just figure out what tender/lighthouse are and how they're related, and no site mentions its inability to receive bug reports without registration. (I'm looking only at open-source-friendly sites, since we don't have any kind of budget for such things.)
There's also the option of installing some system locally, but bug tracking systems tend to be monsters that I'd like to avoid configuring and maintaining, if possible.
So the question is: is there anything obvious that I'm missing? Or to make it more concrete: is there a good comparison page somewhere that lays out popular options and their respective features explicitly?
JIRA is free for open source projects. It's far more user friendly than trac and bugzilla, and allows anonymous submissions and plugins. Unfortunately you'll need to host it on your own server, but from personal experience I can tell you that all you need to do is install a database (it can run without; but that's not a good idea) and it basically maintains itself.
Also is there a particular reason why Google Code or Sourceforge issue tracking tools wouldn't work? You don't need to use all their services if you don't want, you could use them purely for issue tracking.
Did you try trac? It is used by many open source projects.
FogBugz is one option. They'll host or you can run it yourself. My company looked at it but ... political considerations ... meant it is not viable here.
Have you looked at this Comparison of issue tracking systems on Wikipedia?
I have also found fixx, by hedgehoglab. Apparently it has the features that you care more:
Get things done
fixx has an intuitive interface to enable quick bug
reporting. Filling in a bug report is
as easy as sending e-mail.
Ability to add multiple attachments to issues allowing you
to attach screenshots and manage
documents related to issues.
Clever notification options to keep relevant people informed while
preventing issue tracker spam.
Also:
It has an open REST API.
I see that you are using Subversion as SCM. There is a Subversion integration with fixx.
Its unique installation requirement is Sun JDK 1.5.0.
It seems free for Open Source Projects and an hosted version is "Coming soon".
Note that I have never used it, so I cannot give any recommendation.
The open source BugTracker.NET has support for the following areas that are giving you problems:
Attachments
Guest login
Email notifications
SVN commit integration
I found it easy to set up, maintain, and tweak. Of course, you might think otherwise if you are not familiar with .NET and have a Windows server available.
You might look at Unfuddle. They do allow an API for the submission of tickets and have your other points covered including attachments.
Take a look at repositoryhosting.com They have ready made solution with trac / svn / git, for you. Comes with all kinds of bells and whitsles, such as Agilo plug-in and auotomatic backup to the amazon S3 bucket of your choice.
The prices are very reasonable.
Also, jumboxes offers a Trac / SVN virtual appliance that you can host in your own environment.
Redmine is a good open source option. You can check an online demo and a list of features.
It's not hosted though. But it's an interesting option.
And you can always check a list of different open source bug tracking alternatives
I've used ZenDesk in the past and it was rather hassle free.
In addition it has an api: http://www.zendesk.com/api.
Moreover I KNOW it can CC whosoever you want it to whenever anything happens.
We too are looking for a new solution.
At present we're using FogBugz, which is painfully slow.
We need our customers to be able to log bugs via email. Tender looks perfect, with the exception that it doesn't have any obviously usable ID fields that we can pass around. Is there a plugin or similar? I could knock up a browser extension to "goto bug id [whatever]" but that seems kludgy for what should surely be a core feature?
What would be a proper way to simulate a large number of requests to test if my web application can handle it?
You could try using Microsoft's WCAT tools. Look here: http://support.microsoft.com/kb/231282
They're free, too. That's always nice.
Depending on your budget, you may be interested in some load testing software designed for this. A Google search brings up all sorts of alternatives. This is probably the best way to do it.
This one has a free trial version and isn't too pricey, but I would recommend shopping around first.
I've used JMeter in the past, and I find it to be very useful for stress/load testing as website, even ones written in ASP.NET (with or without MVC).
In general you would want to (with any tool) write a script of what an average user of your site would do. You may even end up creating multiple of these scripts. Tools like JMeter even allow for a random element to be added to a script. With these scripts created a load testing tool can then simulate as many users as you desire hitting your site.
I would recommend allow JMeter to slowly ramp up the number of concurrent users and watch the response time graph. At the point where the response time starts increasing too highly is at the point where you've hit the maximum number of users (given you scripts) that your site can handle.
ab and httperf are two, more unixy options, if you don't mind delving in that direction.
There's a nice screencast for using httperf by peepcode.
Use the load testing tools from Visual Studio Team System. 2010 if you can get it.
The tools are great to use and provide wonderful instrumentation. There is also a programming model to go with the tools, allowing you to make some very complex testing scenarios possible.
Post the URL on stackoverflow.
Make it sound like a challenge, so lots of people come check it out: "Can you find the hidden performance problem in this app?"
I'm building a test harness for my Delphi 2009 app. Testing the logic is fairly simple. Making sure the forms work properly is proving a bit more complicated. I'd like a way to simulate real user input, to open a form, make it think there's a user typing certain things and clicking in certain places, and make sure it reacts correctly. I'm sure there's a way to do this, I just don't know what it is. Does anyone know how to do it?
DUnit has GUITesting.pas whicih extends testing so you can send clicks, keys and text to controls on form, but that is about it.
Last year there where mentions of Zombie GUI testing framework that was used internaly by CodeGear developers, but nothing since Steve left for Falafel.
TestComplete is a good choice. Another commercial option for GUI testing is SmarteScript:
well for .net there's NUnitForms for GUI testing a win application.
don't know any open source for delphi though.
Test Complete can test delphi forms but it's not free.
There's 2 parts to this, firstly how do you automate the GUI, and secondly how do I then 'test' whether its working/not working.
Firtsly: To automate the GUI on windows try using AutoIT. Its a free tool for controlling windows interfaces, sending keyboard input events etc. http://www.autoitscript.com/autoit3/
Secondly: Testing is a big field, and I won't try and give you a whirlwind tour. But the mechanics of driving the GUI and testing the results could be handled using AutoIT's built in Basic like language or by using it in conjunction with a language like Ruby and TestUnit (rubys built-in unit testing framework).
If there is nothing Deliphi-specic out there and you need a quick solution try some easy to learn scripting solutions like AutoIt.
For a bit more sophisticated scripting, you might have a look on Scripted GUI Testing with Ruby.
But be aware, that you should not test too much functionality via the GUI, because such tests are very likely to break. If you end up with too much GUI testing you may need to rethink the design: Decouple logic from the GUI and test the logic directly with some xUnit framework.
Also have a look on a similar question about windows forms test automation.
It seems like DUnit has some gui-testing functionality: delphiextreme.com
Not exactly an answer to your question, but there is a very good page (IMHO of course) about GUI Architectures by Martin Fowler, featuring the "Humble View" architecture as the last entry, which is geared specifically towards test-driven software development. Worth looking into.
This will of course not help you with the task of testing whether all controls are wired correctly and handle all necessary events, but it should help to minimize the amount of GUI code that does need testing.
OpenCTF is good for you.
Quote:
OpenCTF is a test framework add-on for Embarcadero Delphi® which
performs automatic checks of all components in Forms (or DataModules).
It provides an easy way to build automatic quality checks for large
projects where many components have to pass repeated tests.
Adding OpenCTF tests to a DUnit test suite requires only a few lines of code.
Writing your own custom component tests needs only a few seconds.
OpenCTF is based on the DUnit open source test framework and extends
it by specialized test classes and helper functions.
Please head here to download.