We have a web application that is sufficiently complicated that it's hard to accurately simulate production load.
One of our coping mechanisms has been to make sure that the hardware in the test environment is always slower that our production hardware, so load and performance problems will be more pronounced.
Are other folks out there doing this, what other strategies are you using?
I have a 7-year-old 600mHz Celeron laptop with 256MB RAM that I keep around for performance profiling. For testing network latency, you could get a delay proxy (or write one trivially); it gets a packet, waits X milliseconds, then passes it on.
Fiddler can help you test low-bandwidth -- can simulate latency of slower networks.
Hook up testmaker. Do some manual tests and let their proxy write the test script by recording what you did manually. Automate using the script(s).
http://www.pushtotest.com/
Open source and free.
Related
Thanks for looking this question, I have an application which reads from JMS Queue and processes the mesages and POST the processed message to external http service. What will be best way to load test using gatling.
I can simulate load on queue using gatling.jms. How to verify POST to external service.
Load testing with Gatling is a fairly complex affair to do it right. I've done it enough to know some of the pitfalls so here is some insight that may be useful:
you want to test over the network and you want the latency to be minimal so that delays due to network latency are minimized/nullified and so that the results show how quickly incoming HTTP requests can be handled/responded to. For this reason, if your application is in the cloud in europe-east, say, you want to run your tests from the same location. If your requests were coming from us-west, there'd be a big delay in routing the requests from the wrong side of the US which could introduce big variations in the response times to/from your application.
Remove all other load from your service. If you can't remove load because you're hoping to test against a live application, then you need to make another deployment to test against that has no active load
Load tests should run for (in my experience) 45 minutes as a minimum to verify your service can handle the load. Reason for this being that it can take time for an unbearable load to accumulate on the server... so you may run at 33req/s which is fine for 40 minutes, but when run for 45-60 mins, its just long enough that the balance between what your application can cope with, vs. what causes catastrophic failure is tipped towards failure.
Notes:
You don't need to test to destruction but it is sometimes a useful metric to be aware of. I find using a binary search strategy works well here to get peak load relatively quickly.
What you should test is that your application can handle the load you expect it to receive in a worst case scenario; Different organisations have different tolerances for how much load they expect their applications to be able to cope with. At some places I've worked they've used a lot of optimisations to minimise load directly to their servers, but if those protections fail, the server is expected to handle 10x more traffic than the usual load. At other places, those same optimisations were not in place, instead there were be disaster recovery systems available, ready to pick up when the main app fails. In this case the application only needed to be able to handle 2x the peak load (as observed by assessing logs/metrics for the past year).
I work predominantly with garbage collected languages on the JVM. I'm aware there are now Zero Garbage Collection designs/capabilities which could help minimize the effects of a buildup of GC tasks... so there are almost always optimisations you can make either with language/memory settings, database indexing, or within your application itself, or the strategies you employ to perform a task effectively, before you start changing the hardware.
Peak load can be assessed from logs/metrics systems
I am building a basic chat application for customer support of a website. It works flawlessly in development on local server. I pushed the changes to the server but it is behaving extremely slow. The application in itself works fast but the pub/sub to acioncable channels is slow.
I am using nginx, puma for webserver and redis for pub/sub. I have four channels and two of them have heavy client side (coffee.erb file).How can I reduce time for actioncable channels? How can I debug what is causing the lag?
Thanking you in advance. If any code is required please mention in the comments of the question and I would add it to the question.
The most common cause for things running way more slowly on a server than locally is because they don't have nearly the same amount of RAM and start swapping. Just like an app being ridiculously slow on older phones.
In one case, the system swaps in and out memory, in the other case, the app swaps in and out resources itself (often implicitly through resource caches provided by the API).
The effect is the same: massive I/O overhead that doesn't exist on your development system / modern phone, leading to runtime behavior that is slower by several orders of magnitude.
I'm using savon gem to interact with a soap api. I'm trying to send three parallel request to the api using parallel gem. Normally each request takes around 13 seconds to complete so for three requests it takes around 39 seconds. After using parallel gem and sending three parallel requests using 3 threads it takes around 23 seconds to complete all three requests which is really nice but I'm not able to figure out why its not completing it in like 14-15 seconds. I really need to lower the total time as it directly affects the response time of my website. Any ideas on why it is happening? Are network requests blocking in nature?
I'm sending the requests as follows
Parallel.map(["GDSSpecialReturn", "Normal", "LCCSpecialReturn"], :in_threads => 3){ |promo_plan| self.search_request(promo_plan) }
I tried using multiple processes also but no use.
I have 2 theories:
Part of the work load can't run in parallel, so you don't see 3x speedup, but a bit less than that. It's very rare to see multithreaded tasks speedup 100% proportionally to the number of CPUs used, because there are always a few bits that have to run one at a time. See Amdahl's Law, which provides equations to describe this, and states that:
The speedup of a program using multiple processors in parallel computing is limited by the time needed for the sequential fraction of the program
Disk I/O is involved, and this runs slower in parallel because of disk seek time, limiting the IO per second. Remember that unless you're on an SSD, the disk has to make part of a physical rotation every time you look for something different on it. With 3 requests at once, the disk is skipping repeatedly over the disk to try to fulfill I/O requests in 3 different places. This is why random I/O on hard drives is much slower than sequential I/O. Even on an SSD, random I/O can be a bit slower, especially if small-block read-write is involved.
I think option 2 is the culprit if you're running your database on the same system. The problem is that when the SOAP calls hit the DB, it gets hit on both of these factors. Even blazing-fast 15000 RPM server hard drives can only manage ~200 IO operations per second. SSDs will do 10,000-100,000+ IO/s. See figures on Wikipedia for ballparks. Though, most databases do some clever memory caching to mitigate the problems.
A clever way to test if it's factor 2 is to run an H2 Database in-memory DB and test SOAP calls using this. They'll complete much faster, probably, and you should see similar execution time for 1,3, or $CPU-COUNT requests at once.
That's actually is big question, it depends on many factors.
1. Ruby language implementation
It could be different between MRI, Rubinus, JRuby. Tho I am not sure if the parallel gem
support Rubinus and JRuby.
2. Your Machine
How many CPU cores do you have in your machine, you can leverage this using parallel process? Have you tried using process do this if you have multiple cores?
Parallel.map(["GDSSpecialReturn", "Normal", "LCCSpecialReturn"]){ |promo_plan| self.search_request(promo_plan) } # by default it will use [number] of processes if you have [number] of CPUs
3. What happened underline self.search_request?
If you running this in MRI env, cause the GIL, it actually running your code not concurrently. Or put it precisely, the IO call won't block(MRI implementation), so only the network call part will be running concurrently, but not all others. That's why I am interesting about what other works you did inside self.search_request, cause that would have impact on the overall performance.
So I recommend you can test your code in different environments and different machines(it could be different between your local machine and the real production machine, so please do try tune and benchmark) to get the best result.
Btw, if you want to know more about the threads/process in ruby, highly recommend Jesse Storimer's Working with ruby threads, he did a pretty good job explaining all this things.
Hope it helps, thanks.
I have 357 tests (534 assertions) for my app (using Shoulda). The whole test suite runs in around 80 seconds. Is this time OK? I'm just curious, since this is one of my first apps where I write tests extensively. No fancy stuff in my app.
Btw.: I tried to use in memory sqlite3 database, but the results were surprisingly worse (around 83 seconds). Any clues here?
I'm using Macbook with 2GB of RAM and 2GHz Intel Core Duo processor as my development machine.
I don't feel this question is rails specific, so I'll chime in.
The main thing about testing is that it should be fast enough for you to run them a lot (as in, all the time). Also, you may wish to split your tests into a few different sets, specifically things like 'long running tests' and 'unit tests'.
One last option to consider, if your database setup is time consuming, would be to create your domain by restoring from a backup, rather than doing a whole bunch of inserts.
Good luck!
You should try this method https://github.com/dchelimsky/rspec/wiki/spork---autospec-==-pure-bdd-joy- using spork to spin up a couple of processes that stay running and batch out your tests. I found it to be pretty quick.
It really depends on what your tests are doing. Test code can be written efficiently or not in exactly the same way as any other code can.
One obvious optimisation in many cases is to write your test code in such a way that everything (or as much as possible) is done in memory, as opposed to many read/writes to the database. However, you may have to change your application code to have the right interfaces to achieve this.
Large test suites can take some time to run.
I generally use "autospec -f" when developing, this only runs the specs that have changed since the last run - makes it much more efficient to keep your tests running.
Of course, if you are really serious, you will run a Continuous Integration setup like Cruise Control - this will automate your build process and run in the background, checking out your latest building and running the suite.
If you're looking to speed up the runtime of your test suite, then I'd use a test server such as this one from Roman Le Négrate.
You can experiment with preloading fixtures, but it will be harder to maintain, and, IMHO, not worth it's speed improvements (20% maximum I think, but it depends)
It's known that SQLite is slower than mysql/pgsql, excepting very small, tiny DBs.
As someone already said, you can put mysql (or other DB) datafiles on some kind of RAMDisk (I use tmpfs on linux).
PS: we have 1319 Rspec examples now, and it runs for 230 seconds on C2D-3Ghz-4GRam, and I think it's fine. So, yours is fine too.
As opposite to in-memory SQLite, you can put a MySQL database on RAMDISK (on Windows) or on tmpfs on Linux.
MySQL has a very efficient buffering, so putting database in memory does not help a lot until you update a lot of data really often.
More significant is the way of test isolation and data preparation for each test.
You can use transactional fixtures. That means that each test will be wrapped into transaction and thus next test will start at the initial point.
This is faster than cleaning up the database before each test.
There are situations when you want to use both transactions and explicit data erasing, here is a good article about it: http://www.3hv.co.uk/blog/2009/05/08/switching-off-transactions-for-a-single-spec-when-using-rspec/
We are using TestComplete from AQTime to test the GUI at client with our Client/Server application. It is compiled with Delphi 2007. The source of client is about 1.4 millions sourcelines. The hardware is a Intel dualcore 2.13 Mhz, 2 GB RAM using Windows XP Pro.
I compile the application with all debug options and also link in TCOpenApp, tcOpenAppClasses, tcPublicInfo, tcDUnitSupport as described in documentation to make it an Open Application. The resulting exe-file is about 50 MB.
Now when running the testscript and it works, but running very very slow. The CPU is running at 100 % and it is a bit frustrating to change the testscript because of the slowness. I have turned off all desktop effects like rounded window corners. No desktop background.
Anyone else with the same experience or even an solution ?
Your problem probably lies in the fact you compiled with debug info and are using the tcXXX units, resulting in an enormous amount of objects being created.
A transcript from AutomatedQA message boards
Did you compile it in debug mode? We have an app that when compiled in
Debug mode is slow when used with TC. This is because of the enormous # of
objects in it. If we compile w/o debug but with the TC enabler(s),
everything is fine.
and this one might help to
A couple of areas where you can
increase speed.
If you are just using record and
playback, then look into replacing the
.Keys("xxx") calls to .wText = "xxx".
The Keys function will use the ms
delay between keystrokes, while wText
just forces the text overwrite
internally.
The second suggestion (which you
likely have already looked at) is
Tools->Default Project
Properties->Project->Playback, setting
the delays to 100 ms, 5 ms, and 5 ms
to keep the pauses to a minimum.
As for the object properties, yes, TC
loads them all. You can force this
with a process refresh on yor
application, so that the data is
forced into being available without a
load delay when called. This might
help with reducing the appearance of
delay.
Edit:
We also have been evaluating TestComplete and also encountered this performance problems. I would be very interested to know if and how you've finally solved them.
That said, I think it is a product with great potential and can really help you with organizing all of your unit, integration and GUI tests.
Now when running the testscript and it works, but running very very slow. The CPU is running at 100 % and it is a bit frustrating to change the testscript because of the slowness. I have turned off all desktop effects like rounded window corners. No desktop background.
Anyone else with the same experience or even an solution ?
I recommend that you try changing the TCP ports that TestComplete use for remote connections. You can change them in the Network Suite Options Dialog. For example, you can set 6100-6102 ports. Does this help? A similar issue was described in the TC 9.20 consuming high 98% cpu SmartBear forum thread.