Are there any recommended Load Test tools / services that are able to cycle through AWS Application Load Balancer logs stored in S3 preferably utilising the time stamps to perform piano roll type functionality?
aws-log-replay seems to be something you're looking for, it can replay requests with defined concurrency.
With regards to more or less popular load testing tools I can only think of Apache JMeter with Access Log Sampler which support out of box access log files from Tomcat, Weblogic, Reisin and SunOne, however you can come up with your own implementation of Generator class or dynamically populate HTTP Request sampler fields using JSR223 PreProcessor like it's described in Stop Making Assumptions! Learn How to Replay Your Production Traffic With JMeter guide.
Actually I don't think you will be able to produce realistic load by replaying your access logs, it might work for something simple like static content, however if your application assumes authentication, sessions, complex workflows, etc. - I'm afraid your "replay" attempt will got stuck at login page.
So instead of trying to replay complex scenarios from the logs I would suggest sticking to the load testing tool of your choice and create it from scratch. Access logs can be used to identify workload distribution (like X % of users are normally doing this, Y % are doing that, etc.) and anticipated concurrency (like at X time we had Y online users).
Related
I created an application that uses Watir to automate logging in and perform a couple of functions within a site.
Right now it's written 100% purely in ruby classes that I was just executing in irb, but I want to put it into a Rails application and put it online. I haven't been able to find much information about using something like Capybara or Watir for anything other than testing. Is this because of how slow they are or is it a capabilities issue?
Would I be able to run a background process that opens a browser with Watir and performs a few functions for each user in production?
Another question I have is how to keep the session over a longer period of time. There are two sites that require 2FA that my app logs into. If I wanted to log in and perform a function once an hour with a Watir browser, I could create it as a background process (if that works). But when the process is done the browser closes and when the background process runs again in an hour it requires 2FA again.
My other worry is speed. If I have 50 users that all need to run a Watir browser at the same time I imagine that will be slow. I am not worried as much about speed as long as they run and collect the data and perform the few actions we need, but how it will effect the applications integrity.
WATIR is specifically designed as a testing tool.. WATIR stands for Web Application Testing In Ruby. It's design is centered around interacting with a browser the way a user would, effectively simulating the same actions a user would take when using a site. That it would be sub-optimal for other tasks is to be expected. Due to scraping and testing having very similar activities, there a a number of people that use watir for that task, but it is not designed for that purpose, and it is unlikely that the WATIR developers would ever add features specific to data scraping verses testing.
With the things you are contemplating, you should ask yourself if you are doing the equivalent of using a socket wrench as a hammer, and if there might be a better tool you could use.
If the sites you are interacting with support an API, then that would be the preferred way to interact with them, to get information from the site. If that is not supported, you may want to consider looking at other gems that would let you request the site HTML or parse the HTML directly (e.g. Nokogiri)
You should also inspect the terms of service for the sites you are interacting with (if you don't own them) to ensure that there are not prohibitions against using 'robots' or other automated means to access the site. If so, then using Watir in the way you propose may end up getting you banned from access to the site if the pattern of your access is obviously the result of an automated process.
This is actually done more often than people think. Maybe not specifically with Watir, but running a browser automation task in a job. The jobs should be queued and run asynchronously, preferably in a different process than your main web app.
I wrote about this strategy here: https://blogstephenarifin.wordpress.com/2018/08/23/integrating-a-third-party-without-an-api-using-rails-5-and-heroku/
If you find yourself having to use Watir then the best way is to use it to render the page (for example in headless mode for javascript), save it and then use Nokogiri to process it. Apis are suggested a lot by people who don't and can't find a use for scraping but at times is necessary and perfectly legit (you may even be scraping your own data). Apis are not a universal option.
Second you should probably regulate its use to a background job. If you have end users (and you really shouldn't have many simultaneous users) many services inform customers data will be available in a few hours to a few days
I am building a website with rails on AWS and I am trying to determine the best ways to stress-test while also getting some idea of the cost I will be paying by user (very roughly). I have looked at tools like Selerium and I am curious if I could do something similar with Postman.
My objects are:
Observe what kind of load the server would be under during the test, how the cpu and memory are affected.
See how the load generated would affect the cpu cycles on the system that would generate cost to me by AWS.
Through Postman I can easily generate REST calls to my rails server and simulate user interaction, If I created some kind of multithreaded application that would make many calls like to the server, would that be an efficient way to measure these objectives?
If not, is there a tool that would help me either either (or both) of these objectives?
thanks,
You can use BlazeMeter to do the load test.
This AWS blog post show you how you can do it.
I am a newbie to rails. I have used feature flags when i was in java world. I found that there are a few gems in rails (rollout and others) for doing it. But how to turn a feature on/off on the fly in rails.
In java we can use a mbean to turn features on the fly. Any idea or pointers on how to do this? I dont want to do a server restart on my machines once a code is deployed.
Unless you have a way of communicating to all your processes at once, which is non-standard, then you'd need some kind of centralized configuration system. Redis is a really fast key-value store which works well for this, but a database can also do the job if a few milliseconds per page load to figure out which features to enable isn't a big deal.
If you're only deploying on a single server, you could also use a static YAML or JSON configuration file that's read before each request is processed. The overhead of this is almost immeasurable.
I've been working on a rails project that's unusual for me in a sense that it's not going to be using a MySQL database and instead will roll with mongoDB + Redis.
The app is pretty simple - "boot up" data from mongoDB to Redis, after which point rails will be ready to take requests from users which will consist mainly of pulling data from redis, (I was told it'd be pretty darn fast at this) doing a quick calculation and sending some of the data back out to the user.
This will be happening ~1500-4500 times per second, with any luck.
Before the might of the user army comes down on the server, I was wondering if there was a way to "simulate" the page requests somehow internally - like running a rake task to simply execute that page N times per second or something of the sort?
If not, is there a way to test that load and then time the requests to get a rough idea of the delay most users will be looking at?
Caveat
Performance testing is a very broad topic, and the right tool often depends on the type and quality of results that you need. As just one example of the issues you have to deal with, consider what happens if you write a benchmark spec for a specific controller action, and call that method 1000 times in a row. This might give a good idea of performance of that controller method, but it might be making the same redis or mongo query 1000 times, the results of which the database driver may be caching. This also ignores the time it'll take your web server to respond and serve up the static assets that are part of the request (this may be okay, especially if you have other tests for this).
Basic Tools
ab, or ApacheBench, is a simple commandline tool that you can use to test the throughput and speed of your app. I usually go to this first when I want to send a thousand requests at a web server, or test how many simultaneous requests my app can handle (e.g. when comparing mongrel, unicorn, thin, and goliath). Because all requests originate from the same server, this is good for a small number of requests, but as the number of requests grow, you'll be limited by the resources on your testing machine (network stack, cpu, and maybe memory).
Benchmark is a standard ruby class, and is great for quickly spitting out some profiling information. It can also be used with Test::Unit and RSpec. If you want a rake task for doing some quick benchmarking, this is probably the place to start
mechanize - I like using mechanize for quickly scripting an interaction with a page. It handles cookies and forms, but won't go and grab assets like images by default. It can be a good tool if you're rolling your own tests, but shouldn't be the first one to go to.
There are also some tools that will simulate actual users interacting with the site (they'll download assets as a browser would, and can be configured to simulate several different users). Most notable are The Grinder and Tsung. While still very much in development, I'm currently working on tsung-rails to make it easier to automate rails load testing with tsung, and would love some help if you choose to go in this direction :)
Rails Profiling Links
Good overview for writing performance tests
Great slide deck covering most of the latest tools for profiling at various levels
We currently use Hp SiteScope for monitoring synthetic transactions across some of our web apps. This works pretty well except for the licensing cost for each synthetic transaction makes it prohibitive to ensure adequate coverage across our applications.
So, an alternative would be to use SiteScope's URL monitoring which can basically call a URL and then provide some basic checks for the certain strings. With that approach, I'd like to create a page that either calls a bunch of pages or try to tap into a MSTest group somehow to run tests.
In the end, I'd like a set of test cases that can be used against multiple environments to be used for production verification, uptime, status, etc.
Thanks,
Matt
Have you taken a look at System Center Operations Manager 2007?
I'm just getting started, but it appears to do what you are describing in your question.
We are looking to monitoring our data center and the a web application...from the few things I have found on the web it is going to fit our need.
Update
I've since moved to Application Insights. A great overview can be found here, https://azure.microsoft.com/en-us/documentation/articles/app-insights-monitor-web-app-availability/
There are two methods one can use, a simple ping, or record a multi-step synthetic user "experience". Basically you act as a user, and using IE and a Visual Studio Web Test project you record navigating around your site and upload that file to Azure.
For example, I record logging in, navigating a few pages, and then logging out. As long as all of those events happen in a timely manner the site is in a good operating state.
If the tests fail, take too long to respond for example, I'll get an email alerting me something isn't exactly right.