I am currently developing a dockerised selenium grid to run around 800 functional tests ( see my post ) and I was hoping I could launch up to 800 nodes with one hub in a single grid [inside a BIG aws instance]. The idea would be to run all my tests at the same time and finish in [let's say] 10 minutes.
As I understand from this post that is not possible.
I launched 20 nodes and my tests are only populating the 10. That is only 10 firefox instances.
Are there restrictions built in the selenium server or the hub's docker image?
I was hoping that the ease of scaling dockerised services would take the selenium grid capabilities to a whole new level...
Is the dockerised implementation of selenium grid 'stuck' to 10 nodes ?
You should be able to run as many concurrent tests as the system can maintain. Without knowing anything about your current platform I have no idea what is limiting to just 10 nodes. In my current stack my test runner has a max fork property that limits the number of concurrent tests. You might want to look into what your test runner has set as the max number of concurrent tests. I know of no limit for selenium grid for tests it would be silly to limit it at 10.
Related
I am developing automated tests for one of our GUI based app using Pytest framework. I've created a docker image with series of tests for a particular GUI functionality and it is stored in AWS ECR as an image.I've also setup an AWS Batch computing env with a cron schedule to trigger the tests (image) at a particlar time/day which is working fine.
I've couple of questions regarding this:
Is there a way to trigger the tests from AWS without using the cron schedule? This is to enable business users with necessary AWS rights so that they can run the tests independently without waiting for the cron to run the tests.
Secondly, what is the best way to run automation tests for more than one GUI functionalities (pages)? There are about 15 different types of pages within the app that needs to be automated for testing. One way is to create 15 different images to test them and store them in ECR. But it sounds little inefficient way of doing things. Is there a better alternative like creating just one image for these 15 different pages. If so, how can I selectively run tests for a particular GUI page.
Thank you.
The answer to your first question is no, you can't manually trigger a CRON scheduled batch job. If you wanted user's to run the tests you would need to switch to event driven jobs and have the user's create events that trigger the job - drop a file into S3, send a message to an SQS queue etc. You could also wrap your batch job in a State Machine, which is then trivial to manually execute
I've couple of questions regarding this: Is there a way to trigger the
tests from AWS without using the cron schedule? This is to enable
business users with necessary AWS rights so that they can run the
tests independently without waiting for the cron to run the tests.
I think you are asking if a user can run this AWS Batch Job Definition manually in addition to the cron scheduled process?
If so the answer is yes, either with access to the Batch management console or using the CLI (or if you have some other GUI application with the SDK). They would need an IAM user with a role permissions for Batch, ECS, and other AWS resources.
Secondly, what is the best way to run automation tests for more than one GUI functionalities (pages)? There are about 15 different types of pages within the app that needs to be automated for testing. One way is to create 15 different images to test them and store them in ECR. But it sounds little inefficient way of doing things. Is there a better alternative like creating just one image for these 15 different pages. If so, how can I selectively run tests for a particular GUI page.
I would look into continuous integration testing methods. This is the problem those systems are designed to solve.
I'm using Locust to load test my web servers. I'm running Locust in distributed mode. The worker nodes are written in Java, and use the Locust/Java port using locust4j. The master node and the worker nodes are containerized, our orchestrator is Kubernetes. When I want to spin up more workers, I'm doing it from there.
The problem that I'm running into is that no matter how many users I add, or worker nodes I add, I can't seem to generate more than ~8000 RPM. This is confirmed by the Locust web frontend, as well as the metrics I'm collecting from my web server.
Does anyone have any ideas why this is happening?
I've attached an image of timings I've collected. The snapshots are from running the load test for 60 seconds, I'm timing it from a stopwatch.
The usual culprit in these kinds of situations is your servers can't handle more than that. In my experience, the behavior you'll see client side as the servers get overwhelmed is you'll start to see a slow but steady increase in response times. This is one big reason why Locust includes those in the metrics it shows you.
Based on what I'm seeing in your screenshots, this is most likely the case for you. You have some very low minimum times but your average, median, and 90%iles are a lot higher than your minimums; your maximums are very significantly higher than those. Without seeing your charts I can't know for sure but that's a big red flag.
For more things to look out for, check out this question in the FAQ (especially see the list of server stats to investigate):
https://github.com/locustio/locust/wiki/FAQ#increase-my-request-raterps
I have a scenario where I have N numbers of nodes and N number of tests, and these tests will be distributed to the nodes. My nodes have a label Windows.
Here's an example:
I have a pipeline job that will manage the distribution of the tests to the nodes. I set my pipeline job to run 10 tests on 10 VMs having the label Windows and it will run smoothly. However, one of my requirements is to concurrently run that pipeline job. The problem I might encounter If I have 10 tests on VMs 1-10 in the first run of the job, and run another job for 5 tests for VMs 11-15, given that I am using the Windows label, there might be a possibility that Jenkins will assign the test to VMs 1-10 but should run on VMs 11-15 or vice versa.
The solution I came up with is to dynamically change the label of the VMs from one of the jobs to a unique label that will only be used for that Job. Unfortunately, I still don't know how to do that.
Basically, I just want to logically group my nodes via label on demand in my pipeline script.
I searched all throughout the internet and yet I still wasn't able to find a solution that fits my needs.
Kindly help me with this. Also, I am open to using a different approach.
My app is ASP.NET MVC.
I am using a lot of parallel processing on my local machine (8 cores) and things are running very smoothly.
But when I roll to Azure Medium Instance (2 cores), during testing, I get weird slow downs and program stops working sometimes.
Is there a way to emulate 1, 2 or another number of cores to match what will happen in production environment?
I guess you could try setting the process affinity for the development server. To do that, (assuming Windows 7) open task manager, right-click the server process and select Set Affinity... and select the cores you want it to run on.
Just managed to find a way around #Dai's answer above, but it means you'll have to start the development server yourself. This .bat file runs notepad.exe using two cores (you can verify that by checking its affinity within task manager):
start /affinity 0x03 notepad.exe
The 0x03 specifies core 1 and core 2. The help command was a bit confusing, but it seems it combines those to get the result (as in, 1 + 2 = 3, unless I've misunderstood it). So if you need to change to a different set of cores, keep that in mind.
#JohnH's method seems the best, but you'd have to do it every time w3wp.exe runs on your machine. An alternative is to restrict your operation to 2 threads (or 4 if using hyperthreading). We need more information about how you're processing information in parallel.
A parralelisable problem should just run 4 times as fast on an 8-core machine as opposed to the 2-core Azure VM, but getting 'program stops working' situations means you've got a bug there.
I'm not sure if this will fully help the situation, but you can set the MaxDegreeOfParalellism for each core. This way you can limit the threads that run.
I have a test pack consisting of more than 3000 scenarios.now the problem is that when i run the scenarios in one shot ..it takes approx 10 hours to complete,i want to divide the scenarios in 4 blocks,each of approx 750 scenarios and wanted to run them parallel in different windows/terminal(VMware).is there a workaround ???
This question has a selenium tag, so (assuming that's accurate) Selenium-Grid would be an option for setting up a distributed parallel testing environment.
orde mentions Selenium Grid and that's one piece of the puzzle. The main benefit you get out of that is that you only have to identify your Server Hub in your code when creating a new selenium instance.
The next thing would be to actually execute your 4 blocks of 750 scenarios at once. I'd recommend using a CI tool like Jenkins to accomplish that and you'll have your results together on Jenkins' web gui.