Selenium Headless Tests running very slow on EC2 instance - docker

Question

Thanks for reading my post. I've got a problem that I'd like to pick your brains about. I have many automated Selenium tests that are written in Java. They leverage JUnit for testing. I am tasked with optimizing the run time of these tests on an Amazon EC2 CentOS server. The tests are pulled from a Github repository and get stored inside a Docker Container on the EC2 instance. Similarly, these tests also run from the client's laptop through Eclipse using Cucumber. On the laptop, the tests take a fraction of the time that they do on the EC2 instance. The EC2 instance and laptop have the same amount of RAM and plenty of CPU processing power to run the tests, so I would think that the tests on the EC2 instance wouldn't take nearly as long as they do. Both the EC2 instance and the laptop have to go through a proxy to get to the Internet as well. I have made sure this part was set up correctly, as I had to set HTTP_PROXY, HTTPS_PROXY, and no_proxy to specific IP addresses.
In this section, I will provide the code that the chromedriver gets instantiated with whenever it is called. I will also provide the exact commands that I am using to run the tests. Please see below:
Here is the method that is called when it gets instantiated:
public static ChromeOptions setupChromeOptions() {
ChromeOptions options = new ChromeOptions();
Map<String, Object> p refs = new HashMap<String, Object>();
prefs.put("profile.default_content_setting_values.notifications", 2);
options.setHeadless(true);
options.setProxy(null);
options.setExperimentalOption("prefs", prefs);
options.addArguments("--disable-dev-shm-usage");
options.addArguments("--disable-extensions");
options.addArguments("--disable-gpu");
options.addArguments("--headless");
options.addArguments("--no-proxy-server");
options.addArguments("--no-sandbox");
options.addArguments("--proxy-bypass-list=*");
options.addArguments("--proxy-server=");
options.addArguments("--proxy-server='direct://'");
options.addArguments("--start-maximized");
options.addArguments("--window-size=1280,720");
return options;
}
Here are the commands that are ran through Jenkins on the EC2 instance:
#!/bin/bash
cd /home/ec2-user/Git/REPOS_NAME # Navigate to where the repos are at on the EC2.
git fetch --all
git reset --hard origin/master
git pull # Pull down the latest.
sudo docker container ls # Display the running containers.
sudo docker exec 0967d39b1967 bash -c 'cd /Selenium/ ; rm -rf REPO_ROOT_FOLDER/ ; exit' # Delete the automated test folder so that it gets copied over fresh.
sudo docker cp /home/ec2-user/Git/REPO_NAME/REPO_ROOT_FOLDER 0967d39b1967:/Selenium/ # Copy the latest from the Github Repository to the Docker container.
sudo docker exec 0967d39b1967 bash -c 'cd /Selenium/REPO_ROOT_FOLDER/ANOTHER_FOLDER/driver ; chmod 0777 chromedriver ; cd .. ; mvn clean test' # Activate the Testrunner that is enabled in pom.xml. This kicks off the automated tests.
Finally, here are the version numbers and OS details of what I'm running:
Amazon EC2: CentOS 6.10
Docker Container OS: Ubuntu 16.04
Selenium: selenium-server-standalone-3.141.59.jar
Chromedriver: 84.0.4147.30
Google Chrome: 84.0.4147.105
Java: OpenJDK 1.8.0_252
Cucumber: 2.0.0
I cannot provide links to an HTML Page that I'm scraping because it's all inside of a Salesforce environment.
EC2 Specs:
AWS T2.XLarge
4 vCPUs
64-Bit
16 GB RAM
54 CPU Credits per hour
Laptop Specs:
Intel Core i7-8665U # 1.90GHz, 2112 MHz, 4 cores, 8 Logocal Processors
16 GB RAM
Windows 10
Group of Tests 1:
EC2: 50 minutes
Laptop through Eclipse: 20 minutes
Group of Tests 2:
EC2: 2 hours 43 minutes
Laptop through Eclipse: 1 hour 15 minutes
There are two breakdowns of the timings for two groups of tests. As you can see, the times off of the EC2 are much longer than that of that laptop.
Finally, here is a log of the set of tests that take 50 minutes to run. I had to change some of the directory and path names to protect the client.
Started by user Scott-IM
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on EC2_INSTANCE in workspace /var/lib/jenkins/workspace/Run mvn test
[Run mvn test] $ /bin/bash /tmp/jenkins166122467778007381.sh
Fetching origin
FIPS mode initialized
HEAD is now at 506f4ef Setting proxy to null for Chromeoptions
FIPS mode initialized
Already up to date.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0967d39b1967 IMAGE_NAME "/Selenium/setup.sh" 4 hours ago Up 4 hours 0.0.0.0:8080->8080/tcp CONTAINER_NAME
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[INFO] Scanning for projects...
[INFO]
[INFO] --------------< test:automation >--------------
[INFO] Building test-automation 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # automation ---
[INFO] Deleting /Selenium/automation/automation/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # automation ---
[WARNING] Using platform encoding (UTF8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Selenium/automation/automation/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # automation ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # automation ---
[WARNING] Using platform encoding (UTF8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Selenium/automation/automation/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # automation ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF8, i.e. build is platform dependent!
[INFO] Compiling 111 source files to /Selenium/salesforce-automation/salesforce-automation/target/test-classes
[WARNING] /Selenium/automation/automation/src/test/java/com/test/mapper/OIDP_Flow/OIDP_Email_Auto_Approval_Flow_2.java:[797,128] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/com/test/mapper/OIDP_Flow/OIDP_Email_Auto_Approval_Flow_2.java:[1842,130] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/com/test/shared/ExtentReporter.java:[126,33] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/com/test/shared/ExtentReporter.java:[127,32] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/com/test/mapper/rsiaa/HD_ISO_VSC_Auto_Approval_of_Resubmitted_Si.java:[4977,95] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/com/test/pages/aao/TrackingRecord.java:[27,39] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/com/test/pages/aao/TrackingRecord.java:[117,84] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/com/test/pages/aao/TrackingRecord.java:[156,92] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/com/test/pages/aao/TrackingRecord.java:[195,84] unmappable character for encoding UTF8
[WARNING] /Selenium/automation/automation/src/test/java/restApiPOC/PrivacyRestApiConnection.java: Some input files use unchecked or unsafe operations.
[WARNING] /Selenium/automation/automation/src/test/java/restApiPOC/PrivacyRestApiConnection.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.19:test (default-test) # automation ---
[WARNING] The parameter forkMode is deprecated since version 2.14. Use forkCount and reuseForks instead.
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Running com.test.runner.TestRunner2
Starting ChromeDriver 84.0.4147.30 (48b3e868b4cc0aa7e8149519690b6f6949e110a8-refs/branch-heads/4147#{#310}) on port 23815
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
[1604696323.888][SEVERE]: bind() failed: Cannot assign requested address (99)
Nov 06, 2020 8:58:44 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Frame count :3
Frame count :4
Frame count :6
Newly Generated Service Item Number is :15355359
Frame count :3
Frame count :3
Clicked on new service item :15355359
Frame count :2
Frame count :2
Frame count :4
Frame count :3
Frame count :3
1 Scenarios (1 passed)
33 Steps (33 passed)
44m57.274s
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2,698.236 sec - in com.salesforcetest.runner.TestRunner2
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 45:05 min
[INFO] Finished at: 2020-11-06T21:43:40Z
[INFO] ------------------------------------------------------------------------
Finished: SUCCESS
If anything was unclear or didn't make sense, please let me know and I will address the issue. I have looked all over the Internet for solutions and haven't found anything yet. I'm almost at a loss of what to do, as I'm not entirely sure what the exact cause of the slowness is. If anyone could provide some clues or things to try, I would be more than willing to give it a shot.
Thanks much for any and all help.

Answer 1

Not a solution per se, but throwing some things out there worth trying.
Since you're comparing Chrome run times I would advise trying to keep the versions in sync. That's tough on windows because you often have no control when Chrome updates. This isn't to say it would solve your problem but it would help eliminate a variable.
Chrome (and chromedriver for that matter) goes through a decent amount of changes that not being on the same version CAN and HAS made a difference for me.
Here are a few of the chrome options I've had to resort to using over the past couple years to get things running smoothly. I noticed 3 you didn't have.
--disable-extensions -- You never know what they might have on their local version of chrome.
--dns-prefetch-disable -- script would hang without this in older versions of chromedriver. https://bugs.chromium.org/p/chromedriver/issues/detail?id=402#c128
--enable-features=NetworkService,NetworkServiceInProcess -- script freezing https://groups.google.com/forum/m/#!topic/chromedriver-users/ktp-s_0M5NM[21-40]