Cannot Run Selenium Chromedriver on M1 Mac - docker

Question

I was able to run the following Docker file on my Mac with Intel chip, but I am getting errors when I run it on a Mac with M1. I then tried docker run --init --platform=linux/amd64 -e SPRING_PROFILES_ACTIVE=dev -e SERVER_FLAVOR=LOCAL_DEV -p 8080:8080 monolith-repo and docker buildx build --platform=linux/amd64 -t monolith-repo .. That got the Docker container to run but I get the following error when trying to call selenium:
org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure.
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:561)
at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:230)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:151)
at org.openqa.selenium.chromium.ChromiumDriver.<init>(ChromiumDriver.java:108)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:104)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:91)
at com.flockta.monolith.scraping.MakeNewWebpageScraperVersion2.getWebDriver(MakeNewWebpageScraperVersion2.java:176)
at com.flockta.monolith.scraping.MakeNewWebpageScraperVersion2.getWebpage(MakeNewWebpageScraperVersion2.java:52)
at com.flockta.monolith.job.ScrapingDataPipelineJob.getHtmls(ScrapingDataPipelineJob.java:308)
at com.flockta.monolith.job.ScrapingDataPipelineJob.processWebPage(ScrapingDataPipelineJob.java:168)
at com.flockta.monolith.job.ScrapingDataPipelineJob.processResult(ScrapingDataPipelineJob.java:153)
at com.flockta.monolith.job.ScrapingDataPipelineJob.processResult(ScrapingDataPipelineJob.java:59)
at com.flockta.monolith.job.AbstractScrapingPipelineJob.processResults(AbstractScrapingPipelineJob.java:57)
at com.flockta.monolith.job.AbstractScrapingPipelineJob.process(AbstractScrapingPipelineJob.java:30)
at com.flockta.monolith.ScheduledJobController.runDataJob(ScheduledJobController.java:71)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.openqa.selenium.WebDriverException: Driver server process died prematurely.
Build info: version: '4.1.1', revision: 'e8fcc2cecf'
System info: host: '1c789f0433ca', ip: '172.17.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '5.10.76-linuxkit', java.version: '15.0.2'
Driver info: driver.version: ChromeDriver
at org.openqa.selenium.remote.service.DriverService.start(DriverService.java:226)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:98)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:543)
... 26 common frames omitted
My full Docker file is:
FROM maven:3.6.3-openjdk-15
#Chrome
ARG CHROME_VERSION=98.0.4758.102-1
ADD google-chrome.repo /etc/yum.repos.d/google-chrome.repo
RUN microdnf install -y google-chrome-stable-$CHROME_VERSION \
&& sed -i 's/"$HERE\/chrome"/"$HERE\/chrome" --no-sandbox/g' /opt/google/chrome/google-chrome
## ChromeDriver
ARG CHROME_DRIVER_VERSION=98.0.4758.102
RUN microdnf install -y unzip \
&& curl -s -o /tmp/chromedriver.zip https://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip \
&& unzip /tmp/chromedriver.zip -d /opt \
&& rm /tmp/chromedriver.zip \
&& mv /opt/chromedriver /opt/chromedriver-$CHROME_DRIVER_VERSION \
&& chmod 755 /opt/chromedriver-$CHROME_DRIVER_VERSION \
&& ln -s /opt/chromedriver-$CHROME_DRIVER_VERSION /usr/bin/chromedriver
ENV CHROMEDRIVER_PORT 4444
ENV CHROMEDRIVER_WHITELISTED_IPS "127.0.0.1"
ENV CHROMEDRIVER_URL_BASE ''
EXPOSE 4444
EXPOSE 8080
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar", "-Xmx600m","/app.jar"]
Also, when I try to run docker build without the buildx build --platform=linux/amd64 I get an error:
docker build -t monolith-repo .
[+] Building 12.0s (7/9)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/maven:3.6.3-openjdk-15 0.3s
=> [internal] load build context 0.0s
=> => transferring context: 122B 0.0s
=> [1/5] FROM docker.io/library/maven:3.6.3-openjdk-15#sha256:aac64d9d716f5fa3926e6c8f43c680fa8404faae0b8a014c0c9b3d73d2d0f66a 0.0s
=> CACHED [2/5] ADD google-chrome.repo /etc/yum.repos.d/google-chrome.repo 0.0s
=> ERROR [3/5] RUN microdnf install -y google-chrome-stable-98.0.4758.102-1 && sed -i 's/"$HERE\/chrome"/"$HERE\/chrome" --no-sandbox/g' /opt/google/chrome/google-chrome 11.6s
------
> [3/5] RUN microdnf install -y google-chrome-stable-98.0.4758.102-1 && sed -i 's/"$HERE\/chrome"/"$HERE\/chrome" --no-sandbox/g' /opt/google/chrome/google-chrome:
#7 0.232 Downloading metadata...
#7 5.286 Downloading metadata...
#7 9.705 Downloading metadata...
#7 11.53 error: Could not depsolve transaction; 1 problem detected:
#7 11.53 Problem: conflicting requests
#7 11.53 - package google-chrome-stable-98.0.4758.102-1.x86_64 does not have a compatible architecture
#7 11.53 - nothing provides libm.so.6(GLIBC_2.2.5)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides ld-linux-x86-64.so.2(GLIBC_2.2.5)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides libdl.so.2(GLIBC_2.2.5)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides librt.so.1(GLIBC_2.2.5)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides libpthread.so.0(GLIBC_2.3.2)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides libpthread.so.0(GLIBC_2.12)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides libpthread.so.0(GLIBC_2.3.4)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides ld-linux-x86-64.so.2()(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
#7 11.53 - nothing provides libpthread.so.0(GLIBC_2.3.3)(64bit) needed by google-chrome-stable-98.0.4758.102-1.x86_64
------
executor failed running [/bin/sh -c microdnf install -y google-chrome-stable-$CHROME_VERSION && sed -i 's/"$HERE\/chrome"/"$HERE\/chrome" --no-sandbox/g' /opt/google/chrome/google-chrome]: exit code: 1
I note two things during the build:
'package google-chrome-stable-98.0.4758.102-1.x86_64 does not have a compatible architecture'
I am using 'chromedriver_linux64.zip (but it never gets to that stage) albeit https://chromedriver.storage.googleapis.com/ shows that there is a chromedriver_mac64_m1 as weak,
Is there a solution to get Chrome working on my local machine? Specifically, I need to be able to run this on my Mac and also deploy it to AWS. I think the AWS can be solved via buildx build --platform=linux/amd64 but I do not know how to get this to run locally. Any ideas?

Answer 1

The high level issue is that linux running on amd (Intel) may not be available on arm yet. Specifically see https://github.com/SeleniumHQ/docker-selenium/issues/1076 and the great work that jamesmortensen did with the https://hub.docker.com/u/seleniarm repo (and specifically https://hub.docker.com/r/seleniarm/standalone-chromium/tags ). To use it, do:
FROM seleniarm/standalone-chromium:4.1.1-alpha-20220119
ENV CHROMEDRIVER_PORT 4444
ENV CHROMEDRIVER_WHITELISTED_IPS "127.0.0.1"
ENV CHROMEDRIVER_URL_BASE ''
EXPOSE 4444
EXPOSE 8080
EXPOSE 5005
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
# For Testing
ENTRYPOINT ["java","-jar", "-Xmx600m","/app.jar"]
Java code is then:
return new ChromeDriver(service, getChromeOptions());
and
private ChromeOptions getChromeOptions() {
ChromeOptions chromeOptions = new ChromeOptions();
// User agent is required because some websites will reject your request if it does not have a user agent
chromeOptions.addArguments(String.format("user-agent=%s", USER_AGENT));
chromeOptions.addArguments("--log-level=OFF");
chromeOptions.setHeadless(true);
List<String> arguments = new LinkedList<>();
arguments.add("--disable-extensions");
arguments.add("--headless");
arguments.add("--disable-gpu");
arguments.add("--no-sandbox");
arguments.add("--incognito");
arguments.add("--disable-application-cache");
arguments.add("--disable-dev-shm-usage");
chromeOptions.addArguments(arguments);
return chromeOptions;
}
Note that the standalone is chromium , not chromedriver but this works because chromedriver is based off of chromium.
The root cause of this is that many packages (for example https://www.ubuntuupdates.org/package/google_chrome/stable/main/base/google-chrome-stable) do not have arm versions yet (they only have amd versions which is Intel based).
As for Docker files, I suggest having two docker files for now (much like zwbetz-gh's comment on Dec 28th, see https://github.com/SeleniumHQ/docker-selenium/issues/1076). To build the arn version you would do:
docker build -f DOCKER_FILE_ARN -t your_tag. Although I have still to test it, for the non-arn file, you would do: docker buildx build --platform=linux/amd64 -f DOCKER_FILE_AMD -t your_tag.