Cannot start Spark on Kubernetes - docker

Question

I am trying to set up Spark on Kubernetes on Mac. I have followed this tutorial web pages and it looks so straightforward for me to understand.
Below is the Dockerfile.
# base image
FROM java:openjdk-8-jdk
# define spark and hadoop versions
ENV SPARK_VERSION=3.0.0
ENV HADOOP_VERSION=3.3.0
# download and install hadoop
RUN mkdir -p /opt && \
cd /opt && \
curl http://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz | \
tar -zx hadoop-${HADOOP_VERSION}/lib/native && \
ln -s hadoop-${HADOOP_VERSION} hadoop && \
echo Hadoop ${HADOOP_VERSION} native libraries installed in /opt/hadoop/lib/native
# download and install spark
RUN mkdir -p /opt && \
cd /opt && \
curl http://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop2.7.tgz | \
tar -zx && \
ln -s spark-${SPARK_VERSION}-bin-hadoop2.7 spark && \
echo Spark ${SPARK_VERSION} installed in /opt
# add scripts and update spark default config
ADD common.sh spark-master spark-worker /
ADD spark-defaults.conf /opt/spark/conf/spark-defaults.conf
ENV PATH $PATH:/opt/spark/bin
After building the Docker image I ran the following commands but the pod doesn't start.
$ kubectl create -f ./kubernetes/spark-master-deployment.yaml
$ kubectl create -f ./kubernetes/spark-master-service.yaml
spark-master-deployment.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: spark-master
spec:
replicas: 1
selector:
matchLabels:
component: spark-master
template:
metadata:
labels:
component: spark-master
spec:
containers:
- name: spark-master
image: spark-hadoop:3.0.0
command: ["/spark-master"]
ports:
- containerPort: 7077
- containerPort: 8080
resources:
requests:
cpu: 100m
spark-master-service.yaml
kind: Service
apiVersion: v1
metadata:
name: spark-master
spec:
ports:
- name: webui
port: 8080
targetPort: 8080
- name: spark
port: 7077
targetPort: 7077
selector:
component: spark-master
To trace the problem, I ran the kubectl describe... command and got the following result.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 45s default-scheduler Successfully assigned default/spark-master-fc7c95485-zn6wf to minikube
Normal Pulled 21s (x3 over 44s) kubelet, minikube Container image "spark-hadoop:3.0.0" already present on machine
Normal Created 21s (x3 over 44s) kubelet, minikube Created container spark-master
Warning Failed 21s (x3 over 43s) kubelet, minikube Error: failed to start container "spark-master": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/spark-master\": stat /spark-master: no such file or directory": unknown
Warning BackOff 8s (x3 over 42s) kubelet, minikube Back-off restarting failed container
It seems that the container didn't start but I didn't figure out why the pod does not start correctly even though I only followed the instruction on the web page.
Below is the GitHub URL that the web page gives me a guide for configuring Spark on Kubernetes.
https://github.com/testdrivenio/spark-kubernetes

Answer 1

I assume you are using Minikube.
For minikube make the following changes:
Eval the docker env using: eval $(minikube docker-env)
Build the docker image: docket build -t my-image
Set the image name as only "my-image" in pod specification in your yaml file.
Set the imagePullPolicy to Never in you yaml file. Here is the example:
apiVersion:
kind:
metadata:
spec:
template:
metadata:
labels:
app: my-image
spec:
containers:
- name: my-image
image: "my-image"
imagePullPolicy: Never

Answer 2

It seems like you didn’t copy the scripts developed by the blogger which are in this project, where in the image there is this command ADD common.sh spark-master spark-worker /, so your image misses the script you need to run the master (you will have the same problem with the workers), you can clone the project then build the image, or use the image published by the blogger mjhea0/spark-hadoop.
Here you are trying to setup a spark standalone cluster on Kubernetes, but you can use Kubernetes itself as a spark manager, where spark in the release 3.1.0 announced that Kubernetes officially become a spark cluster manager (it was experimental since the release 2.3), here is the official documentation, you can also use spark-on-k8s-operator, developed by Google, to submit the jobs and manage them on your Kubernetes cluster.