kubernetes not able to pull image from spark master host - docker

Question

I have 3 node[host a,host b, host c] kubernetes cluster(version 1.12.2). I am trying run spark-pi example jar as mentioned in kubernetes document.
Host a is my kubernetes Master. >> kubectl get nodees list all the three nodes.
I have built the spark docker image using whats provided in spark 2.3.0 binary folder.
>> sudo ./bin/docker-image-tool.sh -r docker.io/spark/spark -t spark230 build
I got the message the image got built successfully.
>> docker images ls
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/spark/spark spark230 6a2b645d7efe About an hour ago 346 MB
docker.io/weaveworks/weave-npc 2.5.0 d499500e93d3 7 days ago 49.5 MB
docker.io/weaveworks/weave-kube 2.5.0 a5103f96993a 7 days ago 148 MB
docker.io/openjdk 8-alpine 97bc1352afde 2 weeks ago 103 MB
k8s.gcr.io/kube-proxy v1.12.2 15e9da1ca195 2 weeks ago 96.5 MB
k8s.gcr.io/kube-apiserver v1.12.2 51a9c329b7c5 2 weeks ago 194 MB
k8s.gcr.io/kube-controller-manager v1.12.2 15548c720a70 2 weeks ago 164 MB
k8s.gcr.io/kube-scheduler v1.12.2 d6d57c76136c 2 weeks ago 58.3 MB
k8s.gcr.io/etcd 3.2.24 3cab8e1b9802 7 weeks ago 220 MB
k8s.gcr.io/coredns 1.2.2 367cdc8433a4 2 months ago 39.2 MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 10 months ago 742 kB
> ./bin/spark-submit
--master k8s://https://<api-server>:<api
> server port> --deploy-mode cluster --name spark-pi
> --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=5 --conf
> spark.kubernetes.container.image=spark/spark:spark230 --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark
> local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar
When I submit above command , it gives proper output sometimes. Other times it throws below error.
> code = Unknown desc = repository docker.io/spark/spark not found:
does not exist or no pull access, reason=ErrImagePull
When I debugged it further, it looks like, whenever node name: host b or host c its throwing above error message.
When node name : host a , then it runs fine. Looks like other nodes are unable to locate the image.
Questions:
Should I install spark on all nodes & build docker image on all nodes?
Is it possible to pass the image reference from single node [host a] to other nodes? i.e how to make other nodes refer the same image from host a.

Answer 1

Yes, you need to build Spark image in all the nodes. You can write the wrapper script to invoke the 'rebuild-image.sh' on all the nodes as below:
for h in hostnames; do
rsync -av /opt/spark ${h}:/opt
ssh ${h} /opt/spark/rebuild-image.sh

Answer 2

You can always save docker image as tar file and then copy that tar file to other host and load the image there.
To save docker image as tar file:
sudo docker save -o <path for generated tar file> <image name>
Now copy your tar file to other host using scp or some other copy tools. And load the docker image using:
sudo docker load -i <path to image tar file>
Hope this helps