How to create a Distributed spark cluster using Docker - docker

Question

I am trying to create a distributed spark cluster with only one worker using this docker-compose
master:
image: gettyimages/spark:2.0.0-hadoop-2.7
command: bin/spark-class org.apache.spark.deploy.master.Master -h master
hostname: master
container_name: spark-master
environment:
SPARK_CONF_DIR: /conf
SPARK_PUBLIC_DNS: <MASTER IP>
expose:
- 7001
- 7002
- 7003
- 7004
- 7005
- 7077
- 6066
ports:
- 4040:4040
- 6066:6066
- 7077:7077
- 8080:8080
volumes:
- ./conf/master:/conf
- ./data:/tmp/data
- ~/spark/data/:/spark/data/
worker:
image: gettyimages/spark:2.0.0-hadoop-2.7
command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
hostname: worker
container_name: spark-worker
environment:
SPARK_CONF_DIR: /conf
SPARK_WORKER_CORES: 2
SPARK_WORKER_MEMORY: 1g
SPARK_WORKER_PORT: 8881
SPARK_WORKER_WEBUI_PORT: 8081
SPARK_PUBLIC_DNS: <WORKER IP>
links:
- master
expose:
- 7012
- 7013
- 7014
- 7015
- 8881
ports:
- 8081:8081
volumes:
- ./conf/worker:/conf
- ./data:/tmp/data
- ~/apps/sparkapp/worker/data:/spark/data/
But the problem is that the docker daemon is creating the containers on the same machine.Which takes away the whole point of having distributed network.How can I create a distributed spark cluster using docker

Answer 1

If the problem in the same ports for Spark workers, actually you have two options:
Do not expose workers' ports at all - you do not need them for workers to connect to master and work. But may be this is not convenient as you cannot access WebUI of workers
Use special syntax like "8081-8999:8081" so each next worker started with docker-compose up --scale worker=2 will use different port.