I have docker-compose.yml with below image and configuration
version: '3'
services:
spark-master:
image: bde2020/spark-master:2.4.4-hadoop2.7
container_name: spark-master
ports:
- "8080:8080"
- "7077:7077"
environment:
- INIT_DAEMON_STEP=setup_spark
spark-worker-1:
image: bde2020/spark-worker:2.4.4-hadoop2.7
container_name: spark-worker-1
depends_on:
- spark-master
ports:
- "8081:8081"
environment:
- "SPARK_MASTER=spark://spark-master:7077"
here the docker-compose up log ---> https://jpst.it/1Xc4K
and here containers up and running and i mean spark worker connected to spark master without any issues , now problem is i created drone.yml and where i added services component with
services:
jce-cassandra:
image: cassandra:3.0
ports:
- "9042:9042"
jce-elastic:
image: elasticsearch:5.6.16-alpine
ports:
- "9200:9200"
environment:
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
janusgraph:
image: janusgraph/janusgraph:latest
ports:
- "8182:8182"
environment:
JANUS_PROPS_TEMPLATE: cassandra-es
janusgraph.storage.backend: cql
janusgraph.storage.hostname: jce-cassandra
janusgraph.index.search.backend: elasticsearch
janusgraph.index.search.hostname: jce-elastic
depends_on:
- jce-elastic
- jce-cassandra
spark-master:
image: bde2020/spark-master:2.4.4-hadoop2.7
container_name: spark-master
ports:
- "8080:8080"
- "7077:7077"
environment:
- INIT_DAEMON_STEP=setup_spark
spark-worker-1:
image: bde2020/spark-worker:2.4.4-hadoop2.7
container_name: spark-worker-1
depends_on:
- spark-master
ports:
- "8081:8081"
environment:
- "SPARK_MASTER=spark://spark-master:7077"
but here spark worker is not connected to spark master getting exceptions, here is exception log details , can some one please guide me why am facing this issue
Note : I am trying to create these services in drone.yml for my integration testing
Answering for better formatting. The comments suggest sleeping. Assuming this is the dockerfile (https://hub.docker.com/r/bde2020/spark-worker/dockerfile) You could sleep by adding the command:
spark-worker-1:
image: bde2020/spark-worker:2.4.4-hadoop2.7
container_name: spark-worker-1
command: sleep 10 && /bin/bash /worker.sh
depends_on:
- spark-master
ports:
- "8081:8081"
environment:
- "SPARK_MASTER=spark://spark-master:7077"
Although sleep 10 is probably excessive, if this would would sleep 5 or sleep 2
Related
I want to deploy a service that will allow me to use Spark and MongoDB in a Jupiter notebook.
I use docker-compose to build up the service, and it`s as followed:
version: "3.3"
volumes:
shared-workspace:
networks:
spark-net:
driver: bridge
services:
spark-master:
image: uqteaching/cloudcomputing:spark-master-v1
container_name: spark-master
networks:
- "spark-net"
ports:
- "8080:8080"
- "7077:7077"
environment:
- INIT_DAEMON_STEP=setup_spark
- "PYSPARK_PYTHON=/usr/bin/python3"
- "PYSPARK_DRIVER_PYTHON=/usr/bin/python3"
spark-worker-1:
image: uqteaching/cloudcomputing:spark-worker-v1
container_name: spark-worker-1
depends_on:
- spark-master
networks:
- "spark-net"
ports:
- "8081:8081"
environment:
- "SPARK_MASTER=spark://spark-master:7077"
- "PYSPARK_PYTHON=/usr/bin/python3"
- "PYSPARK_DRIVER_PYTHON=/usr/bin/python3"
spark-worker-2:
image: uqteaching/cloudcomputing:spark-worker-v1
container_name: spark-worker-2
depends_on:
- spark-master
networks:
- "spark-net"
ports:
- "8082:8082"
environment:
- "SPARK_MASTER=spark://spark-master:7077"
- "PYSPARK_PYTHON=/usr/bin/python3"
- "PYSPARK_DRIVER_PYTHON=/usr/bin/python3"
mongo:
image: mongo
container_name: 'mongo'
networks:
- "spark-net"
ports:
- "27017:27017"
mongo_admin:
image: mongo-express
container_name: 'mongoadmin'
networks:
- "spark-net"
depends_on:
- mongo
links:
- mongo
ports:
- "8091:8091"
jupyter-notebook:
container_name: jupyternb
image: jupyter/all-spark-notebook:42f4c82a07ff
depends_on:
- mongo
- spark-master
links:
- mongo
expose:
- "8888"
networks:
- "spark-net"
ports:
- "8888:8888"
volumes:
- ./nbs:/home/jovyan/work/nbs
- ./events:/tmp/spark-events
environment:
- "PYSPARK_PYTHON=/usr/bin/python3"
- "PYSPARK_DRIVER_PYTHON=/usr/bin/python3"
command: "start-notebook.sh \
--ip=0.0.0.0 \
--allow-root \
--no-browser \
--notebook-dir=/home/jovyan/work/nbs \
--NotebookApp.token='' \
--NotebookApp.password=''
"
And the result is something like this:
I dont know why. Even I set these 2` services to listen to a different port.
They are using 8081/tcp at the same time, which caused them both to crash.
I want to solve this.
mongo-express seems to need port 8081 internal, so use another external port to be able to login to the webui.
http://localhost:8092 would then be something like this:
mongo_admin:
image: mongo-express
container_name: 'mongoadmin'
networks:
- "spark-net"
depends_on:
- mongo
links:
- mongo
ports:
- "8092:8091"
replicating standard_ds3_v2 databricks cluster in docker (14gb 4 cores)
would I just change the spark worker 1 and 2 in dockerfile from 512MB each to 7GB and from 1 core to 2. would this result in the exact same config as the standard_ds3_v2 data bricks cluster?
below is the docker cofig
version: "3.6"
volumes:
shared-workspace:
name: "hadoop-distributed-file-system"
driver: local
services:
jupyterlab:
image: jupyterlab
container_name: jupyterlab
ports:
- 8888:8888
volumes:
- shared-workspace:/opt/workspace
spark-master:
image: spark-master
container_name: spark-master
ports:
- 8080:8080
- 7077:7077
volumes:
- shared-workspace:/opt/workspace
spark-worker-1:
image: spark-worker
container_name: spark-worker-1
environment:
- SPARK_WORKER_CORES=1
- SPARK_WORKER_MEMORY=512m
ports:
- 8081:8081
volumes:
- shared-workspace:/opt/workspace
depends_on:
- spark-master
spark-worker-2:
image: spark-worker
container_name: spark-worker-2
environment:
- SPARK_WORKER_CORES=1
- SPARK_WORKER_MEMORY=512m
ports:
- 8082:8081
volumes:
- shared-workspace:/opt/workspace
depends_on:
- spark-master
I have an Hadoop cluster with one namenode and one datanode instantiated with docker compose. In addition I am trying to launch Hive but the Hive-metastore seems not to find my datanode even if this one is up and running, infact checking the log it shows:
namenode:9870 is available.
check for datanode:9871...
datanode:9871 is not available yet
try in 5s once again ...
Here is my docker-compose.yml
#HADOOP
namenode:
image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
container_name: namenode
restart: always
expose:
- "9870"
- "54310"
- "9000"
ports:
- 9870:9870
- 9000:9000
volumes:
- ./data/hadoop_data/:/hadoop_data
environment:
- CLUSTER_NAME=test
- CORE_CONF_fs_defaultFS=hdfs://namenode:9000
- CORE_CONF_hadoop_http_staticuser_user=root
- CORE_CONF_hadoop_proxyuser_hue_hosts=*
- CORE_CONF_hadoop_proxyuser_hue_groups=*
- CORE_CONF_io_compression_codecs=org.apache.hadoop.io.compress.SnappyCodec
- HDFS_CONF_dfs_webhdfs_enabled=true
- HDFS_CONF_dfs_permissions_enabled=false
- HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false
- HDFS_CONF_dfs_safemode_threshold_pct=0
datanode:
image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
container_name: datanode
restart: always
expose:
- "9871"
environment:
SERVICE_PRECONDITION: "namenode:9870"
ports:
- "9871:9871"
env_file:
- hive.env
hive-server:
image: bde2020/hive:2.3.2-postgresql-metastore
container_name: hive-server
volumes:
- ./employee:/employee
env_file:
- hive.env
environment:
HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: "jdbc:postgresql://hive-metastore/metastore"
SERVICE_PRECONDITION: "hive-metastore:9083"
depends_on:
- hive-metastore
ports:
- "10000:10000"
hive-metastore:
image: bde2020/hive:2.3.2-postgresql-metastore
container_name: hive-metastore
env_file:
- hive.env
command: /opt/hive/bin/hive --service metastore
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode:9871 hive-metastore-postgresql:5432"
depends_on:
- hive-metastore-postgresql
ports:
- "9083:9083"
hive-metastore-postgresql:
image: bde2020/hive-metastore-postgresql:2.3.0
container_name: hive-metastore-postgresql
volumes:
- ./metastore-postgresql/postgresql/data:/var/lib/postgresql/data
depends_on:
- datanode
I am using zipkin distributed tracing with rabbitMQ. But tracing is not showing. When i RUN QUERY in zipkin, it does not show anything. Here is my screen
docker-compose.yaml file is bellow...'
version: '3.7'
services:
api-gatway:
image: mydocker/pocv1-api-gateway:0.0.1-SNAPSHOT
mem_limit: 700m
ports:
- "8765:8765"
networks:
- account-network
depends_on:
- service-registry
- rabbitmq
environment:
EUREKA.CLIENT.SERVICEURL.DEFAULTZONE: http://service-registry:8761/eureka
SPRING.ZIPKIN.BASEURL: http://zipkin-server:9411/
RABBIT_URI: amqp://guest:guest#rabbitmq:5672
SPRING_RABBITMQ_HOST: rabbitmq
SPRING_ZIPKIN_SENDER_TYPE: rabbit
zipkin-server:
image: openzipkin/zipkin-slim
mem_limit: 300m
ports:
- "9411:9411"
networks:
- account-network
depends_on:
- rabbitmq
environment:
environment:
RABBIT_URI: amqp://guest:guest#rabbitmq:5672
restart: always #Restart if there is a problem starting up
rabbitmq:
image: rabbitmq:3-management
mem_limit: 300m
ports:
- "5672:5672"
- "15672:15672"
networks:
- account-network
account-opening:
image: mydocker/pocv1-account-opening:0.0.1-SNAPSHOT
mem_limit: 700m
ports:
- "8081:8081"
networks:
- account-network
depends_on:
- postgres
- service-registry
- rabbitmq
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/accountdb
EUREKA.CLIENT.SERVICEURL.DEFAULTZONE: http://service-registry:8761/eureka
SPRING.ZIPKIN.BASEURL: http://zipkin-server:9411/
RABBIT_URI: amqp://guest:guest#rabbitmq:5672
SPRING_RABBITMQ_HOST: rabbitmq
SPRING_ZIPKIN_SENDER_TYPE: rabbit
networks:
account-network:
In application.yml I use like bellow:
spring:
sleuth:
sampler:
probability: 1.0
I don't know why it is not working. Please help me...
You have made a silly mistake that you have included extra environment property in zipkin configuration. you need to remove that line. I faced the same mistake and got resolved in the same way.
zipkin-server:
image: openzipkin/zipkin-slim
mem_limit: 300m
ports:
- "9411:9411"
networks:
- account-network
depends_on:
- rabbitmq
environment:
***environment:** //remove this line*
RABBIT_URI: amqp://guest:guest#rabbitmq:5672
restart: always #Restart if there is a problem starting up
I'm new and learning to start a hadoop system using Docker but I have been stuck at one point for weeks and finally I have to ask here. I could launch the containers individually without any problems and the namenode always recognized the running namenodes. However, when I tried to set up a Docker-compose.yml file to launch all containers at once, I got into multiple problems, one of which is that the namenode never recognized the running datanodes. Do you have any suggestions to help me fix it?
Here's my docker-compose file:
version: "3"
services:
base:
image: hpcnube-base-image
hpcnube-namenode:
image: hpcnube-namenode-image
depends_on:
- base
hostname: hpcnube-namenode
container_name: hpcnube-namenode
networks:
- dockerfiles1_hpcnube-net
ports:
- "9870:9870"
hpcnube-resourcemanager:
image: hpcnube-resourcemanager-image
container_name: hpcnube-resourcemanager
depends_on:
- hpcnube-namenode
- hpcnube-dnnm2
- hpcnube-dnnm1
hostname: hpcnube-resourcemanager
networks:
- dockerfiles1_hpcnube-net
ports:
- "8088:8088"
hpcnube-dnnm1:
image: hpcnube-dnnm-image
container_name: hpcnube-dnnm1
hostname: hpcnube-dnnm1
depends_on:
- base
- hpcnube-namenode
networks:
- dockerfiles1_hpcnube-net
#command: "/opt/bd/start-daemons.sh"
hpcnube-dnnm2:
#build: ./DataNode-NodeManager
image: hpcnube-dnnm-image
container_name: hpcnube-dnnm2
hostname: hpcnube-dnnm2
depends_on:
- base
- hpcnube-namenode
networks:
- dockerfiles1_hpcnube-net
#command: "/opt/bd/start-daemons.sh"
hpcnube-checkpoint:
image: hpcnube-checkpointnode-image
hostname: hpcnube-checkpointnode
depends_on:
- base
- hpcnube-namenode
- hpcnube-resourcemanager
- hpcnube-dnnm2
- hpcnube-dnnm1
networks:
- dockerfiles1_hpcnube-net
hpcnube-timeline:
image: hpcnube-timelineserver-image
hostname: hpcnube-timelineserver
depends_on:
- base
- hpcnube-namenode
- hpcnube-resourcemanager
- hpcnube-dnnm2
- hpcnube-dnnm1
- hpcnube-checkpoint
networks:
- dockerfiles1_hpcnube-net
hpcnube-frontend:
build: ./FrontEnd
hostname: hpcnube-frontend
depends_on:
- base
- hpcnube-namenode
- hpcnube-resourcemanager
- hpcnube-dnnm2
- hpcnube-dnnm1
- hpcnube-checkpoint
- hpcnube-timeline
hostname: hpcnube-frontend
networks:
- dockerfiles1_hpcnube-net
ports:
- "2345:22"
networks:
dockerfiles1_hpcnube-net:
driver: bridge