How to run zookeeper & kafka-server on docker avoiding `Connection refused` exception? - docker

Question

I am trying to run a kafka-spark streaming application on Docker. Below is my project structure.
Dockerfile contents:
from gcr.io/datamechanics/spark:platform-3.1-dm14
ENV PYSPARK_MAJOR_PYTHON_VERSION=3
WORKDIR /opt/application/
RUN wget https://jdbc.postgresql.org/download/postgresql-42.2.5.jar
RUN mv postgresql-42.2.5.jar /opt/spark/jars
COPY requirements.txt .
RUN pip3 install -r requirements.txt
COPY main.py .
Docker-compose.yml contents:
version: "2"
services:
spark:
image: docker.io/bitnami/spark:3
environment:
- SPARK_MODE=master
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
ports:
- '8080:8080'
spark-worker:
image: docker.io/bitnami/spark:3
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://spark:7077
- SPARK_WORKER_MEMORY=1G
- SPARK_WORKER_CORES=1
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
zookeeper:
image: confluentinc/cp-zookeeper:latest
container_name: zookeeper
ports:
- "22181:22181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
- ZOOKEEPER_CLIENT_PORT=2181
- ZOOKEEPER_TICK_TIME=2000
kafka-server1:
image: confluentinc/cp-kafka:latest
ports:
- '9092:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server
I was able to pull the images and create containers successfully.
(venv) (base) johnny#Johnnys-MBP~/PyCharmProjects/dockerpractice/Johnny> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d47d4b091b2c confluentinc/cp-zookeeper:latest "/etc/confluent/dock…" 16 minutes ago Up 12 minutes 2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:22181->22181/tcp zookeeper
59c06d3cf754 confluentinc/cp-kafka:latest "/etc/confluent/dock…" 48 minutes ago Up 48 minutes 9092/tcp, 0.0.0.0:29092->29092/tcp kafka
26794da88c7d bitnami/spark:3 "/opt/bitnami/script…" About an hour ago Up About an hour dockerpractice-spark-worker-1
5a308035bd18 bitnami/spark:3 "/opt/bitnami/script…" About an hour ago Up About an hour 0.0.0.0:8080->8080/tcp dockerpractice-spark-1
I connected to the kafka zookeeper image and started it like below:
docker run -i -t confluentinc/cp-zookeeper:latest /bin/bash
zookeeper-server-start /etc/kafka/zookeeper.properties
The above command starts zookeeper with no exceptions and I tried to start my kafka-server in the same way:
docker run -i -t confluentinc/cp-kafka:latest /bin/bash
kafka-server-start /etc/kafka/server.properties
But when I submit the command for kafka-server, I see the below exception:
[2022-04-04 18:13:11,276] WARN Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
I tried various combinations like changing the ip address & ports of kafka server to new ones and by creating multiple kafka servers but all of them are resulting in same exception.
Could anyone let me know what is the mistake I am doing here and how can I correct it?