Kafka Server issue in Docker - docker

Question

I am using docker for my sample Spark + Kafka project in windows machine.
I am facing
WARN ClientUtils: Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
[error] (run-main-0) org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
[error] org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
------
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
Below is my docker-compose.yml
version: '2'
services:
test1:
build: test1service/.
depends_on:
- kafka
test2:
build: test2/.
depends_on:
- kafka
- test1
zookeeper:
image: confluentinc/cp-zookeeper:5.1.0
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
extra_hosts:
- "localhost: 127.0.0.1"
kafka:
image: confluentinc/cp-kafka:5.1.0
ports:
- 9092:9092
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
#KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
extra_hosts:
- "localhost: 127.0.0.1"
Below is my sample code in test2 service
val inputStreamDF = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9092")
.option("subscribe", "test1")
.option("startingOffsets", "earliest")
.load()
docker ps command output is
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4c8feb49e12b test1 "/usr/bin/start.sh" About an hour ago Up 50 minutes test1
4535ce246541 test2 "/usr/bin/myservice-…" About an hour ago Up 50 minutes test2
733766f72adb confluentinc/cp-kafka:5.1.0 "/etc/confluent/dock…" About an hour ago Up 51 minutes 0.0.0.0:9092->9092/tcp kafka_1
d915e25cb226 confluentinc/cp-zookeeper:5.1.0 "/etc/confluent/dock…" About an hour ago Up 51 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper_1
Has anyone faced similar issue, how did you resolve?

Answer 1

You try to reach kafka:9092, but docker compose has generated the container kafka_1, which is why there is no name resolution.
Docker gives internal ip to your containers, and use their container name to create an internal DNS on this network (with the embbed dns server)
Your environment variables are not changed by Docker Compose to fit the name it gives to your container.
You should use "container_name: kafka" in your container description to get a static container name.

Answer 2

Only thing you need is network. After that all services will be connected and you can use kafka:9092.
version: '2'
services:
test1:
build: test1service/.
depends_on:
- kafka
networks:
- app-network
test2:
build: test2/.
depends_on:
- kafka
- test1
networks:
- app-network
zookeeper:
image: confluentinc/cp-zookeeper:5.1.0
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
extra_hosts:
- "localhost: 127.0.0.1"
networks:
- app-network
kafka:
image: confluentinc/cp-kafka:5.1.0
ports:
- 9092:9092
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
#KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
extra_hosts:
- "localhost: 127.0.0.1"
networks:
- app-network
networks:
app-network:
driver: bridge

Answer 3

Try with localhost and then the container port:
localhost:9092
Here is an example command for creating a topic named "orders"
docker exec kafka kafka-topics --bootstrap-server localhost:9092 --create --topic orders --partitions 1 --replication-factor 1