My kafka docker container cannot connect to my zookeeper docker container - docker

Question

I want to use both confluent/kafka and confluent/zookeeper and run them on a single Ubuntu server.
I'm using the following configurations:
docker run -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper
docker run --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka
However this results in: Unable to connect to zookeeper:2181
I have other containers that I'd like to connect to, how can I access zookeeper via zookeeper:2181 and kafka via kafka:9092 ?

Answer 1

There are multiple ways to do this. But before we look into it there are two problems in your approach that you need to understand
zookeper host is not reachable when you use docker run as each of the containers is running in a different network isolation
kafka may start and try to connect to zookeeper but zookeeper is not ready yet
Solving the network issue
You can do a lot of things to fix things
use --net=host to run both on the host network
use docker network create <name> and then use --net=<name> while launching both the containers
Or you can run your kafka container on the zookeeper containers network.
use --net=container:zookeeper when launching kafka container. This will make sure zookeeper host is accessible. This is not recommended as such, until unless you have some strong reason to do so. Because as soon as zookeeper container goes down, so will be the network of your kafka container. But for the sake of understanding, I have put this option here
Solving the startup race issue
Either you can keep a gap between starting zookeeper and kafka, to make sure that when kafka starts zookeeper is up and running
Another option is to use --restart=on-failure flag with docker run. This will make sure the container is restarted on failure and will try to reconnect to zookeeper and hopefully that time zookeeper will be up.
Instead of using docker run I would always prefer the docker-compose to get such linked containers to be run. You can do that by creating a simple docker-compose.yml file and then running it with docker-compsoe up
version: "3.4"
services:
zookeeper:
image: confluent/zookeeper
environment:
- ZOOKEEPER_CLIENT_PORT=2181
kafka:
image: confluent/kafka
environment:
- KAFKA_ADVERTISED_HOST_NAME=kafka
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CREATE_TOPICS=testtopic:1:1
depends_on:
- zookeeper
restart: on-failure

Answer 2

I'm running on Mac though, this is working fine. since 'host' networking not working in mac i just create a network called kafka_net and put the containers there.
version: "3.4"
services:
zookeeper:
image: confluent/zookeeper
environment:
- ZOOKEEPER_CLIENT_PORT=2181
networks:
- kafka_net
kafka:
image: confluent/kafka
environment:
- KAFKA_ADVERTISED_HOST_NAME=kafka
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
networks:
- kafka_net
restart: on-failure
networks:
kafka_net:
driver: "bridge"
To make sure all working:
Log into the zookeeper container then
zookeeper-shell localhost:2181 => You should see something like 'Welcome to ZooKeeper!' after all the big chunk of text
Log into kafka container then
kafka-topics --zookeeper zookeeper:2181 --list # empty list
kafka-topics --zookeeper zookeeper:2181 --create --topic first_topic --replication-factor 1 --partitions 1
kafka-topics --zookeeper zookeeper:2181 --list # you will see the first_topic
kafka-console-producer --broker-list localhost:9092 --topic first_topic # type some text and ctrl + c
kafka-console-consumer --bootstrap-server localhost:9092 --zookeeper zookeeper:2181 --topic first_topic --from-beginning # you will see the stuff you typed first_topic
If still giving problems have a look in the official examples. https://github.com/confluentinc/cp-docker-images/tree/5.2.2-post/examples and still giving issues post it, will give a hand.

Answer 3

Docker start containers in isolated network, called default bridge unless you specify network explicitly.
You can succeed in different ways, here are 2 easiest:
Put containers into same user-defined bridge network
# create net
docker network create foo
docker run --network=foo -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper
docker run --network=foo --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka
Expose ports and connect through localhost
docker run -p 2181:2181 -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper
docker run --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=host.docker.internal:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka
Note: in second approach you should use host.docker.internal as a host name and expose (publish) port 2181 for first container to make it available on localhost

Answer 4

docker network create kafka-zookeeper
Wait until Network is Created
docker run -it -d --network=kafka-zookeeper --name zookeeper zookeeper
Wait until ZooKeeper is up and Running
docker run -it -d --network=kafka-zookeeper --name kafka -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 --restart=on-failure -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092 -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 bitnami/kafka
Kafka Should be connecting fine.
These are running in -d detached mode, so you need to go to Docker Desktop to view the logs for each container.