My docker-compose reads
version: '2'
services:
zookeeper:
image: debezium/zookeeper:1.5
ports:
- 2181:2181
- 2888:2888
- 3888:3888
kafka:
image: debezium/kafka:1.5
ports:
- 9092:9092
links:
- zookeeper
environment:
- ZOOKEEPER_CONNECT=zookeeper:2181
- ADVERTISED_HOST_NAME=kafka
List all topics working fine
docker exec -t root_kafka_1 bash /kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --list
Create Topic working fine
docker exec -t root_kafka_1 bash /kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 \
--create \
--topic test_topic \
--partitions 1 \
--replication-factor 1
List a topic detail working fine
docker exec -t root_kafka_1 bash /kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --topic test_topic --describe
But writing events to Kafka topic hangs
docker exec -t root_kafka_1 bash /kafka/bin/kafka-console-producer.sh --topic test_topic --bootstrap-server kafka:9092
>My first event
Not seeing any error also at kafka logs. What I am missing here.
Update
I tried to use Offset Explorer UI tool for Kafka. There I am able to push message to Kafka topic and able to consume the same events.
Related
I'm trying to create a topic in a Kafka container running in Docker using a shell script file.
docker-compose.yml
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "22181:2181"
kafka:
image: confluentinc/cp-kafka:latest
container_name: kafka
depends_on:
- zookeeper
ports:
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
init.sh
#!/bin/zsh
docker compose up -d
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test
When I run docker-compose the containers start-up fine, but the topic creation fails.
[2021-11-04 17:35:07,957] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
However, when I run the same commands in the command line, without any changes, it works fine and creates the topic.
Created topic Test.
I'm at a loss and reading previous questions on similar issues here hasn't help: mostly they were due to typos or adding quotation marks where they are not needed. I have tried changing the #! to bash, sh, and usr/bin/env (amongst others) to no avail.
Any help appreciated.
Docker version: version 20.10.8
Running on a Mac (Intel)
As suggested by the error message, the broker is not available (in your case it hasn't started yet).
You could wait for the broker to complete start up process (by looking at the logs with docker logs -f kafka or by using a sleep in your script) and then run your command to create the topic:
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test
Alternatively, instead of having to figure out if your broker has completed the initialization process, you could use something like this official Confluent example.
In this example, an utility is used to wait for a Kafka cluster to be ready.
Where/when are you running the init.sh script? Maybe the broker hasn't started yet.
Also, use the internal listener when you want to run commands inside the Docker network
docker-compose exec kafka \
bash -c "kafka-topics --create --bootstrap-server kafka:9092 --partitions 1 --replication-factor 1 --topic Test"
Modify init.sh
Run the container, docker-compose up -d, wait for the broker to run. check with the logs if the application is up and running.
Now init.sh should have only create the topic command
#!/bin/zsh
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test1
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test2
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test3
Everyone was right, I had to wait longer (I had tried 10 secs).
As #Luigi Cerone suggested, I used cub to wait for Kafka to be ready, by adding this line before creating a topic:
docker exec kafka cub kafka-ready -b localhost:29092 1 20
I'm new to Kafka and I am trying to create a demo Kafka server and work with it.
I have already created Zookeeper and Kafka containers using a docker-compose file and they are started and running fine.
If I docker exec into the Kafka container and run:
kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic
the topic is created successfully... But I would like to spin up the Kafka broker and then programmatically create the topics without user interaction (Will ultimately be for a pipeline)
I've tried two different Kafka images (other was confluentinc/cp-kafka)... I have also tried changing bash to sh under kafka-1.command
Could really do with some help here. Below is my docker-compose file and the error response I'm getting in my terminal for bitnami image & for confluent image.
ERROR (BITNAMI)
/opt/bitnami/scripts/kafka/entrypoint.sh: line 27: exec: bash -c "kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic": not found
ERROR (CONFLUENT)
Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "bash -c \"kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic\"": executable file not found in $PATH: unknown
DOCKER-COMPOSE.YAML
services:
zookeeper-1:
container_name: zookeeper-1
image: zookeeper
restart: always
ports:
- 2181:2181
environment:
- ZOOKEEPER_CLIENT_PORT=2181
volumes:
- ./config/zookeeper-1/zookeeper.properties:/kafka/config/zookeeper.properties
kafka-1:
container_name: kafka-1
image: bitnami/kafka
depends_on:
- zookeeper-1
ports:
- 29092:29092
- 9092:9092
environment:
- KAFKA_ZOOKEEPER_CONNECT=zookeeper-1:2181
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092
- ALLOW_PLAINTEXT_LISTENER=yes
command:
- bash -c "kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic"
For both errors, the command you are providing is being appended to the container's entrypoints, which do not accept bash commands... You would need to override both the entrypoint and the command, as well as give the full path to the kafka-topics script since it is not on the $PATH.
However, you cannot do this with the command on the same container as the broker because that overrides the command that actually starts the server.
You will need a secondary "init container" that creates the topics , but it would be easier for you to write this topic creation logic into your own producer/consumer applications with an AdminClient.createTopics call (assuming Java).
Otherwise, you can use wurstmeister/kafka image which has an environment variable KAFKA_CREATE_TOPICS for this purpose, or just use docker-compose exec kafka-1 "..." after the containers are up
This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed 2 years ago.
I'm following this guide in order to start a single kafka container .I'm working on win10 pro and my cli is gitbash.
docker-compose-single-broker.yml :
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
The containers are starting without any problem :
$ docker-compose -f docker-compose-single-broker.yml up -d
Creating kafka-docker_zookeeper_1 ... done
Creating kafka-docker_kafka_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
19929c6c3297 wurstmeister/kafka "start-kafka.sh" 17 seconds ago Up 15 seconds 0.0.0.0:9092->9092/tcp kafka-docker_kafka_1
d343a8ecf7ed wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 17 seconds ago Up 15 seconds 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp kafka-docker_zookeeper_1
However, when I try to run any kafka command(create topic,list topics) from inside the container like the following :
> $KAFKA_HOME/bin/kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --bootstrap-server `broker-list.sh`
> $KAFKA_HOME/bin/kafka-topics.sh --describe --topic test --bootstrap-server `broker-list.sh`
I'm getting the following warnings and afterwards a timeout exception :
$ ./start-kafka-shell.sh localhost
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --bootstrap-server `broker-list.sh`
[2020-06-09 11:47:26,241] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/172.17.54.145:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-09 11:47:29,306] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/172.17.54.145:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
The content of the start-kafka-shell.sh :
$ cat start-kafka-shell.sh
#!/bin/bash
docker run --rm -v //var/run/docker.sock:/var/run/docker.sock -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka /bin/bash
I'm not sure what the purpose of start-kafka-shell.sh is. If you're using Docker then you can interact with it from your Windows shell as follows:
docker exec -it tmp_kafka_1 bash -c '$KAFKA_HOME/bin/kafka-topics.sh --create --topic test4 --partitions 1 --replication-factor 1 --bootstrap-server `broker-list.sh`'
docker exec -it tmp_kafka_1 bash -c '$KAFKA_HOME/bin/kafka-topics.sh --describe --topic test4 --bootstrap-server `broker-list.sh`'
Topic: test4 PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: test4 Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
You just need the correct Docker container name (tmp_kafka_1 in my example above), which you can confirm with docker ps.
I found few questions with similar topic but different context: I can connect to create a Topic but I can't list the topics because I got the error mentioned below (as far as I could see, people were facing issues for basic connecting while I am getting only for listing the topic list).
In case it matters, here is my docker-compose.yml:
version: "3"
services:
zookeeper:
image: wurstmeister/zookeeper
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: "localhost"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
My console:
bash-4.4# kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test2
Created topic test2.
bash-4.4# kafka-topics.sh --list --zookeeper localhost:2181
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:321)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
bash-4.4# kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test3
Created topic test3.
bash-4.4# kafka-topics.sh --list --zookeeper localhost:2181
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:321)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Edited
Future readers may be found useful how I could list all topics straight from my Docker Kafka Container without logging in my Docker Zookeper Container (https://stackoverflow.com/a/56595227/4148175)
C:\Users\>docker exec -it test1_kafka_1 bash
bash-4.4# kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
test2
test3
test_topic
bash-4.4#
--zookeeper zookeeper:2181 seems to have worked fine
--zookeeper localhost:2181 will always fail inside the kafka container because it's not running a zookeeper server
I could list all topics straight from my Docker Kafka Container without logging in my Docker Zookeper Container
That's right. Ideally, you should never enter the Zookeeper container. Latest kafka versions support using --bootstrap-server instead, so you could use kafka:9092 or localhost:9092 from the kafka container
I want to use both confluent/kafka and confluent/zookeeper and run them on a single Ubuntu server.
I'm using the following configurations:
docker run -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper
docker run --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka
However this results in: Unable to connect to zookeeper:2181
I have other containers that I'd like to connect to, how can I access zookeeper via zookeeper:2181 and kafka via kafka:9092 ?
There are multiple ways to do this. But before we look into it there are two problems in your approach that you need to understand
zookeper host is not reachable when you use docker run as each of the containers is running in a different network isolation
kafka may start and try to connect to zookeeper but zookeeper is not ready yet
Solving the network issue
You can do a lot of things to fix things
use --net=host to run both on the host network
use docker network create <name> and then use --net=<name> while launching both the containers
Or you can run your kafka container on the zookeeper containers network.
use --net=container:zookeeper when launching kafka container. This will make sure zookeeper host is accessible. This is not recommended as such, until unless you have some strong reason to do so. Because as soon as zookeeper container goes down, so will be the network of your kafka container. But for the sake of understanding, I have put this option here
Solving the startup race issue
Either you can keep a gap between starting zookeeper and kafka, to make sure that when kafka starts zookeeper is up and running
Another option is to use --restart=on-failure flag with docker run. This will make sure the container is restarted on failure and will try to reconnect to zookeeper and hopefully that time zookeeper will be up.
Instead of using docker run I would always prefer the docker-compose to get such linked containers to be run. You can do that by creating a simple docker-compose.yml file and then running it with docker-compsoe up
version: "3.4"
services:
zookeeper:
image: confluent/zookeeper
environment:
- ZOOKEEPER_CLIENT_PORT=2181
kafka:
image: confluent/kafka
environment:
- KAFKA_ADVERTISED_HOST_NAME=kafka
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CREATE_TOPICS=testtopic:1:1
depends_on:
- zookeeper
restart: on-failure
I'm running on Mac though, this is working fine. since 'host' networking not working in mac i just create a network called kafka_net and put the containers there.
version: "3.4"
services:
zookeeper:
image: confluent/zookeeper
environment:
- ZOOKEEPER_CLIENT_PORT=2181
networks:
- kafka_net
kafka:
image: confluent/kafka
environment:
- KAFKA_ADVERTISED_HOST_NAME=kafka
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
networks:
- kafka_net
restart: on-failure
networks:
kafka_net:
driver: "bridge"
To make sure all working:
Log into the zookeeper container then
zookeeper-shell localhost:2181 => You should see something like 'Welcome to ZooKeeper!' after all the big chunk of text
Log into kafka container then
kafka-topics --zookeeper zookeeper:2181 --list # empty list
kafka-topics --zookeeper zookeeper:2181 --create --topic first_topic --replication-factor 1 --partitions 1
kafka-topics --zookeeper zookeeper:2181 --list # you will see the first_topic
kafka-console-producer --broker-list localhost:9092 --topic first_topic # type some text and ctrl + c
kafka-console-consumer --bootstrap-server localhost:9092 --zookeeper zookeeper:2181 --topic first_topic --from-beginning # you will see the stuff you typed first_topic
If still giving problems have a look in the official examples. https://github.com/confluentinc/cp-docker-images/tree/5.2.2-post/examples and still giving issues post it, will give a hand.
Docker start containers in isolated network, called default bridge unless you specify network explicitly.
You can succeed in different ways, here are 2 easiest:
Put containers into same user-defined bridge network
# create net
docker network create foo
docker run --network=foo -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper
docker run --network=foo --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka
Expose ports and connect through localhost
docker run -p 2181:2181 -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper
docker run --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=host.docker.internal:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka
Note: in second approach you should use host.docker.internal as a host name and expose (publish) port 2181 for first container to make it available on localhost
docker network create kafka-zookeeper
Wait until Network is Created
docker run -it -d --network=kafka-zookeeper --name zookeeper zookeeper
Wait until ZooKeeper is up and Running
docker run -it -d --network=kafka-zookeeper --name kafka -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 --restart=on-failure -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092 -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 bitnami/kafka
Kafka Should be connecting fine.
These are running in -d detached mode, so you need to go to Docker Desktop to view the logs for each container.