I am starting my first docker proyect, trying to setup zookeeper and Kafka.
I have the following docker-compose.yml:
version: "3.8"
networks:
mynet:
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
networks:
- mynet
kafka:
image: wurstmeister/kafka:2.12-2.4.0
ports:
- "9092:9092"
expose:
- "9093"
environment:
KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
# Create a topic NAME:PARTITION:REPLICAS
KAFKA_CREATE_TOPICS: "example-topic:1:1"
networks:
- mynet
kafka-manager:
image: sheepkiller/kafka-manager:latest
environment:
ZK_HOSTS: "zookeeper:2181"
ports:
- 9000:9000
networks:
- mynet
I execute the docker-compose.yml:
sudo docker-compose up -d
To check if I have the topic created, I access the Kafka shell through:
sudo docker exec -it <<CONTAINER ID>> sh
In this shell I go to:
cd opt/kafka
And execute:
bin/kafka-topics.sh --list --zookeeper localhost:2181
And the output is a timeout error:
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:321)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Zookeeper isn't running in the Kafka container.
You should use --bootstrap-server localhost:9092 anyway since the Zookeeper argument is deprecated
Related
I get an error when I try to view topics and consumers using UI for apache kafka
docker command i use:
docker run -p 8080:8080 -e KAFKA_CLUSTERS_0_ZOOKEEPER=2181:2181 -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=127.0.0.1:9092 -d provectuslabs/kafka-ui:latest
or docker-compose.yml file
services:
kafka-ui:
container_name: kafka-ui
image: provectuslabs/kafka-ui:latest
ports:
- 8080:8080
depends_on:
- kafka
environment:
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
KAFKA_CLUSTERS_0_JMXPORT: 9997
kafka:
image: johnnypark/kafka-zookeeper
ports:
- "2181:2181"
- "9092:9092"
network_mode: bridge
environment:
ADVERTISED_HOST: 127.0.0.1
NUM_PARTITIONS: 1
volumes:
- /var/run/docker.sock:/var/run/docker.sock
I tried both ways, both didn't work
Where did i go wrong?
2181:2181 is two ports, not a Zookeeper hostname/ip and port
Then, Kafka address is referring to the container you're running, not the actual Kafka server. You'll need to modify your Kafka server properties if it's running on the host, and you need to connect from Docker. Related - Connect to Kafka on host from Docker (ksqlDB)
Beyond that, remove the -d or use docker logs to see the actual error.
I was using the johnnypark/kafka-zookeeper library for both kafka and zookeeper. I was able to solve this problem by using two separate libraries as in the example below
zookeeper1:
image: confluentinc/cp-zookeeper:5.2.4
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka1:
image: confluentinc/cp-kafka:5.3.1
depends_on:
- zookeeper1
ports:
- 9093:9093
- 9998:9998
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
JMX_PORT: 9998
KAFKA_JMX_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka1 -Dcom.sun.management.jmxremote.rmi.port=9998
https://github.com/provectus/kafka-ui/blob/master/docker/kafka-ui.yaml
I've below docker-compose.yml file
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:6.0.0
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-enterprise-kafka:6.0.0
hostname: kafka
container_name: kafka
depends_on:
- zookeeper
ports:
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
When I run docker-compose exec kafka bash I get the bash prompt.
In the bash prompt I've kafka-console-consumer but I don't have access to kafka-avro-console-consumer?
How can I get access to kafka-avro-console-consumer? Is it not in $PATH but in some other directory?
I tried to use find and which commands but those are not present in docker running container
use confluentinc/cp-schema-registry
docker run confluentinc/cp-schema-registry \
kafka-avro-console-consumer --bootstrap-server localhost:29092 --topic quickstart-jdbc-test --from-beginning --max-messages 10
I am using docker for my sample Spark + Kafka project in windows machine.
I am facing
WARN ClientUtils: Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
[error] (run-main-0) org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
[error] org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
------
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
Below is my docker-compose.yml
version: '2'
services:
test1:
build: test1service/.
depends_on:
- kafka
test2:
build: test2/.
depends_on:
- kafka
- test1
zookeeper:
image: confluentinc/cp-zookeeper:5.1.0
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
extra_hosts:
- "localhost: 127.0.0.1"
kafka:
image: confluentinc/cp-kafka:5.1.0
ports:
- 9092:9092
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
#KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
extra_hosts:
- "localhost: 127.0.0.1"
Below is my sample code in test2 service
val inputStreamDF = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9092")
.option("subscribe", "test1")
.option("startingOffsets", "earliest")
.load()
docker ps command output is
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4c8feb49e12b test1 "/usr/bin/start.sh" About an hour ago Up 50 minutes test1
4535ce246541 test2 "/usr/bin/myservice-…" About an hour ago Up 50 minutes test2
733766f72adb confluentinc/cp-kafka:5.1.0 "/etc/confluent/dock…" About an hour ago Up 51 minutes 0.0.0.0:9092->9092/tcp kafka_1
d915e25cb226 confluentinc/cp-zookeeper:5.1.0 "/etc/confluent/dock…" About an hour ago Up 51 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper_1
Has anyone faced similar issue, how did you resolve?
You try to reach kafka:9092, but docker compose has generated the container kafka_1, which is why there is no name resolution.
Docker gives internal ip to your containers, and use their container name to create an internal DNS on this network (with the embbed dns server)
Your environment variables are not changed by Docker Compose to fit the name it gives to your container.
You should use "container_name: kafka" in your container description to get a static container name.
Only thing you need is network. After that all services will be connected and you can use kafka:9092.
version: '2'
services:
test1:
build: test1service/.
depends_on:
- kafka
networks:
- app-network
test2:
build: test2/.
depends_on:
- kafka
- test1
networks:
- app-network
zookeeper:
image: confluentinc/cp-zookeeper:5.1.0
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
extra_hosts:
- "localhost: 127.0.0.1"
networks:
- app-network
kafka:
image: confluentinc/cp-kafka:5.1.0
ports:
- 9092:9092
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
#KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
extra_hosts:
- "localhost: 127.0.0.1"
networks:
- app-network
networks:
app-network:
driver: bridge
Try with localhost and then the container port:
localhost:9092
Here is an example command for creating a topic named "orders"
docker exec kafka kafka-topics --bootstrap-server localhost:9092 --create --topic orders --partitions 1 --replication-factor 1
I'm trying to setup Kafka in a docker container for local development. My docker-compose.yml looks as follows:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181"
hostname: zookeeper
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "9092"
hostname: kafka
environment:
KAFKA_CREATE_TOPICS: "UploadFile:1:1,GetFile:1:1,TrackUpload:1:1,GetEmailContent:1:1" # topic:partition:replicas
KAFKA_ADVERTISED_HOST_NAME: kafka # docker-machine ip
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_PORT: 9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
Unfortunately my node app running on my localhost (not in docker) cannot connect to it. I used the url 'kafka:9092' and even 'localhost:9092'. Nothing works. Any idea what's happening?
Expose the host port 9092 for kafka service & you should be able to connect via "localhost:9092" from the app or host machine.
....
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "9092:9092"
....
I set up kafka in a docker container using this project https://github.com/wurstmeister/kafka-docker. I can successfully create and list topics on it, but once I try to produce a message with either
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
or
bin/kafka-console-producer.sh --broker-list 0.0.0.0:9092 --topic test
I get the following error:
ERROR Error when sending message to topic test with key: null, value:
4 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s)
for test-0: 1536 ms has passed since batch creation plus linger time
My docker-compose.yml lookes like this:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: docker.for.mac.localhost
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
I tried different options for the KAFKA_ADVERTISED_HOST_NAME property, including docker.for.mac.localhost which is supposed to be resolved as the actual docker host ip, and 0.0.0.0, but the result is the same.
You can try with this docker-compose, it's working for me:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
stdin_open: true
tty: true
ports:
- 2181:2181/tcp
kafka:
image: wurstmeister/kafka:0.11.0.1
environment:
KAFKA_CREATE_TOPICS: test:1:1
KAFKA_ZOOKEEPER_CONNECT: <<HOST_IP>>:2181
KAFKA_ADVERTISED_HOST_NAME: <<HOST_IP>>
KAFKA_ADVERTISED_PORT: '9092'
KAFKA_BROKER_ID: '999'
stdin_open: true
tty: true
links:
- zookeeper:zookeeper
ports:
- 9092:9092/tcp
Don't use local IPs like 127.0.0.1. Use the true IP instead.