I have Kafka and Zookeeper running on two separate Docker containers:
<private-domain>/wurstmeister-kafka:0.10.1.0-2
<private-domain>/wurstmeister-zookeeper:3.4.9
Both containers seem to be up, but when I try to create Kafka topics by getting in to the first container:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
I get this error:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2020-06-07 03:10:55,293] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
Please notice that I did read other related questions and tried adding arguments to the command, such as -e ZK_HOSTS="localhost:2181". I know of other people working in the environment as mine who were able to run the commands successfully, so I suspect this might be a configuration issue on my side. Can you please guide?
EDIT: Here are the Docker Compose files:
version: '2'
services:
kafka:
image: <private-domain>/wurstmeister-kafka:0.10.1.0-2
container_name: kafka
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: 127.0.0.1:2181
restart:
"unless-stopped"
and
version: '2'
services:
zk:
image: <private-domain>/wurstmeister-zookeeper:3.4.9
container_name: zk
ports:
- "2181:2181"
restart:
"unless-stopped"
and the output of docker ps:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bf67a49da57a wurstmeister-kafka:0.10.1.0-2 "start-kafka.sh" 5 months ago Up 29 minutes 0.0.0.0:9092->9092/tcp kafka
ef3e908d82b3 wurstmeister-zookeeper:3.4.9 "/bin/sh -c '/usr/sbin/sshd && bash /usr/bin/start-zk.sh'" 5 months ago Up 29 minutes 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp zk
You have two Compose files. Thus, Your containers are on separated networks, and cannot refer each other.
You must add both services in one file, under one services: block, and run only one docker-compose up command
You can find working compose files here across the internet, or you could use minikube / oc with Kafka Helm Charts or Operators, which is how the large companies are testing Kafka in containers.
Related
I am running Kafka and Zookeeper (bitnami) locally on my M1 Macbook in Docker using Docker Compose. If I delete the data volumes for Kafka and Zookeeper and restart, it restarts perfectly and runs at a low CPU utilisation.
I have about 6 different consumer groups and 2 producers all running locally on the Mac that are different Node JS scripts. Everything is great, until I stop the consumers, then shutdown and restart the docker containers (Ctrl-C to stop, followed by Docker Compose Up).
When I do that everything looks to restart fine, but the producers and consumers get Connection Refused errors (though sometimes connect, but don't stay connected). There are no errors or warnings in the Kafka or Zookeeper logs, but when I look at the container for Kafka, Docker reports it is running at 100% CPU utilisation. I can leave it for a stay and it will day at that utilisation level.
I can always resolve this by stopping the containers and deleting the volumes associated with Kafka and Zookeeper. But, why is it doing this? Why is it every time I stop and restart Kafka it gets stuck at 100% CPU?
These are the relevant portions of my docker-compose.yml file:
version: '3.8'
services:
zookeeper:
container_name: zookeeper
image: 'bitnami/zookeeper'
pull_policy: always
environment:
ALLOW_ANONYMOUS_LOGIN: yes
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- '2181:2181'
volumes:
- /Users/localuser/Documents/MintMonsterData/zookeeper:/bitnami/zookeeper
kafka:
container_name: kafka
pull_policy: always
image: 'bitnami/kafka'
restart: always
ports:
- '29092:29092'
environment:
KAFKA_CFG_ADVERTISED_HOST_NAME: kafka
KAFKA_BROKER_ID: 1
KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_LISTENERS: CLIENT://:9092,EXTERNAL://:29092
KAFKA_CFG_ADVERTISED_LISTENERS: CLIENT://kafka:9092,EXTERNAL://localhost:29092
KAFKA_CFG_INTER_BROKER_LISTENER_NAME: CLIENT
KAFKA_CFG_LOG_RETENTION_HOURS: 24
ALLOW_PLAINTEXT_LISTENER: yes
KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
volumes:
- /Users/localuser/Documents/MintMonsterData/kafka:/bitnami/kafka
depends_on:
- zookeeper
I believe that it might be caused by the fact that Bitnami's docker images do not support arm64 architecture. See the following issue on Bitnami GitHub
Absolutely new to Docker.
I am trying to install Kafka image from hub and failing to do so. I keep getting below error (screenshot attached):
Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Here is what I did:
Downloaded Docker for Windows MSI and installed
Installed WSL 2 and Linux (Ubuntu)
Run CMD in admin mode and ran command "docker compose up" which keeps giving error
Troubleshooting: Things I tried hoping to allow me pull/up .yml but nothing worked
Enabled Virtualization
Changed DNS to 1.1.1.1 and then to 8.8.8.8
Restarted docker several times
Reinstalled docker
Switched debug mode on
Login with docker credentials in CMD
I have a WIFI internet connection with 30MBPS download speed.
I have no idea what to do. I am trying from last 6 hours to fix this.
YML file content
---
version: '2'
services:
zookeeper:
image: wurstmeister/cp-zookeeper:7.1.1
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_CREATE_TOPICS: "simpletalk_topic:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Snapshot of CMD running in Admin mode
I have a requirement that i need to setup kafka locally with topics already there in the container.I am using ladoop/fast-data-dev for doing that
How manually i am doing it-
docker run -d --name landoopkafka -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -e ADV_HOST=localhost landoop/fast-data-dev
After running this command my container is up and running.
now i go to bash inside this container like docker -exec -it landopkafka bash
and create topic using this command
kafka-topics --zookeeper localhost:2181 --create --topic hello_topic --partitions 1 --replication-factor 1
My topic is created.
But my requirement is i need to have a docker file which will have topic created and i just need to run it.
OR
A docker compose file which i need to run
Guys i need help on this ,as i am absolutely new to docker and kafka
I had to do it too! What if I did not want to use wurstmeister images? I decided to make a custom script which will do the job, and run this script in a separate container.
Repository
https://github.com/yan-khonski-it/kafka-compose
Note, it will work with kafka versions that use zookeeper.
Is Zookeeper a must for Kafka?
To start kafka with all your topics and zookeeper - docker-compose up -d.
Implementation details.
docker-compose.yml
# These services are kafka related. This docker-compose allows to start kafka locally quickly.
version: '2.1'
networks:
demo-network:
name: demo-network
driver: bridge
services:
zookeeper:
image: "confluentinc/cp-zookeeper:${CONFLUENT_PLATFORM_VERSION}"
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 32181:32181
hostname: zookeeper
networks:
- demo-network
kafka:
image: "confluentinc/cp-kafka:${CONFLUENT_PLATFORM_VERSION}"
container_name: kafka
hostname: kafka
ports:
- 9092:9092
- 29092:29092
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
KAFKA_BROKER_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_HOST://kafka:29092
LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
networks:
- demo-network
# Automatically creates required kafka topics if they were not created.
kafka-topics-creator:
build:
context: kafka-topic-creator
dockerfile: Dockerfile
container_name: kafka-topics-creator
depends_on:
- zookeeper
- kafka
environment:
ZOOKEEPER_HOSTS: "zookeeper:32181"
KAFKA_TOPICS: "topic_v1 topic_v2"
networks:
- demo-network
Then I have a directory kafka-topics-creator.
Here, I have three files
create-kafka-topics.sh, Dockerfile, README.md.
Dockerfile
# It is recommened to use same version as kafka broker is used.
# So no additional images are pulled.
FROM confluentinc/cp-kafka:4.1.2
WORKDIR usr/bin
# Once it is executed, this container is not needed.
COPY create-kafka-topics.sh create-kafka-topics.sh
ENTRYPOINT ["./create-kafka-topics.sh"]
create-kafka-topics.sh
#!/bin/bash
# Simply wait until original kafka container and zookeeper are started.
sleep 15.0s
# Parse string of kafka topics into an array
# https://stackoverflow.com/a/10586169/4587961
kafkatopicsArrayString="$KAFKA_TOPICS"
IFS=' ' read -r -a kafkaTopicsArray <<< "$kafkatopicsArrayString"
# A separate variable for zookeeper hosts.
zookeeperHostsValue=$ZOOKEEPER_HOSTS
# Create kafka topic for each topic item from split array of topics.
for newTopic in "${kafkaTopicsArray[#]}"; do
# https://kafka.apache.org/quickstart
kafka-topics --create --topic "$newTopic" --partitions 1 --replication-factor 1 --if-not-exists --zookeeper "$zookeeperHostsValue"
done
README.md - so other people know how to use it.Always document your stuff - good advise.
# Creates kafka topics automatically.
## Parameters
`ZOOKEEPER_HOSTS` - zookeeper hosts, I used value `"zookeeper:32181"` to run it locally.
`KAFKA_TOPICS` - space separated list of kafka topics. Example, `topic_1, topic_2, topic_3`.
Note, this container should run only **after** your original kafka broker and zookeeper are running.
After this container creates topics, it is not needed anymore.
How to check that the topics were created.
One solution is to check logs of kafka-topics-creator container.
docker logs kafka-topics-creator should print
$ docker logs kafka-topics-creator
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_v1".
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_v2".
You can create a docker-compose file like this...
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper:latest
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:0.10.2.1
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_CREATE_TOPICS: "MY_TOPIC_ONE:1:1,/
MY_TOPIC_TWO:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Put your topics there and run docker-compose up
You should instead try to use the wurstmeister/kafka image which supports an environment variable to create topics during container startup.
Sure, the Landoop container has a bunch of other useful things, but sounds like you only want Kafka and don't want to mess with editing any Dockerfiles
The other solution is to startup a second container after Kafka which runs the create scripts, then stops itself
After I carry out docker-compose up, it starts the containers.
when I do docker ps I get the below, which tells me that the containers are running. However when I do docker network inspect bridge the result shows me that there are no containers part of the docker0 bridge.
When I then carry out docker run meanchat_myserver it actually does show up on docker0 and I am also getting the data that the server is running on port 3000.
Which I don't get by using docker-compose.
What am I doing wrong here?
I am reading that when I use docker0 I can only refer to IP's to connect to other containers and not the name. Can I assume the ip's don't change on the containers and that this works without issue on deploying the app in production?
02cf08b1c3da d57f06ba9c68 "npm start" 33 minutes ago Up 33 minutes 4200/tcp meanchat_client_1
e257063c9e21 meanchat_myserver "npm start" 33 minutes ago Up 33 minutes 3000/tcp meanchat_myserver_1
02441c2e43f5 e114a298eabd "npm start" About an ago Up 33 minutes 0.0.0.0:80->80/tcp meanchat_nginx_1
88d9841d2553 mongo "docker-entrypoint..." 3 hours ago Up 3 hours 27017/tcp meanchat_mongo_1
compose
version: '3'
services:
# Build the container using the client Dockerfile
client:
build: ./
# This line maps the contents of the client folder into the container.
volumes:
- ./:/usr/src/app
myserver:
build: ./express-server
volumes:
- ./:/usr/src/app
depends_on:
- mongo
nginx:
build: ./nginx
# Map Nginx port 80 to the local machine's port 80
ports:
- "80:80"
# Link the client container so that Nginx will have access to it
mongo:
environment:
- AUTH=yes
- MONGO_INITDB_ROOT_USERNAME=superAdmin
- MONGO_INITDB_ROOT_PASSWORD=admin123
- MONGO_INITDB_DATABASE=d0c4ae452a5c
image: mongo
volumes:
- /var/mongodata/data:/data/db
By default Compose sets up a single network for your app.
For more detail, refer this link.
This means containers with compose won't be located in default bridge network by default.
You can check which network the containers with compose are using with the command.
docker inspect $container_name -f "{{.NetworkSettings.Networks}}"
However, If you want containers to be in default bridge network, you can use network_mode.
services:
service_name:
# other options....
network_mode: bridge
I'm trying to get JMX to work with spotify/kafka Docker image.
I have a docker-compose.yml with:
version: '2'
services:
my-kafka:
image: spotify/kafka # Using this instead of wurstmeister's because it didn't work with /var/run/docker.sock on Windows while this one has integrated ZK and works
hostname: my-kafka
restart: unless-stopped
ports:
- "9092:9092" # Kafka
- "2181:2181" # Zookeeper
- "7203:7203" # JMX
environment:
KAFKA_ADVERTISED_HOST_NAME: my-kafka
KAFKA_ADVERTISED_PORT: 9092
KAFKA_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.rmi.port=7203 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=my-kafka"
I then try to test it using jconsole my-kafka:7203 and getting a 'Connection failed: error during JRMP connection establishment; nested exception is: java.io.EOFException.' immediately. By contrast, using a non-existent host leads to "Unknown host" and a different port to "Connection refused".
docker ps shows:
692eb6659aba spotify/kafka "supervisord -n" 10 minutes ago Up 10 minutes 0.0.0.0:2181->2181/tcp, 0.0.0.0:7203->7203/tcp, 0.0.0.0:9092->9092/tcp docker_my-kafka_1
ps aux within the container shows:
root 11 1.8 3.3 5718844 274608 ? Sl 06:16 0:12 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/opt/kafka_2.11-0.10.1.0/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false ...offtopic skipped... -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.rmi.port=7203 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=my-kafka kafka.Kafka /opt/kafka_2.11-0.10.1.0/config/server.properties
The container log is boring without JMX being even mentioned, ends with "kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)".
There is a related question kafka monitoring using JMX when running kafka in docker in boot2docker but I'd prefer to stick to spotify/kafka so that ZK & Kafka live in the same container.
Can you try specifying the host address(where you are running the docker run) in the property :
-Djava.rmi.server.hostname=xx.xx.xx.xx
jmx options are already present in the starting script /opt/kafka_2.11-0.10.1.0/bin/kafka-run-class.sh of the spotify/kafka docker image.
You can grep them from inside the container
grep jmx /opt/kafka_2.11-0.10.1.0/bin/kafka-run-class.sh
and you see them at the beginning of java options of the jvm process inside the container
ps -ef java
So, it's enough you set JMX_PORT environment variable in docker-compose.yml and you remove additional (repeated) KAFKA_OPTS.
version: '2'
services:
my-kafka:
image: spotify/kafka # Using this instead of wurstmeister's because it didn't work with /var/run/docker.sock on Windows while this one has integrated ZK and works
hostname: my-kafka
restart: unless-stopped
ports:
- "9092:9092" # Kafka
- "2181:2181" # Zookeeper
- "7203:7203" # JMX
environment:
KAFKA_ADVERTISED_HOST_NAME: my-kafka
KAFKA_ADVERTISED_PORT: 9092
JMX_PORT: 7203
Then after starting a new container and having resolved my-kafka in your host, jconsole my-kafka:7203 will connect to the container's jmx port.
Docker containers run on an IP set within your machine. To see these, run:
docker inspect docker_my-kafka_1
This will output the IP that the machine is working on, something like:
"IPAddress": "172.17.0.2",
You will need to add the Ip found above to:
-Djava.rmi.server.hostname=$INSERT_IP_HERE
As you prefer to use a solution where both ZK & Kafka live in the same container, have a look at https://github.com/Landoop/fast-data-dev
This docker is focusing on providing an excellent Kafka Development Environment, so among others it exposes all JMXs