I'm trying to get JMX to work with spotify/kafka Docker image.
I have a docker-compose.yml with:
version: '2'
services:
my-kafka:
image: spotify/kafka # Using this instead of wurstmeister's because it didn't work with /var/run/docker.sock on Windows while this one has integrated ZK and works
hostname: my-kafka
restart: unless-stopped
ports:
- "9092:9092" # Kafka
- "2181:2181" # Zookeeper
- "7203:7203" # JMX
environment:
KAFKA_ADVERTISED_HOST_NAME: my-kafka
KAFKA_ADVERTISED_PORT: 9092
KAFKA_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.rmi.port=7203 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=my-kafka"
I then try to test it using jconsole my-kafka:7203 and getting a 'Connection failed: error during JRMP connection establishment; nested exception is: java.io.EOFException.' immediately. By contrast, using a non-existent host leads to "Unknown host" and a different port to "Connection refused".
docker ps shows:
692eb6659aba spotify/kafka "supervisord -n" 10 minutes ago Up 10 minutes 0.0.0.0:2181->2181/tcp, 0.0.0.0:7203->7203/tcp, 0.0.0.0:9092->9092/tcp docker_my-kafka_1
ps aux within the container shows:
root 11 1.8 3.3 5718844 274608 ? Sl 06:16 0:12 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/opt/kafka_2.11-0.10.1.0/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false ...offtopic skipped... -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.rmi.port=7203 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=my-kafka kafka.Kafka /opt/kafka_2.11-0.10.1.0/config/server.properties
The container log is boring without JMX being even mentioned, ends with "kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)".
There is a related question kafka monitoring using JMX when running kafka in docker in boot2docker but I'd prefer to stick to spotify/kafka so that ZK & Kafka live in the same container.
Can you try specifying the host address(where you are running the docker run) in the property :
-Djava.rmi.server.hostname=xx.xx.xx.xx
jmx options are already present in the starting script /opt/kafka_2.11-0.10.1.0/bin/kafka-run-class.sh of the spotify/kafka docker image.
You can grep them from inside the container
grep jmx /opt/kafka_2.11-0.10.1.0/bin/kafka-run-class.sh
and you see them at the beginning of java options of the jvm process inside the container
ps -ef java
So, it's enough you set JMX_PORT environment variable in docker-compose.yml and you remove additional (repeated) KAFKA_OPTS.
version: '2'
services:
my-kafka:
image: spotify/kafka # Using this instead of wurstmeister's because it didn't work with /var/run/docker.sock on Windows while this one has integrated ZK and works
hostname: my-kafka
restart: unless-stopped
ports:
- "9092:9092" # Kafka
- "2181:2181" # Zookeeper
- "7203:7203" # JMX
environment:
KAFKA_ADVERTISED_HOST_NAME: my-kafka
KAFKA_ADVERTISED_PORT: 9092
JMX_PORT: 7203
Then after starting a new container and having resolved my-kafka in your host, jconsole my-kafka:7203 will connect to the container's jmx port.
Docker containers run on an IP set within your machine. To see these, run:
docker inspect docker_my-kafka_1
This will output the IP that the machine is working on, something like:
"IPAddress": "172.17.0.2",
You will need to add the Ip found above to:
-Djava.rmi.server.hostname=$INSERT_IP_HERE
As you prefer to use a solution where both ZK & Kafka live in the same container, have a look at https://github.com/Landoop/fast-data-dev
This docker is focusing on providing an excellent Kafka Development Environment, so among others it exposes all JMXs
Related
running Colima with reachable ip:
colima start --network-address
then colima list
PROFILE STATUS ARCH CPUS MEMORY DISK RUNTIME ADDRESS
default Running x86_64 4 8GiB 100GiB docker 192.168.106.2
and after the above, I'm running my docker-compose:
version: "3"
services:
kafka:
image: docker.io/bitnami/kafka:3.3
ports:
- "9092:9092"
volumes:
- "kafka_data:/bitnami"
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
So I'm basically forwarding port "9092" on the kafka container to the host port "9092".
I tried to reach kafka through:
"192.168.106.2:9092"
"192.168.106.2"
"127.0.0.1:9092"
"localhost:9092"
"0.0.0.0:9092"
but I getting an error.
I know kafkaa container is working as I can access it, if Im trying to use it through a container inside the docker network.
how can I access it from my host machine?
I have Kafka and Zookeeper running on two separate Docker containers:
<private-domain>/wurstmeister-kafka:0.10.1.0-2
<private-domain>/wurstmeister-zookeeper:3.4.9
Both containers seem to be up, but when I try to create Kafka topics by getting in to the first container:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
I get this error:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2020-06-07 03:10:55,293] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
Please notice that I did read other related questions and tried adding arguments to the command, such as -e ZK_HOSTS="localhost:2181". I know of other people working in the environment as mine who were able to run the commands successfully, so I suspect this might be a configuration issue on my side. Can you please guide?
EDIT: Here are the Docker Compose files:
version: '2'
services:
kafka:
image: <private-domain>/wurstmeister-kafka:0.10.1.0-2
container_name: kafka
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: 127.0.0.1:2181
restart:
"unless-stopped"
and
version: '2'
services:
zk:
image: <private-domain>/wurstmeister-zookeeper:3.4.9
container_name: zk
ports:
- "2181:2181"
restart:
"unless-stopped"
and the output of docker ps:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bf67a49da57a wurstmeister-kafka:0.10.1.0-2 "start-kafka.sh" 5 months ago Up 29 minutes 0.0.0.0:9092->9092/tcp kafka
ef3e908d82b3 wurstmeister-zookeeper:3.4.9 "/bin/sh -c '/usr/sbin/sshd && bash /usr/bin/start-zk.sh'" 5 months ago Up 29 minutes 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp zk
You have two Compose files. Thus, Your containers are on separated networks, and cannot refer each other.
You must add both services in one file, under one services: block, and run only one docker-compose up command
You can find working compose files here across the internet, or you could use minikube / oc with Kafka Helm Charts or Operators, which is how the large companies are testing Kafka in containers.
On my linux server, I am running 3 images -
A) Docker and Zookeeper with this docker-compose file -
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper:3.4.6
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:2.11-2.0.0
ports:
- "9092:9092"
expose:
- "9093"
environment:
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
This will open up the kafka broker to the host machine.
B) JupyterHub
docker run -v /notebooks:/notebooks -p 8000:8000 jupyterhub
C) Confluent Schema Registry (I have not tried it yet, but in my final setup I will have a schema registry container as well)
docker run confluentinc/cp-schema-registry
Both are starting up without any issues. But how do I open up jupyterhub container to kafka container and schema registry ports so that my python scripts can access the brokers.
I'm assuming you want to run your jupyter notebook container on demand whereas your zookeeper and kafka containers will always be running separately? You can create a docker network and join all the containers to this network. Then your containers will be able resolve each other by their names.
Create a network
Specify this network in compose file
When starting your other containers with docker run, use --network option.
If you run docker network ls then you can find the name of the network that Compose creates for you; it will be named something like directoryname_default. You can then launch the containers connected to that network,
docker run --net directoryname_default confluentinc/cp-schema-registry
If you can include these files in the same docker-compose.yml file then you won’t need to do anything special. In particular this probably makes sense for the Confluent schema registry, which you can consider a core part of the Kafka stack if you’re using Avro messages.
You can use the Docker Compose service name kafka as a host name here, but since you need to connect to the “inside” listener you’ll need to configure a non-default port 9093. (The Docker Compose expose: directive doesn’t do much and you can safely delete it.)
There are so many answers for this question that I ended up being totally confused about how I can connect to Kafka docker container from an outside client.
I have created two docker machines, a manager and a worker with these commands:
docker-machine create manager
docker-machine create worker1
I have add these two nodes inside a docker swarm.
docker#manager:~$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
6bmovp3hr0j2w5irmexvvjgzq * manager Ready Active Leader 19.03.5
mtgbd9bg8d6q0lk9ycw10bxos worker1 Ready Active 19.03.5
docker-compose.yml:
version: '3.2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:latest
ports:
- target: 9094
published: 9094
protocol: tcp
mode: host
environment:
HOSTNAME_COMMAND: "hostname | awk -F'-' '{print $$2}'"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
volumes:
- /var/run/docker.sock:/var/run/docker.sock
From inside docker, everything works fine. I can create topics and then produce/consume messages.
I created a python script in order to consume messages from outside docker. The simple code is presented below:
from kafka import KafkaConsumer
import json
try:
print('Welcome to parse engine')
consumer = KafkaConsumer('streams-plaintext-input', bootstrap_servers='manager:9094')
for message in consumer:
print(message)
except Exception as e:
print(e)
# Logs the error appropriately.
pass
But the code is stack forever. The connection is not correct. Can anyone provide any help on how to setup a connection?
Since you are using docker-machine you have to either
Run your code also in a container (using kafka:9092)
Run your code within the VM OS (using vm-host-name:9094)
Add PLAINTEXT://localhost:9096 to the advertised listeners, expose 9096 from the VM to your host, then use localhost:9096 in your code (note: 9096 is some random port)
The gist is that clients must be able to connect to the bootstrap address and the advertised one that is being returned. If it cannot connect to the second, code will timeout.
I have a docker-compose file with several service-container definitions. One of the services communicates with Apache Kafka within the same docker-compose run.
So I have the kafka docker definition like this:
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: 127.0.0.1
ADVERTISED_PORT: 9092
I have my service definition in the same docker-compose file. In the startup script of the service I have to figure out somehow the IP address of the Kafka instance.
I know, I can use something like docker inspect to find out which IP address is used by a container.
But how can I do it dynamically in a docker-compose environment?
EDIT
So, the right configuration should be (thank you, #nwinkler):
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: kafka
ADVERTISED_PORT: 9092
myservice:
image: foo
links:
- kafka:kafka
Don't forget to set the ADVERTISED_HOST to kafka (or how you named your kafka container within docker-compose).
You can use the Docker Compose Links feature for this. If you provide a link to the kafka container from your other container, Docker Compose will ensure that your other container can access the Kafka container through its hostname - you will not have to know its IP address.
Example:
kafka:
image: spotify/kafka
ports:
- "2181:2181"
- "9092:9092"
environment:
ADVERTISED_HOST: 127.0.0.1
ADVERTISED_PORT: 9092
myservice:
image: foo
links:
- kafka:kafka
This will allow your myservice container to access the Kafka container through the kafka hostname. So from your myservice container, you can do something like curl http://kafka:9092 to access the service on the Kafka container.
Docker-Compose does this through DNS, it creates a hostname/IP mapping in your container allowing you to access the container without knowing its IP address.
The ip of your container will be the ip you are looking for.
Append the port number (9092 in your case) to the ip of the container to get whatever kafka is serving.