I am running following docker containers, and trying to understand that broker1 is diverting traffic from 0.0.0.0:9092 to 9092 tcp.
However, what I don't understand that why 9092/tcp is mentioned in broker2, and broker3.
Please can you explain what it mean and how it should be read?
3cd460e3fa52 cp-kafka:5.4.0 "/etc…" **9092/tcp**, 0.0.0.0:9093->9093/tcp broker2
b4f98058eadf cp-kafka:5.4.0 "/etc…" **9092/tcp**, 0.0.0.0:9094->9094/tcp broker3
0886c9d90674 cp-kafka:5.4.0 "/etc…" 0.0.0.0:9092->9092/tcp broker1
thank you,
Even if you're not exposing the port when you run the container, the image itself is baked to expose it, per this line in the Dockerfile:
EXPOSE 9092
But if you don't configure your KAFKA_LISTENERS to use it, then whilst the port is exposed, nothing is listening on it. The Dockerfile docs for EXPOSE explain this more.
So in short, this is just an aberration that you can ignore, assuming that your listeners are functioning correctly.
Here's an example against this Docker Compose with three brokers. Each has two listeners (internal and external), and exposes just the external one (see here for why).
The relevant part of the yaml is this:
kafka-1:
image: confluentinc/cp-kafka:5.4.0
ports:
- 9092:9092
environment:
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:39092,HOST://0.0.0.0:9092
…
kafka-2:
image: confluentinc/cp-kafka:5.4.0
ports:
- 19092:19092
environment:
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:49092,HOST://0.0.0.0:19092
…
kafka-3:
image: confluentinc/cp-kafka:5.4.0
ports:
- 29092:29092
environment:
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:59092,HOST://0.0.0.0:29092
With it running, docker ps shows the ports exposed as defined, along with 9092 for kafka-2 and kafka-3:
➜ docker ps|grep 9092
b227c8fbeec8 confluentinc/cp-kafka:5.4.0 […] 9092/tcp, 0.0.0.0:19092->19092/tcp kafka-2
ce1f8c575ed7 confluentinc/cp-kafka:5.4.0 […] 9092/tcp, 0.0.0.0:29092->29092/tcp kafka-3
f914a18d9757 confluentinc/cp-kafka:5.4.0 […] 0.0.0.0:9092->9092/tcp kafka-1
But if we test if the port is open on each container, we can see that it is only on kafka-1 where we have specified it as one of the KAFKA_LISTENERS:
➜ docker exec -it kafka-1 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) open
➜ docker exec -it kafka-2 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused
➜ docker exec -it kafka-3 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused
Related
This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed 6 months ago.
I'm trying to use Debezium with Kafka connect, I followed this tutorial, and everything connected just fine. However, the problem is that I cannot access Kafka from outside of docker containers anymore.
I use these commands to start containers:
docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:2.0.0.Beta1
docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:2.0.0.Beta1
docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets --link kafka:kafka debezium/connect:2.0.0.Beta1
I tried to set KAFKA_ADVERTISED_LISTENERS to PLAINTEXT://127.0.0.1:9092 which allowed me to connect to Kafka from the outside of the container but I could not connect from connect container to kafka container anymore. How can I achieve both?
with this you can access the kafka container from your host on the port 9092
zookeeper:
image: confluentinc/cp-zookeeper:7.2.0
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka-broker:
image: confluentinc/cp-kafka:7.2.0
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-broker:29092,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
I think it's not a Kafka issue, but a docker network one. It's probably accessible via docker network or you need to expose it. https://docs.docker.com/network/network-tutorial-standalone/
This probably just related to WSL in general but Redis is my use case.
This works fine and I can connect like:
docker exec -it redis-1 redis-cli -c -p 7001 -a Password123
But I cannot make any connections from my local windows pc to the container. I get
Could not connect: Error 10061 connecting to host.docker.internal:7001. No connection could be made because the target machine actively refused it.
This is the same error when the container isn't running, so not sure if it's a docker issue or WSL?
version: '3.9'
services:
redis-cluster:
image: redis:latest
container_name: redis-cluster
command: redis-cli -a Password123 -p 7001 --cluster create 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 --cluster-replicas 1 --cluster-yes
depends_on:
- redis-1
- redis-2
- redis-3
- redis-4
- redis-5
- redis-6
network_mode: host
redis-1:
image: "redis:latest"
container_name: redis-1
network_mode: host
entrypoint: >
redis-server
--port 7001
--appendonly yes
--cluster-enabled yes
--cluster-config-file nodes.conf
--cluster-node-timeout 5000
--masterauth Password123
--requirepass Password123
--bind 0.0.0.0
--protected-mode no
# Five more the same as the above
According to the provided docker-compose.yml file, container ports are not exposed, so they are unreachable from the outside (your windows/wls host). Check here for the official reference. More about docker and ports here
As an example for redis-1 service, you should add the following to the definition.
...
redis-1:
ports:
- 7001:7001
...
...
The docker exec ... is working because the port is reachable from inside the container.
I have a situation where, Kafka is running in a docker container using a specific IP address within a network. The network is created using the following command
sudo docker network create --subnet=172.19.0.0/16 --gateway 172.19.0.1 --ip-range=172.19.0.1/24 my_net
Kafka container is started using the following
docker run -d --name kafkanode --net my_net --hostname=kafkahost01 kafka_zook:212-358 -p 2181:2181 -p 9092:9092 tail -f /dev/null
I have producers within the same host from a different container.
Kafka's server.properties a simple configuration like the below works for a producer within the same host and from a different container.
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafkahost01:9092
However, in our case, we will have producers who will also be sending messages from outside of that machine.
Unfortunately, i am not able to get connected from outside the docker host machine. Can someone please help me with the configuration?
We are using Kafka 2.12-2.6.0
Zookeeper -- 3.5.8
Server properties edited with the following values
listeners=INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://kafkahost01:29092,EXTERNAL://10.20.30.40:9092
inter.broker.listener.name=INTERNAL
Thanks
Balaji
Here you have a docker-compose example with inside and outside listeners configured. Try out.
(Replace localhost with your desired IP or DNS)
version: '3.7'
services:
zookeeper:
image: zookeeper:3.5.8
hostname: zookeeper
volumes:
- zookeeper-data:/data
- zookeeper-datalog:/datalog
kafka:
image: wurstmeister/kafka:2.13-2.6.0
hostname: kafka
depends_on:
- zookeeper
ports:
- 9093:9093
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://localhost:9093
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9093
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
volumes:
- kafka:/kafka
volumes:
zookeeper-data:
zookeeper-datalog:
kafka:
Running a producer within the same network:
# note: I just placed my docker-compose.yml in example dir, thats the reason for the example_default network
$ docker run -it --rm \
--name producer \
--network example_default \
wurstmeister/kafka:2.13-2.6.0 bash
bash-4.4# /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka:9092 --topic
example
>some
>test
And consuming from outside docker using kaf:
$ cat ~/.kaf/config
current-cluster: single
clusteroverride: ""
clusters:
- name: single
version: 2.7.0
brokers:
- localhost:9093
SASL: null
TLS: null
security-protocol: PLAINTEXT
schema-registry-url: ""
$ kaf nodes
ID ADDRESS
1 localhost:9093
$ kaf consume example -f --raw
some
test
Hope this example can help you define your own setup.
This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed last month.
I want to expost 9093 to outside of the docker container. When I set the kafka-0 ports exposed to 9093 and the KAFKA_ADVERTISED_LISTENERS as follow, i am unable to connect to localhost:9093 as shown in the following docker-compose file.
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.2.1
container_name: kafka-0
hostname: kafka-0
ports:
- "9093:9092"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=wise-nlp-zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
However, when i change to
ports:
- "9092:9092"
and
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9092
I am able to connect to kafka broker localhost:9092.
How can i change external port to 9093 for applications to connect? I want to set up multiple brokers.
Why's it not working currently?
Advertised listener(s) (as defined in KAFKA_ADVERTISED_LISTENERS) are the host and port that the broker returns to the client in its initial connection for it to use in subsequent connections.
If you want external clients to use 9093 then KAFKA_ADVERTISED_LISTENERS=…PLAINTEXT_HOST://localhost:9093 is correct. However, you've not configured your KAFKA_LISTENERS, which if you check the broker log when it starts up will default to the value set by KAFKA_ADVERTISED_LISTENERS:
kafka-0 | listeners = PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093
So in this state, the broker is listening on port 9093, but with this Docker Compose instruction you've redirected external connections into the container on 9093 to 9092 within the container:
ports:
- "9093:9092"
➜ docker ps
CONTAINER ID IMAGE … PORTS NAMES
8b934ef4145c confluentinc/cp-kafka:5.4.1 … 0.0.0.0:9093->9092/tcp kafka-0
So your external connections will go to port 9092 in the container—and the broker is not listening on this port. You can verify this with nc:
-- Port 9093 is open on the host machine
➜ nc -vz localhost 9093
Connection to localhost port 9093 [tcp/*] succeeded!
-- Port 9092 is _not_ open on the Kafka container
➜ docker exec -it kafka-0 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused
❌ You'll see that a client connection fails
➜ kafkacat -b localhost:9093 -L
% ERROR: Failed to acquire metadata: Local: Broker transport failure
How can you fix it?
You can either:
Change the listener to be on the port that you target with the Docker port redirect. This will work but personally I think is more confusing.
Change the Docker port redirect to target the port on which the listener is on. This is the option I would use as it is clearer (e.g. port 9093 is used throughout, rather than mixing 9092 and 9093 together)
Option 1: Change the listener to be on the port that you target with the Docker port redirect
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.4.1
container_name: kafka-0
ports:
- "9093:9092"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9092
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.4.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
✅Test:
➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
1 brokers:
broker 1 at localhost:9093 (controller)
Option 2: Change the Docker port redirect to target the port on which the listener is on
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.4.1
container_name: kafka-0
ports:
- "9093:9093"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
# If you don't specify KAFKA_LISTENERS it will default to the ports used in
# KAFKA_ADVERTISED_LISTENERS, but IMO it's better to be explicit about these settings
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.4.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
✅Test
➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
1 brokers:
broker 1 at localhost:9093 (controller)
Connecting to Kafka from within the Docker network
The examples above are about connecting to Kafka from the Docker host. If you want to connect to it from within the Docker network (e.g. another container) you need to use kafka-0:29094 as the broker host and IP. If you try to use localhost:9093 then the client container will resolve localhost to its own container, and thus fail.
Multiple brokers
See here for an example Docker Compose with multiple Kafka brokers.
References
https://rmoff.net/2018/08/02/kafka-listeners-explained/
I installed zookeeper and kafka docker container on virtual machine. MY settings look like
zookeeper:
image: confluentinc/cp-zookeeper:latest
name:zookeeper
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "2181:2181"
kafka:
image: confluentinc/cp-kafka:latest
hostname: kafka
ports:
- "9092:9092"
- "29092:29092"
depends_on:
- zookeeper
KAFKA_BROKER_ID: "-1"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181");
ALLOW_PLAINTEXT_LISTENER: "yes"
KAFKA_LISTENERS: "PLAINTEXT://9092, PLAINTEXT_HOST://29092"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092, PLAINTEXT_HOST://localhost:29092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
KAFKA_DEFAULT_REPLICATION_FACTOR: "1"
AUTO.LEADER.REBALANCE.ENABLE: "true"
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
Now i am trying to send some message from my local machine to kafka, but i am getting the following error.
Connection to node -1 (//ip-of-virtualmachine:29092) could not be established. Broker may not be available.
From my local machine i configure kafka producer to send messages. the property in my code which configure the bootstrap server is
String bootstrapServers = "virtual-machine-ip:29092";
Properties kafka-properties = new Properties();
kafka-properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
.
.
.
create producer and send some record etc.
All the example i find over internet are regarding the docker and kafka. None of them are related to kafka, docker and virtual machine. Any help would be appreciated.
Probably you didn't expose port of your kafka container to your host, try to expose port 29092 via:
ports:
- "29092:29092"
or run your container in host network docker -d --net=host... if you run it from docker-compose add network_mode: host to your services
to fix your docker config you can take inspiration here:
https://github.com/simplesteph/kafka-stack-docker-compose/blob/master/zk-single-kafka-single.yml
If your code is running outside of the VM, you'll need to make you have an external port forward from the VM to the host, and through the container.
By the way , Docker Machine would create VMs for you, rather than you installing Docker in one on your own