Kafka in a Docker Container - External and and Internal connections - docker

Question

I have a situation where, Kafka is running in a docker container using a specific IP address within a network. The network is created using the following command
sudo docker network create --subnet=172.19.0.0/16 --gateway 172.19.0.1 --ip-range=172.19.0.1/24 my_net
Kafka container is started using the following
docker run -d --name kafkanode --net my_net --hostname=kafkahost01 kafka_zook:212-358 -p 2181:2181 -p 9092:9092 tail -f /dev/null
I have producers within the same host from a different container.
Kafka's server.properties a simple configuration like the below works for a producer within the same host and from a different container.
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafkahost01:9092
However, in our case, we will have producers who will also be sending messages from outside of that machine.
Unfortunately, i am not able to get connected from outside the docker host machine. Can someone please help me with the configuration?
We are using Kafka 2.12-2.6.0
Zookeeper -- 3.5.8
Server properties edited with the following values
listeners=INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://kafkahost01:29092,EXTERNAL://10.20.30.40:9092
inter.broker.listener.name=INTERNAL
Thanks
Balaji

Answer 1

Here you have a docker-compose example with inside and outside listeners configured. Try out.
(Replace localhost with your desired IP or DNS)
version: '3.7'
services:
zookeeper:
image: zookeeper:3.5.8
hostname: zookeeper
volumes:
- zookeeper-data:/data
- zookeeper-datalog:/datalog
kafka:
image: wurstmeister/kafka:2.13-2.6.0
hostname: kafka
depends_on:
- zookeeper
ports:
- 9093:9093
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://localhost:9093
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9093
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
volumes:
- kafka:/kafka
volumes:
zookeeper-data:
zookeeper-datalog:
kafka:
Running a producer within the same network:
# note: I just placed my docker-compose.yml in example dir, thats the reason for the example_default network
$ docker run -it --rm \
--name producer \
--network example_default \
wurstmeister/kafka:2.13-2.6.0 bash
bash-4.4# /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka:9092 --topic
example
>some
>test
And consuming from outside docker using kaf:
$ cat ~/.kaf/config
current-cluster: single
clusteroverride: ""
clusters:
- name: single
version: 2.7.0
brokers:
- localhost:9093
SASL: null
TLS: null
security-protocol: PLAINTEXT
schema-registry-url: ""
$ kaf nodes
ID ADDRESS
1 localhost:9093
$ kaf consume example -f --raw
some
test
Hope this example can help you define your own setup.