Kafka docker compose external connection [duplicate] - docker

This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed last month.
I want to expost 9093 to outside of the docker container. When I set the kafka-0 ports exposed to 9093 and the KAFKA_ADVERTISED_LISTENERS as follow, i am unable to connect to localhost:9093 as shown in the following docker-compose file.
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.2.1
container_name: kafka-0
hostname: kafka-0
ports:
- "9093:9092"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=wise-nlp-zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
However, when i change to
ports:
- "9092:9092"
and
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9092
I am able to connect to kafka broker localhost:9092.
How can i change external port to 9093 for applications to connect? I want to set up multiple brokers.

Why's it not working currently?
Advertised listener(s) (as defined in KAFKA_ADVERTISED_LISTENERS) are the host and port that the broker returns to the client in its initial connection for it to use in subsequent connections.
If you want external clients to use 9093 then KAFKA_ADVERTISED_LISTENERS=…PLAINTEXT_HOST://localhost:9093 is correct. However, you've not configured your KAFKA_LISTENERS, which if you check the broker log when it starts up will default to the value set by KAFKA_ADVERTISED_LISTENERS:
kafka-0 | listeners = PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093
So in this state, the broker is listening on port 9093, but with this Docker Compose instruction you've redirected external connections into the container on 9093 to 9092 within the container:
ports:
- "9093:9092"
➜ docker ps
CONTAINER ID IMAGE … PORTS NAMES
8b934ef4145c confluentinc/cp-kafka:5.4.1 … 0.0.0.0:9093->9092/tcp kafka-0
So your external connections will go to port 9092 in the container—and the broker is not listening on this port. You can verify this with nc:
-- Port 9093 is open on the host machine
➜ nc -vz localhost 9093
Connection to localhost port 9093 [tcp/*] succeeded!
-- Port 9092 is _not_ open on the Kafka container
➜ docker exec -it kafka-0 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused
❌ You'll see that a client connection fails
➜ kafkacat -b localhost:9093 -L
% ERROR: Failed to acquire metadata: Local: Broker transport failure
How can you fix it?
You can either:
Change the listener to be on the port that you target with the Docker port redirect. This will work but personally I think is more confusing.
Change the Docker port redirect to target the port on which the listener is on. This is the option I would use as it is clearer (e.g. port 9093 is used throughout, rather than mixing 9092 and 9093 together)
Option 1: Change the listener to be on the port that you target with the Docker port redirect
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.4.1
container_name: kafka-0
ports:
- "9093:9092"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9092
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.4.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
✅Test:
➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
1 brokers:
broker 1 at localhost:9093 (controller)
Option 2: Change the Docker port redirect to target the port on which the listener is on
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.4.1
container_name: kafka-0
ports:
- "9093:9093"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
# If you don't specify KAFKA_LISTENERS it will default to the ports used in
# KAFKA_ADVERTISED_LISTENERS, but IMO it's better to be explicit about these settings
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.4.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
✅Test
➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
1 brokers:
broker 1 at localhost:9093 (controller)
Connecting to Kafka from within the Docker network
The examples above are about connecting to Kafka from the Docker host. If you want to connect to it from within the Docker network (e.g. another container) you need to use kafka-0:29094 as the broker host and IP. If you try to use localhost:9093 then the client container will resolve localhost to its own container, and thus fail.
Multiple brokers
See here for an example Docker Compose with multiple Kafka brokers.
References
https://rmoff.net/2018/08/02/kafka-listeners-explained/

Related

connect docker container to local site

I have services openhab and mosquitto.
I have internal network between openhab and mosquitto, it is ok
I have in local network 3 computers 192.168.1.16, 192.168.1.17, 192.168.1.18
on 192.168.1.16 run docker and mosquitto container
Now I need add for mosquitto container new ip 192.168.1.20, because I need send data from others computers in network to mosquitto
How can I do it?
my docker-compose file
version: '3.7'
services:
openhab:
image: "openhab/openhab:3.3.0"
container_name: "openhab"
restart: always
networks:
openhabnet:
aliases:
- openhab
ports:
- 8082:8080
- 8444:8443
volumes:
- "/etc/localtime:/etc/localtime:ro"
- "/etc/timezone:/etc/timezone:ro"
- "./openhab_addons:/openhab/addons"
- "./openhab_conf:/openhab/conf"
- "./openhab_userdata:/openhab/userdata"
environment:
CRYPTO_POLICY: "unlimited"
EXTRA_JAVA_OPTS: "-Duser.timezone=Europe/Berlin"
OPENHAB_HTTP_PORT: "8080"
OPENHAB_HTTPS_PORT: "8443"
USER_ID: "1000"
GROUP_ID: "1000"
mosquitto:
image: "eclipse-mosquitto:latest"
container_name: "mosquitto"
user: "1000:1000"
restart: always
networks:
openhabnet:
aliases:
- mosquitto
ports:
- 1884:1883
- 9001:9001
volumes:
- "./mosquitto/config:/mosquitto/config"
- "./mosquitto/log:/mosquitto/log"
- "./mosquitto/data:/mosquitto/data"
environment:
- TZ=Europe/Bratislava
networks:
openhabnet:
driver: bridge
Your mosquito container is already reacheable on the hosts network with the ip of the docker host, 192.168.1.16 and on the ports you forwarded:
ports:
- 1884:1883
- 9001:9001
So on 192.168.1.16:1884 you can reach the mosquito containers 1883 port and 192.168.1.16:9001 you can reach the mosquito container 9001 port from your other computers too, given you allowed these on the firewalls on the computers, including the docker host.
But if you really want an IP for the mosquito container itself on your host network then you will need to do macvlan: https://docs.docker.com/network/macvlan/
With this your container will get a virtual NIC and will connect to the physical network the docker host is running on. But I think you won't need this, please further explain your use case.

Cannot connect to docker container (redis) in host mode

This probably just related to WSL in general but Redis is my use case.
This works fine and I can connect like:
docker exec -it redis-1 redis-cli -c -p 7001 -a Password123
But I cannot make any connections from my local windows pc to the container. I get
Could not connect: Error 10061 connecting to host.docker.internal:7001. No connection could be made because the target machine actively refused it.
This is the same error when the container isn't running, so not sure if it's a docker issue or WSL?
version: '3.9'
services:
redis-cluster:
image: redis:latest
container_name: redis-cluster
command: redis-cli -a Password123 -p 7001 --cluster create 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 --cluster-replicas 1 --cluster-yes
depends_on:
- redis-1
- redis-2
- redis-3
- redis-4
- redis-5
- redis-6
network_mode: host
redis-1:
image: "redis:latest"
container_name: redis-1
network_mode: host
entrypoint: >
redis-server
--port 7001
--appendonly yes
--cluster-enabled yes
--cluster-config-file nodes.conf
--cluster-node-timeout 5000
--masterauth Password123
--requirepass Password123
--bind 0.0.0.0
--protected-mode no
# Five more the same as the above
According to the provided docker-compose.yml file, container ports are not exposed, so they are unreachable from the outside (your windows/wls host). Check here for the official reference. More about docker and ports here
As an example for redis-1 service, you should add the following to the definition.
...
redis-1:
ports:
- 7001:7001
...
...
The docker exec ... is working because the port is reachable from inside the container.

Understanding docker port diversion

I am running following docker containers, and trying to understand that broker1 is diverting traffic from 0.0.0.0:9092 to 9092 tcp.
However, what I don't understand that why 9092/tcp is mentioned in broker2, and broker3.
Please can you explain what it mean and how it should be read?
3cd460e3fa52 cp-kafka:5.4.0 "/etc…" **9092/tcp**, 0.0.0.0:9093->9093/tcp broker2
b4f98058eadf cp-kafka:5.4.0 "/etc…" **9092/tcp**, 0.0.0.0:9094->9094/tcp broker3
0886c9d90674 cp-kafka:5.4.0 "/etc…" 0.0.0.0:9092->9092/tcp broker1
thank you,
Even if you're not exposing the port when you run the container, the image itself is baked to expose it, per this line in the Dockerfile:
EXPOSE 9092
But if you don't configure your KAFKA_LISTENERS to use it, then whilst the port is exposed, nothing is listening on it. The Dockerfile docs for EXPOSE explain this more.
So in short, this is just an aberration that you can ignore, assuming that your listeners are functioning correctly.
Here's an example against this Docker Compose with three brokers. Each has two listeners (internal and external), and exposes just the external one (see here for why).
The relevant part of the yaml is this:
kafka-1:
image: confluentinc/cp-kafka:5.4.0
ports:
- 9092:9092
environment:
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:39092,HOST://0.0.0.0:9092
…
kafka-2:
image: confluentinc/cp-kafka:5.4.0
ports:
- 19092:19092
environment:
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:49092,HOST://0.0.0.0:19092
…
kafka-3:
image: confluentinc/cp-kafka:5.4.0
ports:
- 29092:29092
environment:
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:59092,HOST://0.0.0.0:29092
With it running, docker ps shows the ports exposed as defined, along with 9092 for kafka-2 and kafka-3:
➜ docker ps|grep 9092
b227c8fbeec8 confluentinc/cp-kafka:5.4.0 […] 9092/tcp, 0.0.0.0:19092->19092/tcp kafka-2
ce1f8c575ed7 confluentinc/cp-kafka:5.4.0 […] 9092/tcp, 0.0.0.0:29092->29092/tcp kafka-3
f914a18d9757 confluentinc/cp-kafka:5.4.0 […] 0.0.0.0:9092->9092/tcp kafka-1
But if we test if the port is open on each container, we can see that it is only on kafka-1 where we have specified it as one of the KAFKA_LISTENERS:
➜ docker exec -it kafka-1 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) open
➜ docker exec -it kafka-2 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused
➜ docker exec -it kafka-3 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused

Problems running Kafka with Docker Compose in Windows

I'm trying to run Kafka locally on Windows 10 Pro and Docker Desktop (not toolbox). Everything seems to work perfectly, but I can't reach Kafka with my application and neither use kafka rest (http://localhost:8082/topics | http://127.0.0.1:8082/topics | http://192.168.1.103:8082/topics - this last one is my docker ip in hosts)
My docker-compose file is:
version: '2'
services:
# https://hub.docker.com/r/confluentinc/cp-zookeeper/tags
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
hostname: zookeeper
network_mode: host
ports:
- "2181:2181"
- "32181:32181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
# https://hub.docker.com/r/confluentinc/cp-kafka/tags
kafka:
image: confluentinc/cp-kafka:5.3.1
container_name: kafka
hostname: kafka
network_mode: host
ports:
- "9092:9092"
- "29092:29092"
restart: always
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_ZOOKEEPER_CONNECT: localhost:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_BROKER_ID: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
depends_on:
- zookeeper
schema-registry:
image: confluentinc/cp-schema-registry:5.3.1
hostname: schema-registry
container_name: schema-registry
network_mode: host
depends_on:
- zookeeper
- kafka
ports:
- "8081:8081"
environment:
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: localhost:2181
SCHEMA_REGISTRY_HOST_NAME: localhost
SCHEMA_REGISTRY_LISTENERS: http://localhost:8081
kafka-rest:
image: confluentinc/cp-kafka-rest:5.3.1
hostname: kafka-rest
container_name: kafka-rest
network_mode: host
depends_on:
- zookeeper
- kafka
ports:
- "8082:8082"
environment:
KAFKA_REST_HOST_NAME: localhost
KAFKA_REST_ZOOKEEPER_CONNECT: localhost:2181
KAFKA_REST_LISTENERS: http://localhost:8082
KAFKA_REST_SCHEMA_REGISTRY_URL: http://localhost:8081
And my hosts file is:
# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host
# localhost name resolution is handled within DNS itself.
# 127.0.0.1 localhost
# ::1 localhost
192.168.1.103 host.docker.internal
192.168.1.103 gateway.docker.internal
# Added by Docker Desktop
192.168.2.236 host.docker.internal
192.168.2.236 gateway.docker.internal
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section
127.0.0.1 kafka
In the logs I've got messages like
"kafka-rest | [2019-10-21 11:40:57,903] INFO Server started, listening for requests... (io.confluent.kafkarest.KafkaRestMain)"
I don't know what I'm doing wrong
I've tried to follow some instructions on other posts:
Kafka with Docker Problems
Confluent Kafka & docker-compose - error running example
Kafka setup with docker-compose
And many other on Google
You need to configure your Kafka listeners correctly given the networking that Docker involves. This post explains how: https://rmoff.net/2018/08/02/kafka-listeners-explained/
You can find a working Docker Compose that includes host-access here: https://github.com/confluentinc/examples/blob/5.3.1-post/cp-all-in-one/docker-compose.yml

Redis connection refused between Vagrant and Docker

I have a docker like this:
version: '3.5'
services:
RedisServerA:
container_name: RedisServerA
image: redis:3.2.11
command: "redis-server --port 26379"
volumes:
- ../docker/redis/RedisServerA:/data
ports:
- 26379:26379
expose:
- 26379
RedisServerB:
container_name: RedisServerB
image: redis:3.2.11
command: "redis-server --port 6379"
volumes:
- ../docker/redis/RedisServerB:/data
ports:
- 6379:6379
expose:
- 6379
Now I do a vagrant ssh and do
ping RedisServerA
ping RedisServerB
They both work.
Now I try to connect to the redis server:
redis-cli -h RedisServerB
Works fine
Then I try to connect to the other
redis-cli -h RedisServerA -p 26739
It says:
Could not connect to Redis at RedisServerA:26739: Connection refused
Could not connect to Redis at RedisServerA:26739: Connection refused
Twice.
What am I missing here?
Typically in this setup you'd let each container run on its "natural" port. For connections from outside Docker you need the ports: mapping, and you'd access a container via its published port on the host's IP address. For connections between Docker containers (assuming they're on the same network, and if you used bare docker run, you manually created that network), you use the container name and the container's internal port number.
We can clean up the docker-compose.yml file by removing some unnecessary lines (container_name: and expose: don't really have a practical effect) and letting the image run its default command: on the default port, and only remapping with ports:. We'd get:
version: '3.5'
services:
RedisServerA:
image: redis:3.2.11
volumes:
- ../docker/redis/RedisServerA:/data
ports:
- 26379:6379
RedisServerB:
image: redis:3.2.11
volumes:
- ../docker/redis/RedisServerB:/data
ports:
- 6379:6379
Between containers, you'd use the default port
redis-cli -h RedisServerA
redis-cli -h RedisServerB
From outside Docker you'd use the server's host name and the published ports
redis-cli -h server.example.com -p 23679
redis-cli -h server.example.com

Resources