I'm trying to run Kafka locally on Windows 10 Pro and Docker Desktop (not toolbox). Everything seems to work perfectly, but I can't reach Kafka with my application and neither use kafka rest (http://localhost:8082/topics | http://127.0.0.1:8082/topics | http://192.168.1.103:8082/topics - this last one is my docker ip in hosts)
My docker-compose file is:
version: '2'
services:
# https://hub.docker.com/r/confluentinc/cp-zookeeper/tags
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
hostname: zookeeper
network_mode: host
ports:
- "2181:2181"
- "32181:32181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
# https://hub.docker.com/r/confluentinc/cp-kafka/tags
kafka:
image: confluentinc/cp-kafka:5.3.1
container_name: kafka
hostname: kafka
network_mode: host
ports:
- "9092:9092"
- "29092:29092"
restart: always
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_ZOOKEEPER_CONNECT: localhost:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_BROKER_ID: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
depends_on:
- zookeeper
schema-registry:
image: confluentinc/cp-schema-registry:5.3.1
hostname: schema-registry
container_name: schema-registry
network_mode: host
depends_on:
- zookeeper
- kafka
ports:
- "8081:8081"
environment:
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: localhost:2181
SCHEMA_REGISTRY_HOST_NAME: localhost
SCHEMA_REGISTRY_LISTENERS: http://localhost:8081
kafka-rest:
image: confluentinc/cp-kafka-rest:5.3.1
hostname: kafka-rest
container_name: kafka-rest
network_mode: host
depends_on:
- zookeeper
- kafka
ports:
- "8082:8082"
environment:
KAFKA_REST_HOST_NAME: localhost
KAFKA_REST_ZOOKEEPER_CONNECT: localhost:2181
KAFKA_REST_LISTENERS: http://localhost:8082
KAFKA_REST_SCHEMA_REGISTRY_URL: http://localhost:8081
And my hosts file is:
# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host
# localhost name resolution is handled within DNS itself.
# 127.0.0.1 localhost
# ::1 localhost
192.168.1.103 host.docker.internal
192.168.1.103 gateway.docker.internal
# Added by Docker Desktop
192.168.2.236 host.docker.internal
192.168.2.236 gateway.docker.internal
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section
127.0.0.1 kafka
In the logs I've got messages like
"kafka-rest | [2019-10-21 11:40:57,903] INFO Server started, listening for requests... (io.confluent.kafkarest.KafkaRestMain)"
I don't know what I'm doing wrong
I've tried to follow some instructions on other posts:
Kafka with Docker Problems
Confluent Kafka & docker-compose - error running example
Kafka setup with docker-compose
And many other on Google
You need to configure your Kafka listeners correctly given the networking that Docker involves. This post explains how: https://rmoff.net/2018/08/02/kafka-listeners-explained/
You can find a working Docker Compose that includes host-access here: https://github.com/confluentinc/examples/blob/5.3.1-post/cp-all-in-one/docker-compose.yml
Related
broker/docker-compose.yml
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:6.0.1
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
broker:
image: confluentinc/cp-enterprise-kafka:6.0.1
hostname: broker
container_name: broker
depends_on:
- zookeeper
ports:
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
ksql/docker-compose.yml
---
version: '2'
services:
ksqldb-server:
image: confluentinc/ksqldb-server:0.14.0
volumes:
- ./my_quiries/select.sql:/opt/my_quiries/select.sql
hostname: ksqldb-server
container_name: ksqldb-server
ports:
- "8088:8088"
environment:
KSQL_LISTENERS: http://0.0.0.0:8088
KSQL_BOOTSTRAP_SERVERS: localhost:29092
KSQL_KSQL_SERVICE_ID: demo_app
KSQL_KSQL_QUERIES_FILE: /opt/my_quiries/select.sql
KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
I want to run Ksqldb (StandAlone) in distributed fashion. so I tried to run broker and ksqldb from individual compose file from different linux machine one is wsl2 and another is linux vm. but I am stuck with this error to run ksqldb compose file
[2021-01-06 03:17:51,522] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:780)
when my broker is running successfully on port localhost:29092.
Using localhost in ksql is not accessing the host (it is just the localhost of the container itself). You need to provide the docker host IP (when on linux) or you can use host.docker.internal (when on windows).
Another option is to just use docker networking but using two independent docker-compose files will produce two networks (broker_default And ksql_default if I‘m not wrong). So ksql isn‘t able to connect to your broker as it is isolated in it‘s own network. You can solve this with f.e. following:
Create some shared network like docker network create shared_network
In your docker-compose files add this network to ksqldb and kafka to be able to communicate
ksql:
version: '2'
services:
ksqldb-server:
...
networks:
- shared_network
networks:
shared_network:
external: true
broker:
services:
zookeeper:
...
networks:
- internal
broker:
...
networks:
- internal
- shared_network
networks:
internal:
shared_network:
external: true
I want to run Ksqldb (StandAlone) in distributed fashion.
That's not a valid reason for using individual Compose files. If you were to scale a service, you'd either use multiple copies of different service blocks in one file, or Swarm/Kubernetes to scale the running replica count of the service.
As the other answer indicates, KSQL_BOOTSTRAP_SERVERS: localhost:29092 should rather be KSQL_BOOTSTRAP_SERVERS: broker:9092 where the services are on the same Docker network (which they are in the example Compose file I suspect you copied from)
I have created a docker compose file where my application wants to use kafka.
docker-compose.yaml is:
version: '3.7'
services:
api:
depends_on:
- kafka
restart: on-failure
build:
context: .
dockerfile: Dockerfile
ports:
- 8080:8080
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.7
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "mytopic:1:1"
192.168.1.7 is my ip that i got from ifconfig.
In my service i am giving broker as 192.168.1.7:9092.
When i do docker ps and exec to my kafka container. I am not able to access the 192.168.1.0
What am i doing wrong here though the strange thing is in my application logs i see that the topic is created.
When i try to create the topic:
You don't need IP addresses other than 127.0.0.1
192.168.1.7 seems like your host IP, not the docker IP, and yet you are not using network_mode: host, and so the network is not allowing you to connect to the broker.
I recommend finding existing, functional Docker Compose files such as ones in this answer
As posted above by #oneCricketeer you don't have to hardcode any of your host ip addresses.
You can connect to broker using "broker" name inside your api itself. And same can be set to advertise host name as well.
My docker-compose file looks like this:
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
expose:
- "2181"
kafka:
build: .
depends_on:
- zookeeper
expose:
- "8778"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:8778
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_OPTS: '-javaagent:/usr/jolokia/agents/jolokia-jvm.jar'
telegraf:
image: telegraf:latest
links:
- "kafka"
- "zookeeper"
environment:
JOLOKIA_AGENT_URL: http://kafka:8778/jolokia/
ZOOKEEPER_CONNECTION_STRING: http://zookeeper:2181
volumes:
- ./telegraf.conf:/etc/telegraf/telegraf.conf
Example: I can ping kafka successfully from telegraf. I can successfully hit the endpoint I want from the kafka container when I'm execed into that container (curl from localhost when inside it). I cannot, however, reach the endpoint /jolokia/read exposed in the kafka container at port 8778 from the telegraf container.
What am I missing?
I suggest you remove the links section. This has been deprecated by Compose for years.
Then, Compose starts its own network bridge layer, so if you exec into telegraf or zookeeper containers, ping kafka should work, therefore DNS would be working and so curl should as well...
Note: adding Jolokia to Zookeeper should also be done
I'll also point out that the Confluent Helm Charts already provide Prometheus and Jolokia integration
This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed last month.
I want to expost 9093 to outside of the docker container. When I set the kafka-0 ports exposed to 9093 and the KAFKA_ADVERTISED_LISTENERS as follow, i am unable to connect to localhost:9093 as shown in the following docker-compose file.
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.2.1
container_name: kafka-0
hostname: kafka-0
ports:
- "9093:9092"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=wise-nlp-zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
However, when i change to
ports:
- "9092:9092"
and
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9092
I am able to connect to kafka broker localhost:9092.
How can i change external port to 9093 for applications to connect? I want to set up multiple brokers.
Why's it not working currently?
Advertised listener(s) (as defined in KAFKA_ADVERTISED_LISTENERS) are the host and port that the broker returns to the client in its initial connection for it to use in subsequent connections.
If you want external clients to use 9093 then KAFKA_ADVERTISED_LISTENERS=…PLAINTEXT_HOST://localhost:9093 is correct. However, you've not configured your KAFKA_LISTENERS, which if you check the broker log when it starts up will default to the value set by KAFKA_ADVERTISED_LISTENERS:
kafka-0 | listeners = PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093
So in this state, the broker is listening on port 9093, but with this Docker Compose instruction you've redirected external connections into the container on 9093 to 9092 within the container:
ports:
- "9093:9092"
➜ docker ps
CONTAINER ID IMAGE … PORTS NAMES
8b934ef4145c confluentinc/cp-kafka:5.4.1 … 0.0.0.0:9093->9092/tcp kafka-0
So your external connections will go to port 9092 in the container—and the broker is not listening on this port. You can verify this with nc:
-- Port 9093 is open on the host machine
➜ nc -vz localhost 9093
Connection to localhost port 9093 [tcp/*] succeeded!
-- Port 9092 is _not_ open on the Kafka container
➜ docker exec -it kafka-0 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused
❌ You'll see that a client connection fails
➜ kafkacat -b localhost:9093 -L
% ERROR: Failed to acquire metadata: Local: Broker transport failure
How can you fix it?
You can either:
Change the listener to be on the port that you target with the Docker port redirect. This will work but personally I think is more confusing.
Change the Docker port redirect to target the port on which the listener is on. This is the option I would use as it is clearer (e.g. port 9093 is used throughout, rather than mixing 9092 and 9093 together)
Option 1: Change the listener to be on the port that you target with the Docker port redirect
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.4.1
container_name: kafka-0
ports:
- "9093:9092"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9092
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.4.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
✅Test:
➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
1 brokers:
broker 1 at localhost:9093 (controller)
Option 2: Change the Docker port redirect to target the port on which the listener is on
version: '3'
services:
kafka-0:
image: confluentinc/cp-kafka:5.4.1
container_name: kafka-0
ports:
- "9093:9093"
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
# If you don't specify KAFKA_LISTENERS it will default to the ports used in
# KAFKA_ADVERTISED_LISTENERS, but IMO it's better to be explicit about these settings
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.4.1
container_name: zookeeper
ports:
- "2182:2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
✅Test
➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
1 brokers:
broker 1 at localhost:9093 (controller)
Connecting to Kafka from within the Docker network
The examples above are about connecting to Kafka from the Docker host. If you want to connect to it from within the Docker network (e.g. another container) you need to use kafka-0:29094 as the broker host and IP. If you try to use localhost:9093 then the client container will resolve localhost to its own container, and thus fail.
Multiple brokers
See here for an example Docker Compose with multiple Kafka brokers.
References
https://rmoff.net/2018/08/02/kafka-listeners-explained/
I am creating two docker-compose files (mainly because I don't want to have to keep restarting my infrastructure while developing my application.) that need to reside on the same docker network so they can use alias names to connect.
The files look similar to the following:
APP:
version: '3.5'
networks:
default:
name: kafka_network
driver: bridge
services:
client:
build:
context: .
dockerfile: ./Dockerfile
working_dir: /app/
command: ./client
environment:
BADDR: kafka:9092
CGROUP: test_group
TOPICS: my-topic
INFRASTRUCTURE:
version: '3.5'
networks:
default:
name: kafka_network
driver: bridge
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
My issue is that the client doesn't resolve kafka:9092 correctly... it always resolves to 127.0.0.1:9092.
ERROR:
Broker: kafka:9092
Consumer_Group: my_group
Topics: [my-topic]
Created Consumer rdkafka#consumer-1
% Error: GroupCoordinator: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
When run locally, it appears to run fine, so I am really confused as to what the issue might be. If anyone knows anything about this I would be very grateful!
LOCAL:
[procyclinsur#P-428 client]$ ./client
Broker: localhost:9092
Consumer_Group: my-group
Topics: [my-topic]
Created Consumer rdkafka#consumer-1
% AssignedPartitions: [my-topic[0]#unset]
% Message on my-topic[0]#0:
hello mate
That's problem related to your Kafka's config - not to docker at all.
Look on:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
It means that you setup 2 listeners for your Kafka which your clients will receive in Kafka's protocol when connecting.
So when you connect on port 9092 your client's will try to get Kafka at "localhost "and when you connect at port 29092 your clients will try to get Kafka at "kafka" DNS name.
It's working locally for you because your Kafka container is exposed on localhost:9092 via docker ports section.
Here is article which is well describing that topic: https://rmoff.net/2018/08/02/kafka-listeners-explained/