I installed zookeeper and kafka docker container on virtual machine. MY settings look like
zookeeper:
image: confluentinc/cp-zookeeper:latest
name:zookeeper
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "2181:2181"
kafka:
image: confluentinc/cp-kafka:latest
hostname: kafka
ports:
- "9092:9092"
- "29092:29092"
depends_on:
- zookeeper
KAFKA_BROKER_ID: "-1"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181");
ALLOW_PLAINTEXT_LISTENER: "yes"
KAFKA_LISTENERS: "PLAINTEXT://9092, PLAINTEXT_HOST://29092"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092, PLAINTEXT_HOST://localhost:29092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
KAFKA_DEFAULT_REPLICATION_FACTOR: "1"
AUTO.LEADER.REBALANCE.ENABLE: "true"
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
Now i am trying to send some message from my local machine to kafka, but i am getting the following error.
Connection to node -1 (//ip-of-virtualmachine:29092) could not be established. Broker may not be available.
From my local machine i configure kafka producer to send messages. the property in my code which configure the bootstrap server is
String bootstrapServers = "virtual-machine-ip:29092";
Properties kafka-properties = new Properties();
kafka-properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
.
.
.
create producer and send some record etc.
All the example i find over internet are regarding the docker and kafka. None of them are related to kafka, docker and virtual machine. Any help would be appreciated.
Probably you didn't expose port of your kafka container to your host, try to expose port 29092 via:
ports:
- "29092:29092"
or run your container in host network docker -d --net=host... if you run it from docker-compose add network_mode: host to your services
to fix your docker config you can take inspiration here:
https://github.com/simplesteph/kafka-stack-docker-compose/blob/master/zk-single-kafka-single.yml
If your code is running outside of the VM, you'll need to make you have an external port forward from the VM to the host, and through the container.
By the way , Docker Machine would create VMs for you, rather than you installing Docker in one on your own
Related
I'm new to Kafka and I'm trying to run a Kafka service on my local machine and use it to transfer some data from one .NET project to another.
I'm using docker-compose.yml file to create two docker containers for zookeeper and Kafka from wurstmeister.
In Kafka definition in environment variables there is KAFKA_ADVERTISED_HOST_NAME which I set to 127.0.0.1.
In docker hub of wurstmeister/kafka says I quote
"modify the KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host IP if you want to run multiple brokers.)".
When I set my topic to have more than 1 replica and more than 1 partition I get this message
"Error while executing topic command : Replication factor: 2 larger than available brokers: 1.".
What is the right IP address to define in KAFKA_ADVERTISED_HOST_NAME which will allow me to get more than 1 broker?
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Note - Running multiple brokers on one machine does not provide true fault tolerance.
use it to transfer some data from one .NET project to another
You only need one broker for this.
First, suggest reading https://github.com/wurstmeister/kafka-docker/wiki/Connectivity
KAFKA_ADVERTISED_HOST_NAME is deprecated. Don't use it.
It has been replaced by KAFKA_ADVERTISED_LISTENERS, which can both contain ${DOCKER_HOST_IP:-127.0.0.1} (or your host IP from ifconfig) since this is the IP your client will use (from the host).
Since brokers are clients of themselves, you also need to advertise the container names. And, if you want to containerize your client, those addresses would also be used by your app.
Beyond Kafka networking configs, KAFKA_BROKER_ID needs to be different between each, and your error is suggesting you need to override all the other replication factor broker configs.
All in all.
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
kafka-1:
image: wurstmeister/kafka
depends_on: [zookeeper]
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_IP:-127.0.0.1}:9092,PLAINTEXT_INTERNAL://kafka-1:29092
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT_INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_DEFAULT_REPLICATION_FACTOR: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
kafka-2:
image: wurstmeister/kafka
ports:
- "9093:9093"
environment:
KAFKA_BROKER_ID: 2 # unique
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_IP:-127.0.0.1}:9093,PLAINTEXT_INTERNAL://kafka-2:29093
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093,PLAINTEXT_INTERNAL://:29093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT_INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_DEFAULT_REPLICATION_FACTOR: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
depends_on: # ensure this joins the other
- zookeeper
- kafka-1
I'm trying to run a Kafka service on my local machine
Set bootstrap.servers="localhost:9092,localhost:9093"
Further reading - Connect to Kafka running in Docker
Example usage
Note that topic creation doesn't seem to automatically work. You should ideally be using your application to actually create / check topic existence.
$ kcat -b localhost:9093 -L
Metadata for all topics (from broker -1: localhost:9093/bootstrap):
2 brokers:
broker 2 at 127.0.0.1:9093
broker 1 at 127.0.0.1:9092 (controller)
0 topics:
producing some data... and consuming it
$ kcat -b localhost:9093 -t test -o earilest
% Auto-selecting Consumer mode (use -P or -C to override)
hello
world
sample
data
of
no
importance
it should be ip address of your machine.
Linux: use ifconfig command and your IP will be inet <IP_ADDRESS>
in windows you will get this by using ipconfig
I want to use kafka with elk stack, where filebeat reads logs from file and send data to kafka which then sends it to logstash. I am running kafka using docker-compose file.
I managed to run it by using port 9092 which is default for kafka, but problem is my organization is already running some other service on port 9092. So I have used 9095 in my docker-compose for host and 9092 inside container.
When I try to use filebeat and logstash with kafka, data is not getting to logstash. I can see that logstash is subscribed to topic I created, but it is not getting any data even when I send message from console-producer. But when I use console-producer and console-consumer it is working fine. But it is not working with logstash and filebeat which are running outside the container.
Here is my docker-compose file for kafka:
version: "3"
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka
container_name: kafka
ports:
- 9095:9092
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9095
KAFKA_ADVERTISED_LISTENERS: INSIDE://172.18.0.3:9092,OUTSIDE://192.168.135.57:9095
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
Here 192.168.135.57 is my host address(public IP of my machine) and 172.18.0.3 is address of docker container running kafka.
and this is my logstash config file:
input {
kafka
{
bootstrap_servers => "192.168.135.57:9095"
topics => "topicname"
}
}
output {
stdout{codec => rubydebug}
elasticsearch {
hosts => "http://localhost:9200"
index => "topic-test"
}
}
This is what I used in filebeat to send data to kafka:
output.kafka:
hosts: ["192.168.135.57:9095"]
topic: "topicname"
I can't figure out what is wrong with this pipeline. I would like to run this docker-compose on port 9095(or any other port except 9092) on host machine. Help would be appreciated.
You're still mapping 9095 on the host to the advertised INSIDE listener
You should use 9095:9095 to map to the OUTSIDE listener if that is what other clients will connect on
I have created a docker compose file where my application wants to use kafka.
docker-compose.yaml is:
version: '3.7'
services:
api:
depends_on:
- kafka
restart: on-failure
build:
context: .
dockerfile: Dockerfile
ports:
- 8080:8080
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.7
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "mytopic:1:1"
192.168.1.7 is my ip that i got from ifconfig.
In my service i am giving broker as 192.168.1.7:9092.
When i do docker ps and exec to my kafka container. I am not able to access the 192.168.1.0
What am i doing wrong here though the strange thing is in my application logs i see that the topic is created.
When i try to create the topic:
You don't need IP addresses other than 127.0.0.1
192.168.1.7 seems like your host IP, not the docker IP, and yet you are not using network_mode: host, and so the network is not allowing you to connect to the broker.
I recommend finding existing, functional Docker Compose files such as ones in this answer
As posted above by #oneCricketeer you don't have to hardcode any of your host ip addresses.
You can connect to broker using "broker" name inside your api itself. And same can be set to advertise host name as well.
I'm trying to start a kafka cluster using docker compose, I'm using the following configurations:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "15092:9092"
hostname: kafka
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:15092
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://:15092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
Both services are up and running, but when I try to produce a message from an external source using broker external-ip:15092 I receive the following error:
dial tcp: lookup kafka: no such host
Can you help me figure out what the configuration is missing?
Thanks
You're getting "no such host", which occurs before the port is even used. You need to run code in another container in the same Docker network for the service name to resolve
Kafka doesn't work like that (a simple port forward), anyway
Both the listeners are still set set at 9092
You'll need to add / change the advertised listener containing externalIP:15092 for it to work, and you can find multiple places where the difference of listeners is documented (including the wiki pages for that container)
So, KAFKA_LISTENERS: INSIDE://:9092 is all you need (or more appropriately, put INSIDE://0.0.0.0:9092)
But you need to edit
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://<your_external_IP>:15092
Here's a configuration that exposes different ports:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
ports:
- "49815:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
broker:
image: wurstmeister/kafka
container_name: broker
ports:
- "49816:9092"
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT
KAFKA_LISTENERS: PLAINTEXT://broker:9092,INTERNAL://broker:29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:49816,INTERNAL://broker:29092
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
Key points:
Your external source needs to connect to the address that is exposed by Docker (localhost:49816) and bound to the broker using the port 9092. So you need to set KAFKA_ADVERTISED_LISTENERS with that value for external connections.
The broker should be listening to 9092 to get the connections coming from the outside through Docker port binding. So you need to set PLAINTEXT://broker:9092 as an external listener in KAFKA_LISTENERS.
You still need to listen for inter container connections for Brokers and Zookeeper to function, so you need to set KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS with internal endpoints INTERNAL://broker:29092 that are visible from inside Docker.
Here's the complete configs link: https://docs.confluent.io/platform/current/installation/configuration/broker-configs.html#brokerconfigs_listeners (it's for Confluence image but it works the same with wurstmeister's)
I'm trying to connect to kafka docker container from another docker container. But it didn't connect.
There are the list of containers using for kafka messaging
Network kafka-docker_default has two containers kafka-docker_zookeeper_1 and kafka-docker_kafka0_1
For running kafka and zookeeper I used docker-compose file:
version: '2'
services:
zookeeper:
image: "confluentinc/cp-zookeeper:latest"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
# This has three listeners you can experiment with.
# BOB for internal traffic on the Docker network
# FRED for traffic from the Docker-host machine (`localhost`)
# ALICE for traffic from outside, reaching the Docker host on the DNS name `never-gonna-give-you-up`
# Use
kafka0:
image: "confluentinc/cp-kafka"
ports:
- '9092:9092'
- '29094:29094'
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://kafka0:9092,LISTENER_ALICE://kafka0:29094
KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://localhost:9092,LISTENER_ALICE://never-gonna-give-you-up:29094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT,LISTENER_ALICE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_BOB
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
Containers dockercompose17138956372294708100_kafkatest.producer_1 and dockercompose17138956372294708100_kafkatest.consumer_1 are for produce and consume messages.
While container dockercompose17138956372294708100_kafkatest.producer_1 is trying to connect to kafka cluster using bootstrap.servers parameter(localhost:9092 or never-gonna-give-you-up:29094) it showed error
%7|1569945335.015|BROKERFAIL|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: never-gonna-give-you-up:29094/bootstrap: failed: err: Local: Host resolution failure: (errno: Bad address)
%3|1569945335.015|FAIL|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: never-gonna-give-you-up:29094/bootstrap: Failed to resolve 'never-gonna-give-you-up:29094': Name or service not known (after 1656ms in state CONNECT)
%3|1569945335.015|ERROR|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: never-gonna-give-you-up:29094/bootstrap: Failed to resolve 'never-gonna-give-you-up:29094': Name or service not known (after 1656ms in state CONNECT)
%7|1569945335.015|STATE|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: never-gonna-give-you-up:29094/bootstrap: Broker changed state CONNECT -> DOWN
%3|1569945335.015|ERROR|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: 1/1 brokers are down
How can I fix it?
Please try this configuration:
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
ports:
- "32181:32181"
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
kafka0:
image: confluentinc/cp-kafka:latest
ports:
- "9090:9090"
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka0:29090,LISTENER_EXTERNAL://localhost:9090
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
With this configuration, if you connect from host to the broker's container, you use the IP localhost:9090. If you connect from other containers with the same network, you use kafka0:29090.
dockercompose17138956372294708100_kafkatest.producer_1 is trying to connect to kafka cluster using bootstrap.servers parameter(localhost:9092 or never-gonna-give-you-up:29094)
From another Docker container, neither of those are correct.
never-gonna-give-you-up does not exist anywhere as a Hostname/DNS record in your Docker configuration. Based on the error, your external DNS server also doesn't know what that is
localhost refers to the Kafka client container, not your host, or the broker container.
According to your compose configuration, you need to connect to kafka0 host/service, which is advertised on port 29092 for the bootstrap.servers
# BOB for internal traffic on the Docker network
KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka0:29092
You should also probably remove LISTENER_ALICE because it isn't providing any useful connection details unless your host's external hostname is actually never-gonna-give-you-up
And you'll want to put your producer and consumer as part of your compose file