https://docs.confluent.io/4.0.0/installation/docker/docs/quickstart.html
I followed the steps given in this document and tried to create a topic, but getting an exception as shown below.
I already checked whether both kafka and zookeeper are up and they are.
I also tried following:
docker-compose exec kafka kafka-topics --list --zookeeper localhost:2181
I get the same error.
This is fixed now by using the correct zookeeper in the command to list the kafka-topics.
docker-compose exec kafka kafka-topics --list --zookeeper zookeeper:2181
I got the zookeeper connect information from the docker-compose.yml file.
Related
How connect two kafka clusters on different VM's together ?
On my Windows maschine with the IP-Address: 192.168.2.22
i downloaded docker-compose.yml from https://github.com/conduktor/kafka-stack-docker-compose/blob/master/zk-single-kafka-single.yml and started it in a cmd-console with docker-compose up -d
Create Topic:
docker exec kafka1 kafka-topics --bootstrap-server localhost:9092 --create --topic my-Windows-Topic-1
Check access to Topic: in different ways (but in the same cmd-console)
docker exec kafka1 kafka-topics --bootstrap-server localhost:9092 --list
docker exec kafka1 kafka-topics --bootstrap-server localhost:19092 --list
docker exec kafka1 kafka-topics --bootstrap-server localhost:29092 --list
docker exec kafka1 kafka-topics --bootstrap-server 192.168.2.22:9092 --list
docker exec kafka1 kafka-topics --bootstrap-server 192.168.2.22:29092 --list
docker run --rm confluentinc/cp-kafka bash -c "kafka-topics --bootstrap-server 192.168.2.22:9092 --list"
docker run --rm confluentinc/cp-kafka bash -c "kafka-topics --bootstrap-server 192.168.2.22:29092 --list"
all the above commands show the same output so far
my-Windows-Topic-1
Now i change to a Linux-Virtual-Maschine with IP: 192.168.94.130
and again
Check access to the Windows Topic:
docker run --rm confluentinc/cp-kafka bash -c "kafka-topics --bootstrap-server 192.168.2.22:9092 --list"
my-Windows-Topic-1
So far everything is fine. I have access to the topic from internal and external!
(Still from the Linux console)
Now i'm creating a second independent cluster in the same way as above:
downloaded https://github.com/conduktor/kafka-stack-docker-compose/blob/master/zk-single-kafka-single.yml and started with docker-compose up -d
Create Linux Topic:
docker exec kafka1 kafka-topics --bootstrap-server localhost:9092 --create --topic my-Linux-Topic
Check Topic:
$ docker exec kafka1 kafka-topics --bootstrap-server localhost:9092 --list
my-Linux-Topic
How to configure both clusters to display both outputs?
my-Windows-Topic-1
and
my-Linux-Topic
at the same time ?
I'm having a hard-time following the question, but if you want multiple brokers, then the repo you've linked to already has an example of that (in one Kafka cluster)
If you have multiple compose files, then they will not share a Docker network, by default, and so will be unable to reach each other.
If you want to connect them, then use a bridge network
configure both clusters to display both output ... at the same time?
You can't; you can only list topics of one cluster at a time.
You could use Conduktor Platform to run a UI where you can link both clusters to it, then display topics there, without using any terminal commands.
When running on the host, I can get all the Kafka topics with:
docker exec broker kafka-topics --bootstrap-server broker:29092 --list
I can't run this from within a container because I'd get docker: not found, even if I installed Docker in the container I don't think it'll work anyway. Also, apparently it's hard and insecure to be able to run an arbitrary command in another Docker container. How else can I get all the Kafka topics from within another Docker container? E.g. can I interface with Kafka through http?
I get docker: not found
That seems to imply docker CLI command is not installed, and has nothing to do with Kafka.
docker is not (typically) installed in "another container", so that explains that... You'll need to install Java and download Kafka cli tools to run kafka-topics.sh in any other environment, and then not use docker exec.
Otherwise, your command is "correct", but if you are using Docker Compose, you should do it like this from your host (change port accordingly).
docker-compose exec broker bash -c \
"kafka-topics --list --bootstrap-server localhost:9092"
I want to run "standalone" Docker container with a configured Kafka server.
I found out on Kafka website (https://kafka.apache.org/quickstart) how to run Kafka topic:)
But when I'm doing everything as instructed I need to run three terminals:
One for run ZooKeeper server:
./bin/zookeeper-server-start.sh config/zookeeper.properties
Second for start kafka server:
./bin/kafka-server-start.sh config/server.properties
Third for create a topic:
./bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
The question is :
How do I run three independent terminals inside Docker while building the docker image?
Because 1 want to only use commands:
docker build . -t kafka
and then
docker start kafka
and have an up and running Kafka server with a created topic.
I done something but I stack on trying to create this terminals.
Here's the project:
https://github.com/mpawel1993/Kafka-Docker
I want to run "standalone" Docker container with a configured Kafka server
If you want a configured Kafka server, any of the existing Docker images work fine. landoop/fast-data-dev includes both Kafka, Zookeeper, Kafka Connect, and a Schema Registry, if by "standalone" you mean have all necessary components in one image
How do I run three independent terminals inside Docker while building the docker image
You wouldn't. Each RUN command its a single terminal
You also should not start Kafka and Zookeeper in one container for fault tolerance and scalability reasons
You also don't need to create Kafka topics while building the container, only once the container is built and server is running can you create topics
I have a question regarding connecting to kafka broker from docker.
I have zookeeper and kafka broker running on my local machine.
I have a docker container running on the same local machine with --network=host
I want to send message from inside the docker container to my local kafka broker.
From inside the docker, I can connect to zookeeper and seek the existing topicsby running:
./bin/kafka-topics.sh --zookeeper localhost:2181 --list
But I cannot connect to the kafka broker with either the kafka-console-consumer.sh or kafka-console-producer.sh,
by running : bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
I get:
[2019-10-17 19:12:04,097] WARN [Consumer clientId=consumer-1, groupId=console-consumer-99825] Error connecting to node aictjt:9092 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: aictjt
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:259)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:326)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:439)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:105)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Any help?
Thanks
The solution I found was:
In kafka/config/server.properties
set listeners=PLAINTEXT://localhost:9092, note the default listeners=PLAINTEXT://:9092 worked for some cases but for this specific machine I got I have to put localhost there.
re-run the kafka broker with the updated config.
And the connection works from inside my docker container (with --network=host).
The above solutions didn't work for me.
After spending some time i found the issue was because zookeeper and kafka instance was not running on my local. This ans is tested on mac but i think will work on all platform
If you installed your zookeeper and kafka via homebrew then:
First run zookeeper at your local by running command:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
Then run
kafka-server-start /usr/local/etc/kafka/server.properties
After this the commands for kafka will run without the error
java.net.UnknownHostException: aictjt
For example
kafka-topics --list --bootstrap-server localhost:9092
For those who installed without homebrew:
Open terminal with root directory where you installed kafka
./bin/zookeeper-server-start.sh /usr/local/etc/kafka/zookeeper.properties
./bin/kafka-server-start.sh /usr/local/etc/kafka/server.properties
After this the commands for kafka will run without the error
java.net.UnknownHostException: aictjt
For example
./bin/kafka-topics.sh --list --bootstrap-server localhost:9092
The reason for local docker container throwing "java.net.UnknownHostException" is, it is not able to connect to the kafka broker. This is connectivity issue.
It would work if you follow below steps,
In kafka/config/server.properties, there is "listener" line which would be commented by default. i.e. listeners=PLAINTEXT://localhost:9092. uncomment this line.
Later restart both kafka broker and zookeeper.
Note: In non local environment, if you are setting up stream for first time, and facing similar issue to connect to broker and observing UnknownHostException. Try restarting broker and zookeeper.
Setup details:
I am setting up openwhisk on my local ubuntu(16.04) vm. in this setup kafka is running in one docker and zookeeper in another docker.
I connect to the the kafka docker using cmd
sudo docker exec -it <container id> sh
once connected i execute the following command to get the list of topics
bin/kafka-topics.sh --list --zookeeper localhost:2181
which gives me an exception
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is:
java.net.BindException: Address already in use
i am unable to understand why is it trying to use 7203 port?
docker ps output
83eba3961247 ches/kafka:0.10.0.1 "/start.sh"
11 days ago Up 23 hours 7203/tcp, 0.0.0.0:9092->9092/tcp
kafka
947fa689a7ef zookeeper:3.4 "/docker-
entrypoin..." 11 days ago Up 23 hours 2888/tcp,
0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper
The Kafka container OpenWhisk is using sets a JMX_PORT by default. That's the 7203 port you're seeing. To get your script to work you need to unset that environment setting:
unset JMX_PORT; bin/kafka-topics.sh --list --zookeeper localhost:2181
Note though, that localhost is not a valid address for your zookeeper instance, as it refers to the localhost of the current container, which is not Zookeeper. If you exchange localhost with the external IP of your VM or the IP of the zookeeper container (get it via docker inspect zookeeper --format {{.NetworkSettings.Networks.bridge.IPAddress}}) your topics should be listed fine.