Unable to list kafka topics in openwhisk setup - docker

Setup details:
I am setting up openwhisk on my local ubuntu(16.04) vm. in this setup kafka is running in one docker and zookeeper in another docker.
I connect to the the kafka docker using cmd
sudo docker exec -it <container id> sh
once connected i execute the following command to get the list of topics
bin/kafka-topics.sh --list --zookeeper localhost:2181
which gives me an exception
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is:
java.net.BindException: Address already in use
i am unable to understand why is it trying to use 7203 port?
docker ps output
83eba3961247 ches/kafka:0.10.0.1 "/start.sh"
11 days ago Up 23 hours 7203/tcp, 0.0.0.0:9092->9092/tcp
kafka
947fa689a7ef zookeeper:3.4 "/docker-
entrypoin..." 11 days ago Up 23 hours 2888/tcp,
0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper

The Kafka container OpenWhisk is using sets a JMX_PORT by default. That's the 7203 port you're seeing. To get your script to work you need to unset that environment setting:
unset JMX_PORT; bin/kafka-topics.sh --list --zookeeper localhost:2181
Note though, that localhost is not a valid address for your zookeeper instance, as it refers to the localhost of the current container, which is not Zookeeper. If you exchange localhost with the external IP of your VM or the IP of the zookeeper container (get it via docker inspect zookeeper --format {{.NetworkSettings.Networks.bridge.IPAddress}}) your topics should be listed fine.

Related

Get all Kafka topics from another Docker container?

When running on the host, I can get all the Kafka topics with:
docker exec broker kafka-topics --bootstrap-server broker:29092 --list
I can't run this from within a container because I'd get docker: not found, even if I installed Docker in the container I don't think it'll work anyway. Also, apparently it's hard and insecure to be able to run an arbitrary command in another Docker container. How else can I get all the Kafka topics from within another Docker container? E.g. can I interface with Kafka through http?
I get docker: not found
That seems to imply docker CLI command is not installed, and has nothing to do with Kafka.
docker is not (typically) installed in "another container", so that explains that... You'll need to install Java and download Kafka cli tools to run kafka-topics.sh in any other environment, and then not use docker exec.
Otherwise, your command is "correct", but if you are using Docker Compose, you should do it like this from your host (change port accordingly).
docker-compose exec broker bash -c \
"kafka-topics --list --bootstrap-server localhost:9092"

Docker jenkins not accessible from host browser

I have docker running on centos vm(on vmware) with host machine of windows. I can able to ping centos vm IP address from host(windows) machine ip.
I tried below on centos vm,
docker run -it -p 8080:8080 -p 50000:50000 jenkins/jenkins
2020-11-21 18:41:28.924+0000 [id=27] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2020-11-21 18:41:29.116+0000 [id=20] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running
Below is the output of "docker ps"
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
16a6d00b0933 jenkins/jenkins "/sbin/tini -- /usr/…" 8 minutes ago Up 8 minutes 0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp bhaskara
But when i try to access from host machine(windows) browser like below,
http://centos_vm_ip:8080
Am getting like "Site can't be reached"
Also, On centos vm I have opened port 8080
# firewall-cmd --list-all
public (default, active)
interfaces: eno16777736
sources:
services: dhcpv6-client ssh
ports: 80/tcp 8080/tcp
masquerade: no
forward-ports:
icmp-blocks:
rich rules:
Any idea, how to solve this issue?
TIA
You probably need to configure port forwarding in VMware, so you can access Jenkins from your host.
I hope you have followed the steps mentioned Configuring a Web server on a virtual machine that uses NAT mode networking

failed to connect to my local kafka broker from a local docker container

I have a question regarding connecting to kafka broker from docker.
I have zookeeper and kafka broker running on my local machine.
I have a docker container running on the same local machine with --network=host
I want to send message from inside the docker container to my local kafka broker.
From inside the docker, I can connect to zookeeper and seek the existing topicsby running:
./bin/kafka-topics.sh --zookeeper localhost:2181 --list
But I cannot connect to the kafka broker with either the kafka-console-consumer.sh or kafka-console-producer.sh,
by running : bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
I get:
[2019-10-17 19:12:04,097] WARN [Consumer clientId=consumer-1, groupId=console-consumer-99825] Error connecting to node aictjt:9092 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: aictjt
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:259)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:326)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:439)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:105)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Any help?
Thanks
The solution I found was:
In kafka/config/server.properties
set listeners=PLAINTEXT://localhost:9092, note the default listeners=PLAINTEXT://:9092 worked for some cases but for this specific machine I got I have to put localhost there.
re-run the kafka broker with the updated config.
And the connection works from inside my docker container (with --network=host).
The above solutions didn't work for me.
After spending some time i found the issue was because zookeeper and kafka instance was not running on my local. This ans is tested on mac but i think will work on all platform
If you installed your zookeeper and kafka via homebrew then:
First run zookeeper at your local by running command:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
Then run
kafka-server-start /usr/local/etc/kafka/server.properties
After this the commands for kafka will run without the error
java.net.UnknownHostException: aictjt
For example
kafka-topics --list --bootstrap-server localhost:9092
For those who installed without homebrew:
Open terminal with root directory where you installed kafka
./bin/zookeeper-server-start.sh /usr/local/etc/kafka/zookeeper.properties
./bin/kafka-server-start.sh /usr/local/etc/kafka/server.properties
After this the commands for kafka will run without the error
java.net.UnknownHostException: aictjt
For example
./bin/kafka-topics.sh --list --bootstrap-server localhost:9092
The reason for local docker container throwing "java.net.UnknownHostException" is, it is not able to connect to the kafka broker. This is connectivity issue.
It would work if you follow below steps,
In kafka/config/server.properties, there is "listener" line which would be commented by default. i.e. listeners=PLAINTEXT://localhost:9092. uncomment this line.
Later restart both kafka broker and zookeeper.
Note: In non local environment, if you are setting up stream for first time, and facing similar issue to connect to broker and observing UnknownHostException. Try restarting broker and zookeeper.

How to communicate (network) with Docker at DigitalOcean?

I'm trying to use Docker - Build, Ship, and Run Any App, Anywhere with Simple Cloud Infrastructure for Developers | DigitalOcean, using following container: Docker Hub, np1/docker-tor-clientonly.
Per author's instructions, I was able to run container:
mbp:~ alexus$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bdcdabe8ab1d nagev/tor "/bin/sh -c '/usr/loc" 40 minutes ago Up 40 minutes 127.0.1.1:9150->9150/tcp tor_instance
mbp:~ alexus$
What IP address should I use to set proxy inside of my browser?
If you install docker with toolbox, here is the fix:
# run the container with all IPs, do not limit to 127.0.1.1
$ docker run -d --name tor_instance -p 9150:9150 nagev/tor
# find out the docker IP
$ docker-machine ip default
192.168.99.100
# test the IP and port is available.
telnet 192.168.99.100 9150
Trying 192.168.99.100...
Connected to 192.168.99.100.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
Then you should be fine to set the sockets now.

docker container not available at port 80 like it should

Im using docker registry and the docker frontend is listed as running when I invoke docker ps but it is not available at localhost:80:
e2a54694e434 konradkleine/docker-registry-frontend "/bin/sh -c $START_S 26 seconds ago Up 2 seconds 443/tcp, 0.0.0.0:8080->80/tcp serene_tesla
Do you use boot2docker or docker-machine? If so, you should use the VMs IP address instead of localhost.
for boot2docker usually 192.168.59.103.
for docker-machines IP address type docker-machine ip <yourmachine>.

Resources