Unable to query Kafka in Kubernetes deployment - docker

Question

On a single Ubuntu 14.04 box
I've followed the same configuration as
http://dojoblog.dellemc.com/dojo/deploy-kafka-cluster-kubernetes/
I use Kubernetes version v1.10.2
( I also use apiVersion: apps/v1 in yml files. )
Basically I have setup a kubernetes service for kafka, and a kafka deployment using image wurstmeister/kafka. Zookeeper is working ok. Zookeeper and Kafka services are up.
Kafka deployment is configured as per the blog : KAFKA_ADVERTISED_HOST_NAME = the kafa service cluster IP which is for me 10.106.84.132
deployment config :
....
containers:
- name: kafka
image: wurstmeister/kafka
ports:
- containerPort: 9092
env:
- name: KAFKA_ADVERTISED_PORT
value: "9092"
- name: KAFKA_ADVERTISED_HOST_NAME
value: 10.106.84.132
- name: KAFKA_ZOOKEEPER_CONNECT
value: zoo1:2181
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_CREATE_TOPICS
value: topic1:3:3
Then I test the kafka subscribe and publish from outside the kafka container on my host, but that fails as follow :
root#edmitchell-virtual-machine:~# kafkacat -b 10.106.84.132:9092 -t topic1
% Auto-selecting Consumer mode (use -P or -C to override)
% ERROR: Topic topic1 error: Broker: Leader not available
The best I could do overall was
I delete and recreate a kafka deployment with
name: KAFKA_ADVERTISED_HOST_NAME
value: localhost
I can then subscribe and publish but only from within the kafka container, it doesn't work from outside. If I change the value to anything else than localhost, nothing works.
Any idea ?
It looks as if Kafka is not good to be used with Kubernetes ?
maybe I should not deploy Kafka not using kubernetes..
many thanks
ed
Thank you, I understand better now the nodeport function.
I still have the same issue :
root#fnature-virtual-machine:~/Zookeeper# kafkacat -b 192.168.198.160:32748 -t topic1 % Auto-selecting Consumer mode (use -P or -C to override) % ERROR: Topic topic1 error: Broker: Leader not available
I created the nodeport service as you said.
kafka-nodeport NodePort 10.111.234.104 9092:32748/TCP 27m
kafka-service LoadBalancer 10.106.84.132 9092:30351/TCP 1d
I also delete/create the kafka deployment with following env :
KAFKA_ADVERTISED_PORT: 32748
KAFKA_ADVERTISED_HOST_NAME: 192.168.198.160
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
KAFKA_BROKER_ID: 1
KAFKA_CREATE_TOPICS: topic1:3:3
—
also if I run the following from inside the kafka container, I get similar error
"Leader not available". kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic1 --from-beginning
if I create the kafka deployment with KAFKA_ADVERTISED_HOST_NAME: localhost, then above command works inside the kafka container
and 192.168.198.160 is the ip of default interface ens33 in my Ubuntu VM
I can’t seem to find any logs for kafka

Answer 1

Kafka broker registers an address to zookeeper via KAFKA_ADVERTISED_HOST_NAME. However, this address is a kubernetes cluster ip (10.106.84.132), which is only reachable within Kubernetes cluster. So a client outside the cluster can not reach Kafka broker using this address.
To resolve this problem, you can expose kafka service to a public ip, either through NodePort or LoadBalancer. For example, run kubectl expose svc $YOUR_KAFKA_SERVICE_NAME --name=kafka-nodeport --type=NodePort, then lookup what nodeport is exposed: kubectl get svc kafka-nodeport -o yaml | grep nodePort. In this example, kafka broker will be accessible via this address: $KUBERNETES_NODE_IP:$NODEPORT.

Answer 2

in k8s, deployment kafka.yaml
env:
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_CREATE_TOPICS
value: "test:1:1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INSIDE://:9092,OUTSIDE://kafka-com:30322"
- name: KAFKA_LISTENERS
value: "INSIDE://:9092,OUTSIDE://:30322"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INSIDE"
kafka service,the external service invocation address, or traefik proxy address
---
kind: Service
apiVersion: v1
metadata:
name: kafka-com
namespace: dev
labels:
k8s-app: kafka
spec:
selector:
k8s-app: kafka
ports:
- port: 9092
name: innerport
targetPort: 9092
protocol: TCP
- port: 30322
name: outport
targetPort: 30322
protocol: TCP
nodePort: 30322
type: NodePort
Ensure that Kafka external port and nodePort are consistent,Other services call kafka-com:30322, my blog write this config_kafka_in_kubernetes, hope to help U !