Kafka Chart on Kubernetes: Simple test string produce + consume - docker

Question

I install the incubator Kafka chart. Version kafka-0.8.5 as of this writing.
helm install --name kafka \
--set replicas=1 \
--set persistence.enabled=false \
--set zookeeper.replicaCount=1 \
incubator/kafka
To try this out I run a separate pod as just a bash shell with Kafka cli tools. I'm using the exact same Docker image confluentinc/cp-kafka:4.1.1-2 that the kafka-0 pod is using so that there is a perfect version match between the client and server:
kubectl run shell --rm -i --tty --image confluentinc/cp-kafka:4.1.1-2 -- /bin/bash
Listing topics, publishing messages, getting topic offsets all works perfectly, as shown below. However, when I try to run kafka-console-consumer and see the test record in the topic, it hangs indefinitely. Why?
root#shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-topics --zookeeper kafka-zookeeper:2181 --list
__confluent.support.metrics
root#shell-5c6ddf5d99-tbsvm:/# echo "abcxyz" | /usr/bin/kafka-console-producer --broker-list kafka:9092 --topic test-topic
>[2018-08-07 16:43:26,110] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
root#shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-topics --zookeeper kafka-zookeeper:2181 --list
__confluent.support.metrics
test-topic
root#shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-run-class kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic test-topic --time -1
test-topic:0:1
root#shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-console-consumer --bootstrap-server kafka:9092 --from-beginning --topic test-topic
<hangs indefinitely>
FYI, this is a local minikube development cluster with the latest minikube with Kubernetes 1.10.x server-side and 1.10.x kubectl client tools. This is a clean, new minikube, with nothing else running besides kafka, kafka-zookeeper, and my shell pod.
Also, writing a small Java client test app to consume gets a similar result of polling indefinitely with no messages. When my Java client subscribes to test-topic, it never gets notification callbacks of being assigned to the one topic partition.

Answer 1

This cost me hours myself, there's a bug in minikube which prevents Kafka from working.
I'm not familiar with the Helm deployment, but you have to make sure of two things. First, the Kafka advertised host has to be the same as your Kubernetes service IP (or DNS name in kube dns), and second, you have to put minikube's network interface in promiscuous mode:
minikube ssh sudo ip link set docker0 promisc on
If you don't do this workaround, kafka can't contant itself via the Kubernetes service, and its leader election fails. I've found it to be very fragile inside a container deployment environment.