Kubernetes (docker) access to Kafka in docker compose - docker

Question

docker-compose.yml
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:6.0.0
hostname: zookeeper
container_name: zookeeper
restart: unless-stopped
ports:
- "2181:2181"
volumes:
- ./data/zoo/zk-data:/var/lib/zookeeper/data
- ./data/zoo/zk-txn-logs:/var/lib/zookeeper/log
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
broker:
image: confluentinc/cp-server:6.0.0
hostname: broker
container_name: broker
restart: unless-stopped
depends_on:
- zookeeper
volumes:
- ./data/kafka:/var/lib/kafka/data
ports:
- "9092:9092"
- "9101:9101"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9101
KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
CONFLUENT_METRICS_ENABLE: 'true'
CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
schema-registry:
image: confluentinc/cp-schema-registry:6.0.0
hostname: schema-registry
container_name: schema-registry
restart: unless-stopped
depends_on:
- broker
ports:
- "8081:8081"
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
dev.yml
apiVersion: v1
kind: Service
metadata:
name: auth-server
labels:
app: auth-server
spec:
type: LoadBalancer
ports:
- port: 8882
protocol: TCP
targetPort: 8882
nodePort: 30080
selector:
app: auth-server
status:
loadBalancer:
ingress:
- hostname: localhost
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-server
spec:
selector:
matchLabels:
app: auth-server
replicas: 1
template:
metadata:
labels:
app: auth-server
spec:
containers:
- name: auth-server
image: auth-server:latest
imagePullPolicy: Never
ports:
- containerPort: 8080
volumeMounts:
- name: auth-server
mountPath: "/config"
volumes:
- name: auth-server
configMap:
name: auth-server
I need connect from my app on kubernetes cluser (docker) connect to docker-compose kafka.
I tried change kafka url in properties: http://localhost:9092, http://broker:9092 and
THIS tutorial but won't work too.
In docker ps I see: kafka and auth-server from kubernetes.
Can someone explain me. How can I do it ?

Answer 1

Since I don't know your flavor of local K8s, I can only speculate. But the problem here could be that Kubernetes and docker compose run their containers in different virtual networks.
What you have to do to create connectivity is:
Expose the Kafka port in your compose file with port mappings to the host:
ports:
- 9092:9092
Connect to it from Kubernetes via the IP address of your docker0 interface (typically 172.17.0.1).