Quarkus can't connect to kafka from inside docker [duplicate] - docker

Question

This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed 1 year ago.
I've created a quarkus service that reads from a bunch of Kstreams, joins them and then post the join result back into a kafka topic.
During development, I was running kafka and zookeeper from inside a docker-compose and then running my quarkus service on dev mode with:
mvn quarkus:dev
At this point, everything was working fine. I'm able to connect to the broker without problem and read/write the Kstreams.
Then I tried to create a docker container that runs this quarkus service, but when the service runs inside the container, it doesn't reach the broker.
I tried several different configs inside my docker-compose, but none worked. It just can't connect to the broker.
Here is my Dockerfile:
####
# This Dockerfile is used in order to build a container that runs the Quarkus application in JVM mode
#
# Before building the container image run:
#
# mvn package
#
# Then, build the image with:
#
# docker build -f src/main/docker/Dockerfile.jvm -t connector .
#
# Then run the container using:
#
# docker run -i --rm -p 8080:8080 connector
#
# If you want to include the debug port into your docker image
# you will have to expose the debug port (default 5005) like this : EXPOSE 8080 5050
#
# Then run the container using :
#
# docker run -i --rm -p 8080:8080 -p 5005:5005 -e JAVA_ENABLE_DEBUG="true" connector
#
###
FROM docker.internal/library/quarkus-base:latest
ARG RUN_JAVA_VERSION=1.3.8
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en'
USER root
RUN apk update && apk add libstdc++
# Configure the JAVA_OPTIONS, you can add -XshowSettings:vm to also display the heap size.
ENV JAVA_OPTIONS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
#ENV QUARKUS_LAUNCH_DEVMODE=true \
# JAVA_ENABLE_DEBUG=true
# -Dquarkus.package.type=mutable-jar
# We make four distinct layers so if there are application changes the library layers can be re-used
COPY --chown=1001 target/quarkus-app/lib/ ${APP_HOME}/lib/
COPY --chown=1001 target/quarkus-app/*-run.jar ${APP_HOME}/app.jar
COPY --chown=1001 target/quarkus-app/app/ ${APP_HOME}/app/
COPY --chown=1001 target/quarkus-app/quarkus/ ${APP_HOME}/quarkus/
EXPOSE 8080
USER 1001
#ENTRYPOINT [ "/deployments/run-java.sh" ]
And here is my docker-compose:
version: '2'
services:
zookeeper:
container_name: zookeeper
image: confluentinc/cp-zookeeper
ports:
- "2181:2181"
- "2888:2888"
- "3888:3888"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
- ZOOKEEPER_TICK_TIME=2000
networks:
- kafkastreams-network
kafka:
container_name: kafka
image: confluentinc/cp-kafka
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
- KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1
- KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
- KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=100
networks:
- kafkastreams-network
connect:
container_name: connect
image: debezium/connect
ports:
- "8083:8083"
depends_on:
- kafka
environment:
- BOOTSTRAP_SERVERS=kafka:29092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_connect_statuses
networks:
- kafkastreams-network
schema-registry:
image: confluentinc/cp-schema-registry:5.5.0
container_name: schema-registry
ports:
- "8081:8081"
depends_on:
- zookeeper
- kafka
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
networks:
- kafkastreams-network
kafdrop:
image: obsidiandynamics/kafdrop
container_name: kafdrop
restart: "no"
ports:
- "9001:9000"
environment:
KAFKA_BROKERCONNECT: kafka:29092
JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
depends_on:
- kafka
- schema-registry
networks:
- kafkastreams-network
connector:
image: connector
depends_on:
- zookeeper
- kafka
- connect
environment:
QUARKUS_KAFKA_STREAMS_BOOTSTRAP_SERVERS: kafka:9092
networks:
- kafkastreams-network
networks:
kafkastreams-network:
name: ks
The error I'm getting is:
2021-08-05 11:52:35,433 WARN [org.apa.kaf.cli.NetworkClient] (kafka-admin-client-thread | connector-18d10d7d-b619-4715-a219-2557d70e0479-admin) [AdminClient clientId=connector-18d10d7d-b619-4715-a219-2557d70e0479-admin] Connection to node -1 (kafka/172.21.0.3:9092) could not be established. Broker may not be available.
Am I missing any config on either the Dockerfile or the docker compose?

Answer 1

I figured out that there were 2 problems:
In my docker-compose, I had to change the property KAFKA_ADVERTISED_LISTENERS to PLAINTEXT://kafka:29092,PLAINTEXT_HOST://kafka:9092
In my quarkus application.properties, I had 2 properties pointing to the wrong place:
quarkus.kafka-streams.bootstrap-servers=localhost:9092
quarkus.kafka-streams.application-server=localhost:9999