Docker (Compose) client connects to Kafka too early - docker

Question

I am trying to run Kafka with Docker and Docker Compose. This is the docker-compose.yml:
version: "2"
services:
zookeeper:
image: "wurstmeister/zookeeper"
ports:
- "2181:2181"
kafka:
build:
context: "./services/kafka"
dockerfile: "Dockerfile"
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: "0.0.0.0"
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
users:
build:
context: "./services/users"
dockerfile: "Dockerfile"
ports:
- "4001:4001"
environment:
NODE_ENV: "develop"
ZOOKEEPER_HOST: "zookeeper"
ZOOKEEPER_PORT: "2181"
volumes:
- "./services/users:/service"
The users service only tries to connect (using kafka-node in Node.js) and listens on a topic and publishes one message to it every time it is ran.
The problem is that I keep getting Connection Refused errors. I am using Dockerize to wait for the kafka port to be available in the Dockerfile with the line CMD dockerize -wait tcp://kafka:9092 node /service/index.js.
It waits for the port to be available before starting the users container and this system works, but it is not at the right time. It seems that Kafka is opening the 9092 port before it has elected a leader.
When I run Kafka first and let it start completely and then run my app, it runs smoothly.
How do I wait for the correct moment before starting my service?

Answer 1

Try the docker-compose version 2.1 or 3, as it includes an healthcheck directive.
See "Docker Compose wait for container X before starting Y" as an example.
You can:
depends_on:
kafka:
condition: service_healthy
And in kafka add:
healthcheck:
test: ["CMD", ...]
interval: 30s
timeout: 10s
retries: 5
with a curl command for instance which would test if kafka has elected a leader.

Answer 2

A full example; this is what I use in docker compose.
tldr; use a kafka healthcheck
["CMD", "kafka-topics.sh", "--list", "--zookeeper", "zookeeper:2181"]
integration test app depends on kafka
app depends on kafka
kafka depends on zookeeper
Since the integration test and the app are starting at the same time, I think this helps with total execution time.
Also, both are starting after kafka's healthcheck is passing.
version: '2.1'
services:
my-integration-tests:
image: golang:1.16
volumes:
- myapp:/app
command: go test -tags=integration -mod=vendor -cover -v --ginkgo.v --ginkgo.progress --ginkgo.failFast
depends_on:
kafka:
condition: service_healthy
my-app:
image: local/my-app
build:
context: .
depends_on:
kafka:
condition: service_healthy
zookeeper:
image: wurstmeister/zookeeper:3.4.6
expose:
- "2181"
tmpfs:
- /opt/zookeeper-3.4.6/data
kafka:
image: wurstmeister/kafka:latest
depends_on:
- zookeeper
expose:
- 9092
tmpfs:
- /kafka
environment:
KAFKA_ADVERTISED_LISTENERS: INSIDE://localhost:9094,OUTSIDE://kafka:9092
KAFKA_LISTENERS: INSIDE://:9094,OUTSIDE://:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
LOG4J_LOGGER_KAFKA_AUTHORIZER_LOGGER: DEBUG, authorizerAppender
healthcheck:
test: ["CMD", "kafka-topics.sh", "--list", "--zookeeper", "zookeeper:2181"]
interval: 5s
timeout: 10s
retries: 5