How do I create a Kafka topic using a docker-compose file? - docker

Question

I'm new to Kafka and I am trying to create a demo Kafka server and work with it.
I have already created Zookeeper and Kafka containers using a docker-compose file and they are started and running fine.
If I docker exec into the Kafka container and run:
kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic
the topic is created successfully... But I would like to spin up the Kafka broker and then programmatically create the topics without user interaction (Will ultimately be for a pipeline)
I've tried two different Kafka images (other was confluentinc/cp-kafka)... I have also tried changing bash to sh under kafka-1.command
Could really do with some help here. Below is my docker-compose file and the error response I'm getting in my terminal for bitnami image & for confluent image.
ERROR (BITNAMI)
/opt/bitnami/scripts/kafka/entrypoint.sh: line 27: exec: bash -c "kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic": not found
ERROR (CONFLUENT)
Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "bash -c \"kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic\"": executable file not found in $PATH: unknown
DOCKER-COMPOSE.YAML
services:
zookeeper-1:
container_name: zookeeper-1
image: zookeeper
restart: always
ports:
- 2181:2181
environment:
- ZOOKEEPER_CLIENT_PORT=2181
volumes:
- ./config/zookeeper-1/zookeeper.properties:/kafka/config/zookeeper.properties
kafka-1:
container_name: kafka-1
image: bitnami/kafka
depends_on:
- zookeeper-1
ports:
- 29092:29092
- 9092:9092
environment:
- KAFKA_ZOOKEEPER_CONNECT=zookeeper-1:2181
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092
- ALLOW_PLAINTEXT_LISTENER=yes
command:
- bash -c "kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic"

Answer 1

For both errors, the command you are providing is being appended to the container's entrypoints, which do not accept bash commands... You would need to override both the entrypoint and the command, as well as give the full path to the kafka-topics script since it is not on the $PATH.
However, you cannot do this with the command on the same container as the broker because that overrides the command that actually starts the server.
You will need a secondary "init container" that creates the topics , but it would be easier for you to write this topic creation logic into your own producer/consumer applications with an AdminClient.createTopics call (assuming Java).
Otherwise, you can use wurstmeister/kafka image which has an environment variable KAFKA_CREATE_TOPICS for this purpose, or just use docker-compose exec kafka-1 "..." after the containers are up