Starting a Kafka topics using Docker Compose with spotify/kafka? - docker

Question

I am attempting to connect Kafka topics to my front-end Java Spring application. I am utilizing Docker Compose and have tried to connect using two different Kafka images.
With wurstmeister/kafka I have been able to get the Kafka topics up and running by this service in my docker.compose.yml file. But I have not been able to connect the created topics to my front-end Java Spring application.
kafka:
image: wurstmeister/kafka:0.10.2.0
ports:
- "9092:9092"
expose:
- "9092"
- "2181"
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_CREATE_TOPICS: "test-topic1:1:1, test-topic2:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- zookeeper
Secondly, with spotify/kafka, I am having difficulty actually creating the topics with Kafka. In the documentation, it is looking for the topics as an environment variable, but the following docker-compose.yml service is not creating a topic. I have also tried putting quotes around the test-topic but that did not work as well.
kafka:
image: spotify/kafka
ports:
- "9092:9092"
- "2181:2181"
hostname: kafka
expose:
- "9092"
- "2181"
environment:
TOPICS: test-topic
I do not know if this is necessary, but my entire docker-compose.yml file is as follows, take note that the zookeeper service is only required if you use wurstmeister/kafka.
docker-compose.yml
version: '2'
services:
# zookeeper:
# image: wurstmeister/zookeeper
# ports:
# - "2181:2181"
kafka:
image: spotify/kafka
ports:
- "9092:9092"
- "2181:2181"
hostname: kafka
expose:
- "9092"
- "2181"
environment:
TOPICS: test-topic
redis:
image: redis
ports:
- "6379"
restart: always
kafka-websocket-connector:
build: ./kafka-websocket-connector
image: andrewterra/kafka-websocket-connector
ports:
- "8077:8077"
# - "9092:9092"
depends_on:
- kafka
- redis
# - zookeeper
links:
- kafka
- redis

Answer 1

Rather late, but you could use something like the following to use a shell script to create your topic:
command: >
bash -c
"(sleep 15s &&
/opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh
--create
--zookeeper
localhost:2181 --replication-factor 1 --partitions 1
--topic my_topic &) && (supervisord -n)"

Answer 2

Container run command to create topic
Use the container command docker run --net=host --rm. In the following example, the zookeeper is running on port 22181, please use the respective topic name, port.
Create
docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-topics --create --topic customer --partitions 1 --replication-factor 1 --if-not-exists --zookeeper localhost:22181
Describe
docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-topics --zookeeper localhost:22181 --topic customer --describe
List
docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-topics --list --zookeeper localhost:22181
Delete
docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-topics --delete --topic customer --zookeeper localhost:22181

Answer 3

The TOPICS environment variable is only used for the kafkaproxy image. https://github.com/spotify/docker-kafka#running-the-proxy
For kafka image, you will need to create the topics with a client.