How to fix an inconsistent clusterId value running kafka in Docker - docker

Question

I have a docker-compose file that starts 3 zookeeper containers, and 3 kafka containers. In essence, I have a cluster with three brokers.
When running the docker-compose file, I get the following error:
mcflow_kafka_1 | [2022-09-11 03:50:52,600] INFO [feature-zk-node-event-process-thread]: Starting (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
mcflow_kafka_1 | [2022-09-11 03:50:52,747] INFO Updated cache from existing <empty> to latest FinalizedFeaturesAndEpoch(features=Features{}, epoch=0). (kafka.server.FinalizedFeatureCache)
mcflow_kafka_1 | [2022-09-11 03:50:52,750] INFO Cluster ID = 8Z5JcWoMRUC_qnTWg9q1wQ (kafka.server.KafkaServer)
mcflow_kafka_1 | [2022-09-11 03:50:52,755] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
mcflow_kafka_1 | [2022-09-11 03:50:52,755] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
mcflow_kafka_1 | kafka.common.InconsistentClusterIdException: The Cluster ID 8Z5JcWoMRUC_qnTWg9q1wQ doesn't match stored clusterId Some(maywBQemSJuPZzgIh3BE5A) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
mcflow_kafka_1 | at kafka.server.KafkaServer.startup(KafkaServer.scala:252)
mcflow_kafka_1 | at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
mcflow_kafka_1 | at kafka.Kafka$.main(Kafka.scala:82)
mcflow_kafka_1 | at kafka.Kafka.main(Kafka.scala)
This is occurring because some file meta.properties is specifying a different clusterId value than what my container's clusterId is. Problem is, I'm not sure how to correct/locate this meta.properties file, since I'm running kafka in a Docker container.
Below is the docker-compose.yml file I'm using:
---
version: "3.0"
services:
zookeeper1:
image: confluentinc/cp-zookeeper:6.1.0
restart: always
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: "2181"
ZOOKEEPER_TICK_TIME: "2000"
ZOOKEEPER_SERVERS: "zookeeper1:22888:23888"
ports:
- "2181:2181"
zookeeper2:
image: confluentinc/cp-zookeeper:6.1.0
restart: always
environment:
ZOOKEEPER_SERVER_ID: 2
ZOOKEEPER_CLIENT_PORT: "2182"
ZOOKEEPER_TICK_TIME: "2000"
ZOOKEEPER_SERVERS: "zookeeper2:22888:23888"
ports:
- "2182:2181"
zookeeper3:
image: confluentinc/cp-zookeeper:6.1.0
restart: always
environment:
ZOOKEEPER_SERVER_ID: 3
ZOOKEEPER_CLIENT_PORT: "2183"
ZOOKEEPER_TICK_TIME: "2000"
ZOOKEEPER_SERVERS: "zookeeper2:22888:23888"
ports:
- "2183:2181"
kafka1:
container_name: mcflow_kafka_1
image: confluentinc/cp-kafka:6.1.0
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
ports:
# Exposes 29092 for external connections to the broker
# Use kafka1:9092 for connections internal on the docker network
# See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
#- "29092:29092"
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper1:2181,zookeeper2:2182,zookeeper3:2183"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >-
PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: >-
PLAINTEXT://kafka1:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_BROKER_ID: 1
KAFKA_BROKER_RACK: "r1"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_SCHEMA_REGISTRY_URL: "schemaregistry1:8085"
kafka2:
container_name: mcflow_kafka_2
image: confluentinc/cp-kafka:6.1.0
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
ports:
# Exposes 29092 for external connections to the broker
# Use kafka1:9092 for connections internal on the docker network
# See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
#- "29092:29092"
- "9093:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper1:2181,zookeeper2:2182,zookeeper3:2183"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >-
PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: >-
PLAINTEXT://kafka2:9093,PLAINTEXT_HOST://localhost:29092
KAFKA_BROKER_ID: 2
KAFKA_BROKER_RACK: "r1"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_SCHEMA_REGISTRY_URL: "schemaregistry1:8085"
kafka3:
container_name: mcflow_kafka_3
image: confluentinc/cp-kafka:6.1.0
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
ports:
# Exposes 29092 for external connections to the broker
# Use kafka1:9092 for connections internal on the docker network
# See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
#- "29092:29092"
- "9094:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper1:2181,zookeeper2:2182,zookeeper3:2183"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >-
PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: >-
PLAINTEXT://kafka3:9094,PLAINTEXT_HOST://localhost:29092
KAFKA_BROKER_ID: 3
KAFKA_BROKER_RACK: "r1"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_SCHEMA_REGISTRY_URL: "schemaregistry1:8085"
How can I resolve this issue?