schema-registry in KAFKA not able to retrieve Cluster ID - docker

Question

I try to install a Kafka environment based on confluent images. After "docker-compose up" all my containers are up and running, but after one minute the schema-registry failed
in the scheme-registry log I found this error message explaining that it failed to get the Kafka cluster Id
I checked the kafka logs and found this :
"[2021-08-05 15:59:17,074] INFO Cluster ID = ddchQ8odQM-hF67TJO97Ng (kafka.server.KafkaServer)"
So Cluster ID is well created. It seems that schema-registry is not able to retreive the Cluster ID but I really don't understand what happen here, I think it is a network issue, I tried many things to fix it but whithout success
here my docker-compose.yaml
services:
zookeeper:
image: confluentinc/cp-zookeeper
hostname: zookeeper
container_name: zookeeper
# networks:
# - my-network
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
deploy:
resources:
limits:
cpus: "1.00"
memory: "1024M"
kafka:
image: confluentinc/cp-kafka
container_name: kafka
depends_on:
- zookeeper
# networks:
# - my-network
ports:
- 9092:9092
- 30001:30001
environment:
# KAFKA_CREATE_TOPICS: toto
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100
KAFKA_JMX_PORT: 30001
KAFKA_JMX_HOSTNAME: kafka
KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
deploy:
resources:
limits:
cpus: "1.00"
memory: "2048M"
kafka-jmx-exporter:
build: ./materials/tools/prometheus-jmx-exporter
container_name: jmx-exporter
ports:
- 8080:8080
links:
- kafka
# networks:
# - my-network
environment:
JMX_PORT: 30001
JMX_HOST: kafka
HTTP_PORT: 8080
JMX_EXPORTER_CONFIG_FILE: kafka.yml
deploy:
resources:
limits:
cpus: "1.00"
memory: "1024M"
prometheus:
build: ./materials/tools/prometheus
container_name: prometheus
# networks:
# - my-network
ports:
- 9090:9090
spark-master:
container_name: spark-master
build: ./materials/spark
user: root
# networks:
# - my-network
volumes:
- ./materials/spark/connectors:/connectors
- ./materials/spark/scripts:/scripts/
- ./materials/consumer:/scripts/consumer
- ./secrets:/scripts/secrets
- ./materials/spark/jars_dir:/opt/bitnami/spark/.ivy2:z
ports:
- 8085:8080
- 7077:7077
- 4040:4040
environment:
- INIT_DAEMON_STEP=setup_spark
deploy:
resources:
limits:
cpus: "1.00"
memory: "1024M"
# - SPARK_MODE=master
# - SPARK_RPC_AUTHENTICATION_ENABLED=no
# - SPARK_RPC_ENCRYPTION_ENABLED=no
# - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
# - SPARK_SSL_ENABLED=no
spark-worker-1:
container_name: spark-worker-1
build: ./materials/spark
user: root
# networks:
# - my-network
depends_on:
- spark-master
ports:
- 8083:8085
- 4041:4040
environment:
- "SPARK_MASTER=spark://spark-master:7077"
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://spark-master:7077
- SPARK_WORKER_MEMORY=1G
- SPARK_WORKER_CORES=1
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
deploy:
resources:
limits:
cpus: "1.00"
memory: "2048M"
reservations:
cpus: "1.00"
memory: "1024M"
schema-registry:
image: confluentinc/cp-schema-registry
hostname : schema-registry
container_name : schema-registry
#command: /bin/sh -c 'tail -f /dev/null'
command: /bin/schema-registry-start /etc/schema-registry/schema-registry.properties
depends_on:
- kafka
ports:
- 8081:8081
# networks:
# - my-network
environment:
# SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:29092
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka-1:9092
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
SCHEMA_REGISTRY_DEBUG: "true"
SCHEMA_REGISTRY_KAFKASTORE.INIT.TIMEOUT.MS: 120000
deploy:
resources:
limits:
cpus: "1.00"
memory: "2048M"
producer:
build: ./materials/producer
container_name: producer
depends_on:
- kafka
# networks:
# - my-network
environment:
KAFKA_BROKER_URL: kafka-1:9092
TRANSACTIONS_PER_SECOND: 30
kafkastream:
build: ./materials/kafkastream
container_name: kafkastream
depends_on:
- kafka
# networks:
# - my-network
environment:
KAFKA_BROKER_URL: kafka-1:9092
TRANSACTIONS_PER_SECOND: 5
rest-proxy:
image: confluentinc/cp-kafka-rest
depends_on:
- kafka
- schema-registry
# networks:
# - my-network
ports:
- 8082:8082
hostname: rest-proxy
container_name: rest-proxy
#command: /bin/kafka-rest-start
environment:
KAFKA_REST_HOST_NAME: rest-proxy
KAFKA_REST_BOOTSTRAP_SERVERS: kafka:29092
KAFKA_REST_LISTENERS: http://0.0.0.0:8082
KAFKA_REST_SCHEMA_REGISTRY_URL: http://schema-registry:8081
#networks:
#my-network:
# external: false
# my-network:
My last try was to completly remove the network in the docker-compose file, that is why all the lines related to networks are commented here.
Any hint or idea will be appreciate
Thanks

Answer 1

I finally found the solution. My mystake was to add the following line in my docker-compose.yml file : "command: /bin/schema-registry-start /etc/schema-registry/schema-registry.properties". In that way, schema-registry start by taken into acount the default configuration of the schema-registry.properties file that is of course not suitable to my local installation and ignore all the environment parameter passed in the docke-compose.yaml file.

Answer 2

PLAINTEXT_HOST://localhost:9092 , change to kafka-1 or use kafka:29092