Apache Kafka, NotLeaderOrFollowerException when sending message - docker

Question

Trying to run Apache Kafka in docker containers, using the following docker-compose.yml:
version: '3.1'
services:
zookeeper:
container_name: zookeeper
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
volumes:
- /home/vagrant/myapp/runtime.local/volumes/zookeeper/01/data:/var/lib/zookeeper/data
- /home/vagrant/myapp/runtime.local/volumes/zookeeper/01/log:/var/lib/zookeeper/log
ports:
- 2181:2181
kafka:
container_name: kafka
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
- 9092:9092
volumes:
- /home/vagrant/myapp/runtime.local/volumes/kafka/01/data:/var/lib/kafka/data
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_NUM_PARTITIONS: 3
KAFKA_HEAP_OPTS: -Xmx512M -Xms512M
It seems that I can create a topic
docker exec -it kafka kafka-topics --create --topic test --partitions 1 --replication-factor 1 --if-not-exists --zookeeper zookeeper:2181
Created topic test.
But when I try to send a message via kafka-console-producer I got the following error:
docker exec -it kafka kafka-console-producer --topic test --bootstrap-server kafka:9092
>My first message
>[2021-07-14 15:39:21,010] WARN [Producer clientId=console-producer] Got error produce response with correlation id 4 on topic-partition test-2, retrying (2 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2021-07-14 15:39:21,011] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test-2 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2021-07-14 15:39:21,112] WARN [Producer clientId=console-producer] Got error produce response with correlation id 6 on topic-partition test-2, retrying (1 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2021-07-14 15:39:21,112] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test-2 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2021-07-14 15:39:21,643] WARN [Producer clientId=console-producer] Got error produce response with correlation id 8 on topic-partition test-2, retrying (0 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2021-07-14 15:39:21,643] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test-2 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2021-07-14 15:39:21,751] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.
[2021-07-14 15:39:21,753] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test-2 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
The following is a partial stack trace in the kafka container:
[2021-07-14 15:39:12,232] ERROR [Broker id=1] Error while processing LeaderAndIsr request correlationId 1 received from controller 1 epoch 1 for partition test-1 (state.change.logger)
java.io.IOException: Invalid argument
at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1002)
at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:124)
at kafka.log.OffsetIndex.<init>(OffsetIndex.scala:54)
at kafka.log.LazyIndex$.$anonfun$forOffset$1(LazyIndex.scala:106)
at kafka.log.LazyIndex.$anonfun$get$1(LazyIndex.scala:63)
at kafka.log.LazyIndex.get(LazyIndex.scala:60)
at kafka.log.LogSegment.offsetIndex(LogSegment.scala:64)
at kafka.log.LogSegment.readNextOffset(LogSegment.scala:456)
at kafka.log.Log.$anonfun$recoverLog$6(Log.scala:921)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
at scala.Option.getOrElse(Option.scala:201)
at kafka.log.Log.recoverLog(Log.scala:921)
at kafka.log.Log.$anonfun$loadSegments$3(Log.scala:801)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2465)
at kafka.log.Log.loadSegments(Log.scala:801)
at kafka.log.Log.<init>(Log.scala:328)
at kafka.log.Log$.apply(Log.scala:2601)
at kafka.log.LogManager.$anonfun$getOrCreateLog$1(LogManager.scala:830)
at scala.Option.getOrElse(Option.scala:201)
at kafka.log.LogManager.getOrCreateLog(LogManager.scala:783)
at kafka.cluster.Partition.createLog(Partition.scala:344)
at kafka.cluster.Partition.createLogIfNotExists(Partition.scala:324)
at kafka.cluster.Partition.$anonfun$makeLeader$1(Partition.scala:564)
at kafka.cluster.Partition.makeLeader(Partition.scala:548)
at kafka.server.ReplicaManager.$anonfun$makeLeaders$5(ReplicaManager.scala:1568)
at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:1566)
at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:1411)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:258)
at kafka.server.KafkaApis.handle(KafkaApis.scala:171)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:74)
at java.base/java.lang.Thread.run(Thread.java:829)
Am I doing something wrong?