Using Spark Streaming with Kafka docker container errors? - docker

Question

I am using Kafka docker-compose setting with below docker-compose.yml installed in VM-Ware machine.
When I connect to it by pyspark.streaming.kafka.KafkaUtils, it released some errors.
Please help me resolve this problems.
I used configuration from https://rmoff.net/2018/08/02/kafka-listeners-explained/
docker-compose.yml file
version: '3.7'
services:
zookeeper:
image: "confluentinc/cp-zookeeper:latest"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- "2181:2181"
# This has three listeners you can experiment with.
# BOB for internal traffic on the Docker network
# FRED for traffic from the Docker-host machine (`localhost`)
# ALICE for traffic from outside, reaching the Docker host on the 192.168.231.145
# Use
kafka0:
image: "confluentinc/cp-enterprise-kafka:latest"
ports:
- '9092:9092'
- '29094:29094'
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 0
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://kafka0:9092,LISTENER_ALICE://0.0.0.0:29094
KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://localhost:9092,LISTENER_ALICE://192.168.231.145:29094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT,LISTENER_ALICE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_BOB
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100
kafkacat:
image: confluentinc/cp-kafkacat
command: sleep infinity
python code i used to connect from vm-ware hosted machine
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
from confluent_kafka import Producer, Consumer
import socket
import json
if __name__ == "__main__":
sc = SparkContext(appName="Processing_raw_data")
ssc = StreamingContext(sc, 1)
in_stream = KafkaUtils.createStream(ssc, "192.168.231.145:2181", socket.gethostname(), {"testing": 1}, {"auto.offset.reset": "smallest"})
in_stream.pprint()
ssc.start()
ssc.awaitTermination()
Errors
21/06/19 19:44:24 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
21/06/19 19:44:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
-------------------------------------------
Time: 2021-06-19 19:44:27
-------------------------------------------
21/06/19 19:44:27 WARN AppInfo$: Can't read Kafka version from MANIFEST.MF. Possible cause: java.lang.NullPointerException
[Stage 0:> (0 + 1) / 1]-------------------------------------------
Time: 2021-06-19 19:44:28
-------------------------------------------
21/06/19 19:44:28 WARN ClientUtils$: Fetching topic metadata with correlation id 0 for topics [Set(testing)] from broker [id:0,host:kafka0,port:29092] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
21/06/19 19:44:28 WARN ConsumerFetcherManager$LeaderFinderThread: [dino-computer_dino-computer-1624106667579-fbe9ab2d-leader-finder-thread], Failed to find leader for Set([testing,0])
kafka.common.KafkaException: fetching topic metadata for topics [Set(testing)] from broker [ArrayBuffer(id:0,host:kafka0,port:29092)] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
... 3 more