Kafka Spark Streaming : Broker may not be available [Docker] - docker

Question

I'm new to docker. I'm trying to run a spark streaming application using docker.
I have kafka and spark streaming application running separately in 2 containers.
My kafka service is up and running fine. I tested with $KAFKA_HOME/bin/kafka-console-producer.sh and $KAFKA_HOME/bin/kafka-console-consumer.sh. I'm able to receive messages.
But when I'm running my spark streaming application, it's showing:
[Consumer clientId=consumer-1, groupId=consumer-spark] Connection to node -1 could not be established. Broker may not be available.
So, I'm not able to consume messages.
kafka : docker-compose.yml
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_LISTENERS: PLAINTEXT://:9092
depends_on:
- zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Spark Streaming code:
val sparkConf = new SparkConf().setAppName("Twitter Ingest Data")
sparkConf.setIfMissing("spark.master", "local[2]")
val ssc = new StreamingContext(sparkConf, Seconds(2))
val kafkaTopics = "sentiment"
val kafkaBroker = "kafka:9092"
val topics : Set[String] = kafkaTopics.split(",").map(_.trim).toSet
val kafkaParams = Map[String,Object](
"bootstrap.servers" -> kafkaBroker,
"group.id" -> "consumer-spark",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer]
)
logger.info("Connecting to broker...")
logger.info(s"kafkaParams: $kafkaParams")
val tweetStream = KafkaUtils.createDirectStream[String, String](
ssc,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams))
I'm not sure if I'm missing anything.
Any help would be highly appreciated!!

Answer 1

If you're new to Docker, I wouldn't recommend having Kafka or Spark being the first things you're trying it with. Besides, seems like you just copied the wurstmeister example one without reading the README about configuring it... (which I can tell because you don't need the build: . property because that container already exists on DockerHub)
Basically, Kafka is only available within your Docker network via this configuration
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
You will need to edit this to have the port forwarding work properly from outside of Docker Compose's default network, or you must run your Spark code within a container as well.
If the Spark code is not in a container, then pointing it at kafka:9092 won't work at all
Ref. Kafka listeners explained
And lots of previous questions with similar problems (the issue is not just Spark related)