spark standalone worker not able to connect driver - docker

Question

I have setup a simple standalone spark cluster using docker.
I have two docker containers that have spark and one is running as master and anther as worker.
These two containers share a custom bridge network.
I have opened up webui and submit port of master container and can successfully see master webui.
Worker container webui port is also opened and can successfully be seen from my web browser.
The problem happens when I try to run pyspark.
Now from my machine(not inside and docker container), I try to run the following python script but it never completes.
from pyspark import SparkContext, SparkConf
sc = SparkContext("spark://localhost:10077","test")
rdd = sc.parallelize([1,2,3])
for a in rdd.collect():
print(a)
I checked the master and worker webui and it seems that worker node is continuously exiting and recreating a executer.
The error of executor is like this:
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:424)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:413)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:444)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:442)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
... 4 more
Caused by: java.io.IOException: Failed to connect to devmachine/172.17.0.1:40314
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:288)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: devmachine/172.17.0.1:40314
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
I think the main cause it that the worker cannot connect back to the driver(devmachine) with a given port.
However, I have configured the master and worker container to be able to resolve "devmachine" ip address and have confirmed that the resolution works(tested separately with a simple flask running on devmachine and make http request from a worker container. it worked)
From what I understand, I am submitting my pyspark job in "client" mode (instead of cluster mode) and thus my current machine(devmachine) is where driver exists and since client mode states that the workers communicate with the driver, thus the error where worker is trying to connect to devmachine.
Any pointers on what I am doing wrong?
update 22.07.21
okay I have eventually found a workaround to make this simple spark job to work.
Since the original problem seemed to be caused by difficulty of network communication between the driver(host machine) and the worker docker container for some unknown reason, I though perhaps if we submitted the spark job from another docker container which is bridge networked to the master & worker docker container could solve the problem.
I prepare a docker image on top of the image that was used for master/worker, where I only installed python3 and pip and installed pyspark since it lacked these packages.
After that, I lauched a docker container with this new image, made sure that it was connected to the custom bridge network used by master and worker node.
I submitted the python script (after changing the master url inside the python file to spark://master:7077) with the following command:
$ spark-submit t1.py
and it worked well!
Here was the output:
22/07/21 07:25:49 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/07/21 07:25:49 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/07/21 07:25:49 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-ad4336e4-af0c-4100-852f-6b51f8803f94
22/07/21 07:25:49 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
22/07/21 07:25:49 INFO SparkEnv: Registering OutputCommitCoordinator
22/07/21 07:25:49 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/07/21 07:25:49 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
22/07/21 07:25:49 INFO TransportClientFactory: Successfully created connection to master/172.18.0.2:7077 after 27 ms (0 ms spent in bootstraps)
22/07/21 07:25:50 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20220721072549-0005
22/07/21 07:25:50 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220721072549-0005/0 on worker-20220721060838-172.18.0.3-45161 (172.18.0.3:45161) with 8 core(s)
22/07/21 07:25:50 INFO StandaloneSchedulerBackend: Granted executor ID app-20220721072549-0005/0 on hostPort 172.18.0.3:45161 with 8 core(s), 1024.0 MiB RAM
22/07/21 07:25:50 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37799.
22/07/21 07:25:50 INFO NettyBlockTransferService: Server created on 46dd0a966c6f:37799
22/07/21 07:25:50 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/07/21 07:25:50 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 46dd0a966c6f, 37799, None)
22/07/21 07:25:50 INFO BlockManagerMasterEndpoint: Registering block manager 46dd0a966c6f:37799 with 434.4 MiB RAM, BlockManagerId(driver, 46dd0a966c6f, 37799, None)
22/07/21 07:25:50 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 46dd0a966c6f, 37799, None)
22/07/21 07:25:50 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 46dd0a966c6f, 37799, None)
22/07/21 07:25:50 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220721072549-0005/0 is now RUNNING
22/07/21 07:25:50 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
22/07/21 07:25:50 INFO SparkContext: Starting job: collect at /root/t1.py:10
22/07/21 07:25:50 INFO DAGScheduler: Got job 0 (collect at /root/t1.py:10) with 2 output partitions
22/07/21 07:25:50 INFO DAGScheduler: Final stage: ResultStage 0 (collect at /root/t1.py:10)
22/07/21 07:25:50 INFO DAGScheduler: Parents of final stage: List()
22/07/21 07:25:50 INFO DAGScheduler: Missing parents: List()
22/07/21 07:25:50 INFO DAGScheduler: Submitting ResultStage 0 (ParallelCollectionRDD[0] at readRDDFromFile at PythonRDD.scala:274), which has no missing parents
22/07/21 07:25:50 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.9 KiB, free 434.4 MiB)
22/07/21 07:25:50 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1757.0 B, free 434.4 MiB)
22/07/21 07:25:50 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 46dd0a966c6f:37799 (size: 1757.0 B, free: 434.4 MiB)
22/07/21 07:25:50 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1513
22/07/21 07:25:50 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (ParallelCollectionRDD[0] at readRDDFromFile at PythonRDD.scala:274) (first 15 tasks are for partitions Vector(0, 1))
22/07/21 07:25:50 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0
22/07/21 07:25:52 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.18.0.3:59260) with ID 0, ResourceProfileId 0
22/07/21 07:25:52 INFO BlockManagerMasterEndpoint: Registering block manager 172.18.0.3:33016 with 434.4 MiB RAM, BlockManagerId(0, 172.18.0.3, 33016, None)
22/07/21 07:25:52 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (172.18.0.3, executor 0, partition 0, PROCESS_LOCAL, 4464 bytes) taskResourceAssignments Map()
22/07/21 07:25:52 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (172.18.0.3, executor 0, partition 1, PROCESS_LOCAL, 4491 bytes) taskResourceAssignments Map()
22/07/21 07:25:52 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.18.0.3:33016 (size: 1757.0 B, free: 434.4 MiB)
22/07/21 07:25:52 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 456 ms on 172.18.0.3 (executor 0) (1/2)
22/07/21 07:25:52 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 446 ms on 172.18.0.3 (executor 0) (2/2)
22/07/21 07:25:52 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/07/21 07:25:52 INFO DAGScheduler: ResultStage 0 (collect at /root/t1.py:10) finished in 2.357 s
22/07/21 07:25:52 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
22/07/21 07:25:52 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
22/07/21 07:25:52 INFO DAGScheduler: Job 0 finished: collect at /root/t1.py:10, took 2.401307 s
1
2
3
22/07/21 07:25:53 INFO SparkContext: Invoking stop() from shutdown hook
22/07/21 07:25:53 INFO SparkUI: Stopped Spark web UI at http://46dd0a966c6f:4040
22/07/21 07:25:53 INFO StandaloneSchedulerBackend: Shutting down all executors
22/07/21 07:25:53 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
22/07/21 07:25:53 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/07/21 07:25:53 INFO MemoryStore: MemoryStore cleared
22/07/21 07:25:53 INFO BlockManager: BlockManager stopped
22/07/21 07:25:53 INFO BlockManagerMaster: BlockManagerMaster stopped
22/07/21 07:25:53 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/07/21 07:25:53 INFO SparkContext: Successfully stopped SparkContext
22/07/21 07:25:53 INFO ShutdownHookManager: Shutdown hook called
22/07/21 07:25:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-aca43d84-54ab-45a6-87ec-25a644286af0
22/07/21 07:25:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-aca43d84-54ab-45a6-87ec-25a644286af0/pyspark-c7c9f3aa-5810-405b-a375-28e8b3344f68
22/07/21 07:25:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-bfcd5ac6-e629-4fdb-8621-fc76d17ffed3
and here is the worker node's output for this job to compare with the original error log.
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:424)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:413)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:444)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:442)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
... 4 more
Caused by: java.io.IOException: Failed to connect to cl-aicrdev03/172.17.0.1:44731
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:288)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: cl-aicrdev03/172.17.0.1:44731
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
It seems that it had no problem connecting to the driver this time.
But then still it begs theh question.
Why does this simply spark submit work when the driver is a docker container sharing the same bridge network with the master & worker
while it doesn't work when the driver is outside of docker and the worker container has been ensured(I think..) to be able to look at the driver outside?