Unable to connect Apache Spark running in docker - docker

Question

I am trying to connect a Spark Cluster running within the docker via the host system. I tried both the python script and spark-shell both gave the same results:
Within Docker
park-master_1 | 20/07/24 10:13:26 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
spark-master_1 | java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 1574364215946805297, local class serialVersionUID = 6543101073799644159
spark-master_1 | at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
spark-master_1 | at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
spark-master_1 | at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
spark-master_1 | at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
spark-master_1 | at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
spark-master_1 | at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
spark-master_1 | at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
spark-master_1 | at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
spark-master_1 | at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
spark-master_1 | at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
spark-master_1 | at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
spark-master_1 | at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108)
spark-master_1 | at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1$$anonfun$apply$1.apply(Nett
Running spark-shell on the command line from the host system gives the following error:
➜
docker-spark-cluster git:(master) ✗ spark-shell --master spark://localhost:7077
20/07/24 15:13:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/07/24 15:14:25 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
20/07/24 15:14:25 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
20/07/24 15:14:25 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
20/07/24 15:14:26 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:92)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:565)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2555)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:930)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:42)
at $line3.$read.<init>(<console>:44)
at $line3.$read$.<init>(<console>:48)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
Docker Containers
git:(master) ✗ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dfe3d47790ee spydernaz/spark-worker:latest "/bin/bash /start-wo…" 42 hours ago Up 23 minutes 0.0.0.0:32769->8081/tcp docker-spark-cluster_spark-worker_2
c5e36b94efdd spydernaz/spark-worker:latest "/bin/bash /start-wo…" 42 hours ago Up 23 minutes 0.0.0.0:32768->8081/tcp docker-spark-cluster_spark-worker_3
60f3d29e9059 spydernaz/spark-worker:latest "/bin/bash /start-wo…" 42 hours ago Up 23 minutes 0.0.0.0:32770->8081/tcp docker-spark-cluster_spark-worker_1
d11c67d462fb spydernaz/spark-master:latest "/bin/bash /start-ma…" 42 hours ago Up 23 minutes 6066/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:9090->8080/tcp docker-spark-cluster_spark-master_1
➜ docker-spark-cluster git:(master) ✗
Spark Shell Commands
spark-shell --master spark://localhost:7077

Answer 1

As #koiralo already mentioned in the comments, this is due to the version difference in pySpark running in local and on the server.
Had the same error and it was fixed once the versions at both the places were matched.