Download spark-submit without all the Spark framework to make lite Docker image - docker

Question

Most of the Docker image that embed Apache Spark have the whole spark archive in it.
Also most of the time, we submit the spark application on kubernetes, hence the spark job is running on other Docker container.
As such, I am wondering, in order to make the Docker image smaller, how to embed spark-submit feature?

Answer 1

That's a great question! I had a look (for the latest one on the Downloads page: 3.3.1) and found the following:
Looking at the contents of $SPARK_HOME/bin/spark-submit, you can see the following line:
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$#"
Ok, so it looks like the $SPARK_HOME/bin/spark-submit script calls the $SPARK_HOME/bin/spark-class script. Let's have a look at that one.
Similar to spark-submit, spark-class calls the load-spark-env.sh script like so:
. "${SPARK_HOME}"/bin/load-spark-env.sh
This load-spark-env.sh script calls other scripts of its own as well. But there is also a bit about Spark jars in spark-class:
# Find Spark jars.
if [ -d "${SPARK_HOME}/jars" ]; then
SPARK_JARS_DIR="${SPARK_HOME}/jars"
else
SPARK_JARS_DIR="${SPARK_HOME}/assembly/target/scala-$SPARK_SCALA_VERSION/jars"
fi
if [ ! -d "$SPARK_JARS_DIR" ] && [ -z "$SPARK_TESTING$SPARK_SQL_TESTING" ]; then
echo "Failed to find Spark jars directory ($SPARK_JARS_DIR)." 1>&2
echo "You need to build Spark with the target \"package\" before running this program." 1>&2
exit 1
else
LAUNCH_CLASSPATH="$SPARK_JARS_DIR/*"
fi
# Add the launcher build dir to the classpath if requested.
if [ -n "$SPARK_PREPEND_CLASSES" ]; then
LAUNCH_CLASSPATH="${SPARK_HOME}/launcher/target/scala-$SPARK_SCALA_VERSION/classes:$LAUNCH_CLASSPATH"
fi
So as you can see, it is referencing the spark jars directory (288MB of the total 324MB for Spark 3.3.1) and putting that on the launch classpath. Now, it's very possible that not all of those jars are needed in the case of submitting an application on kubernetes. But at least you need some kind of library to translate a spark application to some kubernetes configuration that your Kubernetes API server can understand.
So my conclusion from this bit is:
We can quite easily follow which files are exactly needed. In first instance, I would say anything in $SPARK_HOME/bin and $SPARK_HOME/conf. But that is no issue since they are all very small scripts/conf files.
Some of those scripts though, are putting the jars directory in your classpath for the final java command.
Maybe they don't need all the jars, but they will need some kind of library to connect to the Kubernetes API server. So I would expect some jar to be needed there. I see there is a jar called kubernetes-model-core-5.12.2.jar. Maybe this one?
Since most of the size of this $SPARK_HOME folder comes from those jars, you can try to delete some jars and run your spark-submit jobs to see what happens. I would think that, amongst others, jars like commons-math3-3.6.1.jar or spark-mllib_2.12-3.3.1.jar would not be necessary for a simple spark-submit to a Kubernetes API Server.
(All those specific jars just come from that one Spark version I talked about in the start of this post)
Really interesting question, I hope this helps you a bit! Just try deleting some jars, run your spark-submit and see what happens!