Install pyspark + pytest in docker container - docker

Question

I'm trying to unit test my pyspark code using pytest but can't figure out the proper steps and method of installation. I was able to get this working locally on my Mac using this tutorial. I've tried 2 methods to accomplish this:
Try to replicate what I did on my Mac in the Dockerfile. i.e. install pypark, apache-spark, java 8, scala, pytest, and make sure I get the ENV paths correct.
Use an image from docker like bitnami.
I attempted (1) but could not find the right RUN command to install java properly.
For (2), is there any way in the Dockerfile for me to install bitnami separately from pytest since bitnami does not give root access?
Note:
Bitnami does not put py4j in the PYTHONPATH so I had to add this line to the docker file:
ENV PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.9.3-src.zip:${PYTHONPATH}"

Answer 1

How about building your image FROM bitnami:spark and adding pytest?
I created test_spark.py:
from pyspark.sql import SparkSession
def test1():
spark = SparkSession.builder.getOrCreate()
data = spark.sql("SELECT 1").collect()
assert data == [(1,)]
and a Dockerfile:
FROM bitnami/spark:latest
RUN pip install pytest py4j
COPY test_spark.py .
CMD python -m pytest test_spark.py
Now I can build and run my container and execute the pytests:
docker build . -t pytest_spark && docker run pytest_spark
[+] Building 0.1s (8/8) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 36B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/bitnami/spark:latest 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 35B 0.0s
=> [1/3] FROM docker.io/bitnami/spark:latest 0.0s
=> CACHED [2/3] RUN pip install pytest py4j 0.0s
=> CACHED [3/3] COPY test_spark.py . 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:33b5f945afb750aecb0a8e1b2e811eb71b2bb2e67752e1b73a2c321bcc433841 0.0s
=> => naming to docker.io/library/pytest_spark 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
08:13:35.34
08:13:35.34 Welcome to the Bitnami spark container
08:13:35.35 Subscribe to project updates by watching https://github.com/bitnami/containers
08:13:35.35 Submit issues and feature requests at https://github.com/bitnami/containers/issues
08:13:35.35
============================= test session starts ==============================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
rootdir: /opt/bitnami/spark
collected 1 item
test_spark.py . [100%]
============================== 1 passed in 10.11s ==============================