How to execute a prefect Flow on a docker image? - docker

Question

My goal:
I have a built docker image and want to run all my Flows on that image.
Currently:
I have the following task which is running on a Local Dask Executor.
The server on which the agent is running is a different python environment from the one needed to execute my_task - hence the need to run inside a pre-build image.
My question is: How do I run this Flow on a Dask Executor such that it runs on the docker image I provide (as environment)?
import prefect
from prefect import task, Flow
from prefect.engine.executors import LocalDaskExecutor
from prefect.environments import LocalEnvironment
#task
def hello_task():
logger = prefect.context.get("logger")
logger.info("Hello, Docker!")
with Flow("My Flow") as flow:
results = hello_task()
flow.environment = LocalEnvironment(
labels=[], executor=LocalDaskExecutor(scheduler="threads", num_workers=2),
)
I thought that I need to start the server and the agent on that docker image first (as discussed here), but I guess there can be a way to simply run the Flow on a provided image.
Update 1
Following this tutorial, I tried the following:
import prefect
from prefect import task, Flow
from prefect.engine.executors import LocalDaskExecutor
from prefect.environments import LocalEnvironment
from prefect.environments.storage import Docker
#task
def hello_task():
logger = prefect.context.get("logger")
logger.info("Hello, Docker!")
with Flow("My Flow") as flow:
results = hello_task()
flow.storage = Docker(registry_url='registry.gitlab.com/my-repo/image-library')
flow.environment = LocalEnvironment(
labels=[], executor=LocalDaskExecutor(scheduler="threads", num_workers=2),
)
flow.register(project_name="testing")
But this created an image which it then uploaded to the registry_url provided. Afterwards when I tried to run the registered task, it pulled the newly created image and the task is stuck in status Submitted for execution for minutes now.
I don't understand why it pushed an image and then pulled it? Instead I already have an image build on this registry, I'd like to specify an image which should be used for task execution.

Answer 1

The way i ended up achieve this is as follows:
Run prefect server start on the server (i.e. not inside docker).
Apparently docker-compose in docker is not a good idea.
Run prefect agent start inside the docker image
Make sure the flows are accessible by the docker image (i.e. by mounting a shared volume between the image and the server for
example)
You can see the source of my answer here.