In MLflow project using docker environment, how to setup aws credentials - docker

Question

I am working on using 'MLflow' project and one use case is like this.
The MLflow running target/environment is docker.
Data lives on aws s3
When developing on a laptop. The laptop has an aws profile to access data.
(When developing on EC2, the EC2 have role attached to access s3)
Currently, I have credentials stored on the host as '~/.aws/credential', and can access s3 in the host. Question is: In MLflow project, how do I make program running on docker access s3 files?
Note that the question is not "in general" how to setup docker. The question is the recommended way to do the aws setup/configuration in a MLflow project. Thanks!

Answer 1

You can use a volume, for application data.
Specifically, for aws credentials, you can mount the credentials directory itself,
Obviously, you'll need to make sure to install any required dependencies for aws or mlflow. But here are the required parts for adding a user and mounting the credentials as a volume.
First, in your Dockerfile,
# add user with home directory
RUN useradd -m mlflow
# set default user
USER mlflow
# set working directory
WORKDIR /home/mlflow
Then to mount during run,
docker run -it -v "${HOME}"/.aws:/home/mlflow/.aws \
mlflow
Note: make sure to never hard-code credentials inside of any Docker containers.