So, I'm trying to package my WordPress image in a way that all files except the uploads are persisted. In order to do so, I have created my Dockerfile which uses the official WordPress image as its base, and adds the files from an archive (containing all the WordPress files, themes, plugins, etc.), like so:
FROM wordpress
ADD archive.tar.gz /var/www/html/
Since I want the uploads to be persisted, I have created a separate data volume container, e.g. test2.com-wp-data:
docker create -v /var/www/html/wp-content/uploads —name test2.com-wp-data wordpress
Then I simply mount it via —-volumes-from flag:
docker run —name test2.com --volumes-from test2.com-wp-data -d --link test2.com-mysql:mysql myimage
However, when I inspect my newly created container, I cannot find /var/www/html/wp-content/uploads:
# docker inspect -f '{{.HostConfig.VolumesFrom}}' test2.com
[test2.com-wp-data]
# docker inspect -f '{{.Volumes}}' test2.com
map[/var/www/html:/var/lib/docker/vfs/dir/4fff1d36d5aacd0b2c73977acf8fe680bda6fd891f2c4410a90f6c2dca4aaedf]
I can see that both /var/www/html and /var/www/html/wp-content/uploads are set up as volumes in my test2.com-wp-data data container:
# docker inspect -f '{{.Config.Volumes}}' test2.com-wp-data
map[/var/www/html:map[] /var/www/html/wp-content/uploads:map[]]
I know that the wordpress image by default creates a /var/www/html volume, for which I don't really mind, but does that mean that anything that is below that folder is ignored if mounted separately? Will I need to build my own WordPress image in order to have /var/www/html/wp-content/uploads set as a volume in my WordPress container?
Thank you very much for your time!
EDIT: I've tested a different setup with a folder that has nothing to do with /var/www/html, and the result is the same: —-volumes-from is ignored.
Version 1.4 + of docker should be what you need to get this working. Older versions of docker don't seem to play nicely with data-only containers instantiated with "create" rather than "run".
Well, after some further testing I've realised that despite what the documentation indicates, docker create alone is not enough to get a working data volume working. I've only managed to get working data volumes by instantiating them with the docker run command, as follows:
docker run —-name data -v /var/www/html/wp-content/uploads mysql true
This way the container exits immediately, but if I use it to attach the data volume to another container it works as expected.
If anyone knows any specific reason behind this behaviour, I'd be glad to learn more, especially since the documentation seems to be misleading.
Thanks!
EDIT: It turns out I was using Docker 1.3.x, which hadn't implemented this feature yet, hence why the documentation was misleading for me!
Related
I'm fairly new to using Docker and Docker Compose (using Docker Compose for this particular problem). Here is what I know so far about the problem I am facing: When using volumes when there are contents available in the host folder as well as the container's folder, the files inside the container's folder are hidden and the host's files are then made available to the container.
I want to use it the other way round. I would like to make available the container's files (that were copied into the image in the Dockerfile) to the host folder.
Is there a way to do that?
Here are a bunch of screenshots of my Dockerfile and Docker Compose to show my setup.
Dockerfile Screenshot
DockerCompose Screenshot
Thanks in advance! :)
I've come across the same thing many times and the way I go about it is as follows.
As the host volume will always take priority over the container filesystem, you have to copy the files out of the container to the host first, then volume mount them back - this way you get what was there originally, and also what might change in the future (by the container).
The following is all pseudo code, but should hopefully simulate the concept:
First run the main container:
docker run --rm -d --name my-container registry/image-name
Then copy the files you want from it to the local filesystem
docker cp my-container:/files/i/want ./files
Then stop the original container
docker stop my-container
Then mount them back into the container on the next run
docker run --rm -d --name my-container -v ./files:/files/i/want registry/image-name
Obviously you've mentioned compose there also, so just reflect the volume mapping into the compose format - the copy stuff will need to be done via standard docker however in line with the above.
Note: I wrote the above commands blind, but will check them over at lunch and correct any mistypes - but the concept is correct
I have a Meteor App deployed with Meteor UP to Ubuntu.
From this App I need to read a file which is located outside App container on the host server.
How can I do that?
I've tried to set up volumes in the mup.js but no luck. It seems that I'm missing how to correctly provide /host/path and /container/path
volumes: {
// passed as '-v /host/path:/container/path' to the docker run command
'/host/path': '/container/path',
'/second/host/path': '/second/container/path'
},
Read the docs for Docker mounting volumes but obviously can't understand it.
Let's say file is in /home/dirname/filename.csv.
How to correctly mount it into App to be able to access it from the Application?
Or maybe there are other possibilities to access it?
Welcome to Stack Overflow. Let me suggest another way of thinking about this...
In a scalable cluster, docker instances can be spun up and down as the load on the app changes. These may or may not be on the same host computer, so building a dependency on the file system of the host isn't a great idea.
You might be better to think of using a file storage mechanism such as S3, which will scale on its own, and disk storage limits won't apply.
Another option is to determine if the files could be stored in the database.
I hope that helps
Let's try to narrow the problem down.
Meteor UP is passing the configuration parameter volumes directly on to docker, as they also mention in the comment you included. It therefore might be easier to test it against docker directly - narrowing the components involved down as much as possible:
sudo docker run \
-it \
--rm \
-v "/host/path:/container/path" \
-v "/second/host/path:/second/container/path" \
busybox \
/bin/sh
Let me explain this:
sudo because Meteor UP uses sudo to start the container. See: https://github.com/zodern/meteor-up/blob/3c7120a75c12ea12fdd5688e33574c12e158fd07/src/plugins/meteor/assets/templates/start.sh#L63
docker run we want to start a container.
-it to access the container (think of it like SSH'ing into the container).
--rm to automatically clean up - remove the container - after we're done.
-v - here we give the volumes as you define it (I here took the two directories example you provided).
busybox - an image with some useful tools.
/bin/sh - the application to start the container with
I'd expect that you also cannot access the files here. In this case, dig deeper on why you can't make a folder accessible in Docker.
If you can, which would sound weird to me, you can start the container and try to access into the container by running the following command:
docker exec -it my-mup-container /bin/sh
You can think of this command like SSH'ing into a running container. Now you can check around if it really isn't there, if the credentials inside the container are correct, etc.
At last, I have to agree it #mikkel, that it's not a good option to mount a local directoy, but you can now start looking into how to use docker volume to mount a remote directory. He mentioned S3 by AWS, I've worked with AzureFiles on Azure, there are plenty of possibilities.
I use mlflow in a docker environment as described in this example and I start my runs with mlflow run ..
I get output like this
2019/07/17 16:08:16 INFO mlflow.projects: === Building docker image mlflow-myproject-ab8e0e4 ===
2019/07/17 16:08:18 INFO mlflow.projects: === Created directory /var/folders/93/xt2vz36s7jd1fh9bkhkk9sgc0000gn/T/tmp1lxyqqw9 for downloading remote URIs passed to arguments of type 'path' ===
2019/07/17 16:08:18 INFO mlflow.projects: === Running command 'docker run
--rm -v /Users/foo/bar/mlruns:/mlflow/tmp/mlruns -e
MLFLOW_RUN_ID=ef21de61d8a6436b97b643e5cee64ae1 -e MLFLOW_TRACKING_URI=file:///mlflow/tmp/mlruns -e MLFLOW_EXPERIMENT_ID=0 mlflow-myproject-ab8e0e4 python train.py' in run with ID 'ef21de61d8a6436b97b643e5cee64ae1' ===
I would like to mount a docker volume named my_docker_volume to the container
at
the path /data. So instead of the docker run shown above, I would like to
use
docker run --rm --mount source=my_docker_volume,target=/data -v /Users/foo/bar/mlruns:/mlflow/tmp/mlruns -e MLFLOW_RUN_ID=ef21de61d8a6436b97b643e5cee64ae1 -e MLFLOW_TRACKING_URI=file:///mlflow/tmp/mlruns -e MLFLOW_EXPERIMENT_ID=0 mlflow-myproject-ab8e0e4 python train.py
I see that I could in principle run it once without mounted volume and then
copy the docker run ... and add --mount source=my_volume,target=/data but
I'd rather use something like
mlflow run --mount source=my_docker_volume,target=/data .
but this obviously doesn't work because --mount is not a parameter for
mlflow run.
What's the recommened way of mounting a docker volume then?
A similar issue has been brought up on the mlflow issue tracker, see "Access large data from within a Docker environment". An excerpt from it says:
However, MLFlow Docker environments currently only have access to data baked into the repository or image or must download a large dataset for each run.
...
A potential solution is to enable the user to mount a volume (e.g. local directory containing the data) into the Docker container.
Looks like this is feature others would benefit from too. Best course of action here would be to contribute support for mounts, or keep track of the issue until someone else implements it.
Why do you need to mount /data folder in the first place? There's another issue, a PR containing a fix related to storing artifacts in a custom location on host machine, could it be something you're looking for?
Finally, to avoid the above problem and facilitate volume mounting, I now run my experiments using three interacting docker containers. One that runs the machine learning code, one that runs an mlflow server and one that runs a postgresql server. I closely followed this walk-through article to set things up. It works nicely and docker-compose makes volume mounting easy. Metrics, parameters and meta data are stored in a database that is mounted to a local persistent volume. Artifacts are logged in the directory /mlflow or if you prefer in a docker volume.
Note: There's a typo in the cited walk-through article
In docker-compose.yml it shouldn't be
volumes:
- ./postgres-store:/var/lib/postgresql/data
which would bind a local folder named postgres-store.
Instead, to mount the docker volume postgres_store, you should use
volumes:
- postgres-store:/var/lib/postgresql/data
Before anything, I have read this question and the related links in it, but I am still confused about how to resolve this on my setup.
I wrote my own docker file to install Archiva, which is very similar to this file. I created an image from the docker file using docker build -t archiva . and have a container which I run using docker run archiva. As seen in the docker file, the user data that I want to preserve is in a volume.
Now I want to upgrade to Archive 2.2.0. How can I update my container, so that the user-data thats in the volume is preserved? If I change the docker file by h=just changing the version number, and run the docker build again, it will just create another container.
Best practice
The option --volume of the docker-run enables sharing files between host and container(s) and especially preserve consistent [user] data.
The problem is ..
.. it appears that you are not using --volume and that the user data are in the image. (and that's a bad practice beacuse it leads to the situation you are in: unable to upgrade a service easily.
One solution (the best IMO) is
Back-up the user data
To use the command docker-cp: "Copy files/folders between a container and the local filesystem."
docker cp [--help] CONTAINER:SRC_PATH DEST_PATH
Upgrade your Dockerfile
By editing your Dockerfile and changing the version.
Use the --volume option
Use docker run -v /host/path/user-data:container/path/user-data archiva
And you're good!
What is the best way to persist containers data with docker? I would like to be able to retain some data and be able to get them back when restarting my container. I have read this interesting post but it does not exactly answer my question.
As far as I understand, I only have one option:
docker run -v /home/host/app:/home/container/app
This will mount the countainer folder onto the host.
Is there any other option? FYI, I don't use linking containers (--link )
Using volumes is the best way of handling data which you want to keep from a container. Using the -v flag works well and you shouldn't run into issues with this.
You can also use the VOLUME instruction in the Dockerfile which means you will not have to add any more options at run time, however they're quite tightly coupled with the specific container, you'd need to use docker start, rather than docker run to get the data back (or of course -v to the volume which was created in the past, likely in /var/ somewhere).
A common way of handling volumes is to create a data volume container with volumes defined by -v Then when you create your app container, use the --volumes-from flag. This will make your new container use the same volumes as the container you used the -v on (your data volume container). Of course this may seem like you're shifting the issue somewhere else.
This makes it quite simple to share volumes over multiple containers. Perhaps you have a container for your application, and another for logstash.
create a volume-container: this format of -v creates a volume, directory e.g. /var/lib/docker/volume/d3b0d5b781b7f92771b7342824c9f136c883af321a6e9fbe9740e18b93f29b69
which is still a bind mounted /container/path/vol
docker run -v /foo/bar/vol --name volbox ubuntu
I can now use this container, as my volume.
docker run --volumes-from volbox --name foobox ubuntu /bin/bash
root#foobox# ls /container/path/vol
Now, if I distribute these two containers, they will just work. The volume will always be available to foobox, regardless which host it is deployed to.
The snag of course comes if you don't want your storage to be in /var/lib/docker/volumes...
I suggest you take a look at some of the excellent post by Michael Crosby
https://docs.docker.com/userguide/dockervolumes/
and the docker docs
https://docs.docker.com/userguide/dockervolumes/