I have a Docker container that runs a Python app and generates a file as a result.
I would like to persist this file in my local file system and not to lose it at the end of the container's execution.
After reading other posts and documentation of volumes and data storage in Docker I have tried to solve it in different ways like:
Using a volume (created before):
docker run --name my_container -v my_volume:/container_file_path my_container
I'm missing something here because I understand that I'm not referencing the host's route at any time.
Or directly referencing host path and container path (with this option I also got some problems with absolute or relative paths usage):
docker run --name my_container -v host_path:container_file_path my_container
I also tried some other "variants" of the commands above (--mount instead of -v, changing target/source values, etc.) but I couldn't get it to work.
I'm using Windows Subsystem for Linux (WSL), which I've read may be the cause of the problem.
Could you guide me in what I am doing wrong? Thanks!
The second option with the bind-mount volume sounds good to me. You do need to use absolute paths, but you can use e.g. $(pwd) to make it simpler.
Related
I'm fairly new to using Docker and Docker Compose (using Docker Compose for this particular problem). Here is what I know so far about the problem I am facing: When using volumes when there are contents available in the host folder as well as the container's folder, the files inside the container's folder are hidden and the host's files are then made available to the container.
I want to use it the other way round. I would like to make available the container's files (that were copied into the image in the Dockerfile) to the host folder.
Is there a way to do that?
Here are a bunch of screenshots of my Dockerfile and Docker Compose to show my setup.
Dockerfile Screenshot
DockerCompose Screenshot
Thanks in advance! :)
I've come across the same thing many times and the way I go about it is as follows.
As the host volume will always take priority over the container filesystem, you have to copy the files out of the container to the host first, then volume mount them back - this way you get what was there originally, and also what might change in the future (by the container).
The following is all pseudo code, but should hopefully simulate the concept:
First run the main container:
docker run --rm -d --name my-container registry/image-name
Then copy the files you want from it to the local filesystem
docker cp my-container:/files/i/want ./files
Then stop the original container
docker stop my-container
Then mount them back into the container on the next run
docker run --rm -d --name my-container -v ./files:/files/i/want registry/image-name
Obviously you've mentioned compose there also, so just reflect the volume mapping into the compose format - the copy stuff will need to be done via standard docker however in line with the above.
Note: I wrote the above commands blind, but will check them over at lunch and correct any mistypes - but the concept is correct
Docker newcomer here.
I have a simple image of a django website with a volume defined for the app directory.
I can bind this volume to the actual folder where I do the development with this command :
docker container run --rm -p 8000:8000 --mount type=bind,src=$(pwd)/wordcount-project,target=/usr/src/app/wordcount-project wordcount-django
This works fairly well.
Now I tried to push that simple example in a swarm. Note that I have set up a local registry for the image to be available.
So to start my service I'd do :
docker service create -p 8000:8000 --mount type=bind,source=$(pwd)/wordcount-project,target=/usr/src/app/wordcount-project 127.0.0.1:5000/wordcount-django
It will work after some tries but only because it run on the local node (where the actual folder is) and not a remote node (where there is no wordcount-project folder).
Any idea how to solve this so that this folder can be accessible to all node and yet, still be accessible locally for development ?
Thanks !
Using bind-mount in docker swarn is not recommended, as you can read in the doc. In particular :
Important: Bind mounts can be useful but they can also cause problems. In most cases, it is recommended that you architect your application such that mounting paths from the host is unnecessary.
However, if you still want to use bind-mount, then you have two possibility :
Make sure your folder exists on all the nodes. The main problem here is that you'll have to update it everytime on every node.
Use a shared filesystem (such as sshfs for example) and mount it on a directory on each node. However, now that you have a shared filesystem, then you can just use a docker data volume and change the driver.
You can find some documentation on changing the volume data driver here
I have CoreOS running inside a Vagrant box. My working directory is /home/core.
I then run docker run -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home jenkins. The container starts without any problems.
Unfortunately I can't find the mounted volume inside my Vagrant box. I assumed it would be in /home/core/jenkins_home but it ain't there. /home/core/ is empty.
Where is this directory?
The advice to use an absolute host path for volumes is in the comments (hat tip #Hy L), but the question of where the data went with the relative path was still an open issue. I'm on WSL2 and I finally found my data at
/mnt/wsl/docker-desktop-data/data/docker
and I needed sudo to look inside the following directory to see data from my relative mount (pipeline_runs):
sudo ls volumes/pipeline_runs/_data
I don't have a full understanding but I take it that this is where the Hyper-V machine is storing its data, so when you specify a relative path, it's actually relative to a path on that (virtual) machine.
I was just going through this tutorial on Youtube, trying to understand the use of the -v command.
Why is the author using the -v command? He uses the command, like so:
docker run -v /var/lib/mysql --name=my_datastore -d busybox echo "my datastore"
Now I understand the above command to an extent:
--name=my_datastore gives the container a specific name.
-d busybox starts a container in detached mode, based on the busybox image.
After the command is executed the below line is echoed to the console.
my datastore
Now, the part I don't understand is the following:
-v /var/lib/mysql
Why is the -v command used here, and why the path /var/lib/mysql specified?
I am having difficulty understanding why the above line is used and in what context. Can anybody explain?
The -v (or --volume) argument to docker run is for creating storage space inside a container that is separate from the rest of the container filesystem. There are two forms of the command.
When given a single argument, like -v /var/lib/mysql, this allocates space from Docker and mounts it at the given location. This is primarily a way of allocating storage from Docker that is distinct from your service container. For example, you may want to run a newer version of a database application, which involves tearing down your existing MySQL container and starting a new one. You want your data to survive this process, so you store it in a volume that can be accessed by your database container.
When given two arguments (host_path:container_path), like -v /data/mysql:/var/lib/mysql, this mounts the specified directory on the host inside the container at the specified path (and, to be accurate, this can also be used to expose host files inside the container; for example -v /etc/localtime:/etc/localtime would make /etc/localtime on the host available as /etc/localtime inside the container). This is a way of either feeding information into your container, or providing a way for your container to make files accessible to the host.
If a container has volumes available, either through the use of the -v command line argument or via the VOLUME directive in a Dockerfile, these volumes can be accessed from another container using the --volumes-from option:
docker run --volumes-from my_datastore ...
This will make any volumes defined in the source container available in the container you're starting with --volumes-from.
This is discussed in more detail in the Docker Volumes documentation.
I have a Docker container which is running some code and creating some HTML reports. I want these reports to be published into a specific directory on the host machine, i.e. at /usr/share/nginx/reports
The way I have gone about doing this is to mount this host directory as a data volume, i.e. docker run -v /usr/share/nginx/reports --name my-container com.containers/my-container
However, when I ssh into the host machine, and check the contents of the directory /usr/share/nginx/reports, I don't see any of the report data there.
Am I doing something wrong?
The host machine is an Ubuntu server, and the Docker container is also Ubuntu, no boot2docker weirdness going on here.
From "Managing data in containers", mounting a host folder to a container would be:
docker run -v /Users/<path>:/<container path>
(see "Use volume")
Using only -v /usr/share/nginx/reports would declare the internal container path /usr/share/nginx/reports as a volume, but would have nothing to do with the host folder.
This is one of the type of mounts available:
The answer to this question is problematic because it varies depending on your operating system and your full requirements. The answer by VonC makes some assumptions that should be addressed and is therefore only correct in some contexts. Other answers on this topic generally ignore the fact that some people are running linux, others windows, and still others are on OSX or other weird OS's.
As VonC mentioned in his answer, in a lot of cases it is possible to bind-mount a host directory straight into the container, using a -v host-path:container-path argument to the docker command (you can also use --volume for added readability or --mount for rocket-science).
One of the biggest problems (in 2020) is the use of the Windows Subsystem for Linux (WSL), where bind-mounting a host volume is fraught with error and may or may not work as expected depending on whether the path mounted is in the linux filesystem or the windows filesystem. VonC's answer was written before WSL became a big problem, but it still makes assumptions about the local filesystem being real rather than mounted into a virtual-machine of some kind.
I have found that a lot of engineers prefer to bypass this unnecessary confusion through the use of docker volumes. A docker volume can be created with the command:
docker volume create <name>
Listed with
docker volume ls
and removed with
docker volume rm <name>
You can mount this by specifying the name of the volume on the left-hand-side of the --volume argument. If your volume was called, for example, 'logs', you could use something like --volume logs:/usr/share/nginx/reports to bind it to the log dir you're interested in. You can view the contents of the directory with something like this:
docker run -it --rm --volume logs:/logs alpine ls -AlF /logs/
This should list the files in that directory. If you have a file called 'nginx.log' for example, you could view it like this:
docker run -it --rm --volume logs:/logs alpine less /logs/nginx.log
And the contents would be paged to your terminal.
You can bind this volume to multiple containers simultaneously if needed. This is useful if, for example, you're writing to your logs with one container, and paging them to a console with another.
If you want to copy the example log file from above into a tmp directory on your local filesystem you can achieve that with:
docker run -it --rm --volume logs:/logs --volume /tmp:/local_tmp alpine cp /logs/nginx.log /local_tmp/
I am using Docker toolbox on windows. I am Working on a Spring Boot Application using Docker. My application writes logs to
users/path/service.log
So when i started my application from host terminal the Log file was successfully updated.
But the same when i did on docker no file was created and neither updated.
So i changed my log file location to match with the Container's Directories
var/log/service.log
I started my container again and my file was updated again.
You can choose any location as long as it matches with the container Directory. Just bash into the container and see what suits you.
Next step is to copy log files from container to host.
So in order to copy those logs to your host. You can use one of two ways i know of-
1- use Volumes in docker
2- use following Docker command to copy file from docker container to host-:
docker cp <containerId>:/file/path/within/container /host/path/target
First, you need to create a directory where you want to share the data
mkdir -p /abc/def/
Now, you need to create a docker volume using the below command. As we see here, we are specifying device as '/abc/def/'
docker volume create --driver local \
--opt type=none \
--opt device=/abc/def/ \
--opt o=bind \
spark-volume
Now, start your container with below command..
docker run -d \
--mount type=volume,dst=/abc/def/,volume-driver=local,volume-opt=type=none,volume-opt=o=bind,volume-opt=device=/opt/spark/ \
--network host \
img:tag
Now, docker container will use /abc/def/ in local Filesystem as its storage and you will have all contents of /abc/def/ in docker container available in Local Filesystem
In your application, if you set a working directory for your php code (report path), the path must be the one on the container. Then docker will copie automaticly copy to your host directory. It wasn't docker mis-configuration, but my application that was writing to the wrong place. Weird at first, but did work in my case.