I am using k8s with version 1.11 and CephFS as storage.
I am trying to mount the directory created on the CephFS in the pod. To achieve the same I have written the following volume and volume mount config
in the deployment configuration
Volume
{
"name": "cephfs-0",
"cephfs": {
"monitors": [
"10.0.1.165:6789",
"10.0.1.103:6789",
"10.0.1.222:6789"
],
"user": "cfs",
"secretRef": {
"name": "ceph-secret"
},
"readOnly": false,
"path": "/cfs/data/conf"
}
}
volumeMounts
{
"mountPath": "/opt/myapplication/conf",
"name": "cephfs-0",
"readOnly": false
}
Mount is working properly. I can see the ceph directory i.e. /cfs/data/conf getting mounted on /opt/myapplication/conf but following is my issue.
I have configuration files already present as a part of docker image at the location /opt/myapplication/conf. When deployment tries to mount the ceph volume then all the files at the location /opt/myapplication/conf gets disappear. I know it's the behavior of the mount operation but is there any way by which I would be able to persist the already existing files in the container on the volume which I am mounting so that other pod which is mounting the same volume can access the configuration files. i.e. the files which are already there inside the pod at the location /opt/myapplication/conf should be accessible on the CephFS at location /cfs/data/conf.
Is it possible?
I went through the docker document and it mentions that
Populate a volume using a container
If you start a container which creates a new volume, as above, and the container has files or directories in the directory to be mounted (such as /app/ above), the directory’s contents are copied into the volume. The container then mounts and uses the volume, and other containers which use the volume also have access to the pre-populated content.
This matches with my requirement but how to achieve it with k8s volumes?
Unfortunately Kubernetes' volume system is very different from Docker's so this is not possible directly. If there is a single file (or a small number) you can use subPath projection like this:
volumeMounts:
- name: cephfs-0
mountPath: /opt/myapplication/conf/foo.conf
subPath: foo.conf
Repeat that for each file. But if you have a lot of files, or if they can vary, then you have to handle this at runtime or use templating tools. Usually that means mounting it somewhere else and setting up symlinks before your main process starts.
Very easy ! you have to use init container here. With init container use the same deployment image of your application.
Assuming your container path is /opt/myapplication/conf
your init container will share the cephfs PVC
with init container define volume mount at /opt/data
in init container config run the command mv to move your existing data to mounted volume path /opt/data.
with main application container, mount the volume at correct location i.e. /opt/myapplication/conf
now when you deploy your application,
your init container mounts the cephfs pv and moves container path data to volume.
now your main application starts and mounts the volume on correct path , now the volume when mounting here have data also.
I was able to fix this by having my ENTRYPOINT be a bash script that mv my config files i wanted mounted to their correct location. It seems this device or resource is busy errors were happening because the files were not mounted yet.
I also encountered this very niche issue not being able to mount a folder to a specific path with content from my built image. This ends up empty.
However my workaround is to use ENTRYPOINT in de dockerfile refering to a shellscript that runs the commands to initialize DB or do something with files that affects the mounted target folder.
So it seems that entrypoint runs after the volume is being mounted by kubernetes.
I did tried to symlink the path in the entrypoint script, but that didn't work out.
Related
I have a docker container image that requires me to mount a volume containing a specific configuration file, in order for that container to properly start (this image is not one that I have control over, and is vendor supplied). If that volume is not mounted, the container will exit because the file is not found. So I need to put a configuration file in /host/folder/, and then:
docker run --name my_app -v /host/folder:/container/folder image_id
The application will then look in /container/folder/ for the file it needs to start.
I want to create/commit a new image with that file inside /container/folder/, but when that folder is mounted as volume from the host, docker cp will not help me do this, as far as I have tried. I think, as far as docker is concerned, the file copied there is no different than the files in the mounted volume, and will disappear when the container is stopped.
Part of the reason I want to do this, is because the file will not be changed, and should be there by default. The other reason is that I want to run this container in Kubernetes, and avoid using persistent volumes there to mount these directories. I have looked into using configmaps, but I'm not seeing how I can use those for this purpose.
If you can store the file into the ConfigMap you can mount the file to volume and use it inside the Kubernetes.
I am not sure with the type of file you have to use.
ConfigMap will inject this file into the volume of a POD so the application could access and use it.
In this case there will be no PVC required.
You can also follow this nice example showing how to mount the file into a volume inside a pod.
OR
Also, I am not sure about the docker image but if you can use that docker image you can add the file into the path, something like:
FROM <docker image>
ADD file ./container/folder/
In this case, you might have to check you can use the vendor docker image as a base and add the file into it.
docker run -ti --rm -v DataVolume3:/var ubuntu
Lets say I have a volume DataVolume 3 which pulls the contents of /var in the ubuntu container
even after killing this ubuntu container the volume remains and I can use this volume DataVolume3 to mount it to other containers.
This means with the deletion of container the volume mounts are not deleted.
How does this work ?
Does that volume mount mean that it copies the contents of /var into some local directory because this does not look like a symbolic link ?
If I have the container running and I create a file in the container then the same file gets copied to the host path ?
How does this whole process of volume mount from container to host and host to container work ?
Volumes are used for persistent storage and the volumes persists independent of the lifecycle of the container.
We can go through a demo to understand it clearly.
First, let's create a container using the named volumes approach as:
docker run -ti --rm -v DataVolume3:/var ubuntu
This will create a docker volume named DataVolume3 and it can be viewed in the output of docker volume ls:
docker volume ls
DRIVER VOLUME NAME
local DataVolume3
Docker stores the information about these named volumes in the directory /var/lib/docker/volumes/ (*):
ls /var/lib/docker/volumes/
1617af4bce3a647a0b93ed980d64d97746878564b141f30b6110d0818bf32b76 DataVolume3
Next, let's write some data from the ubuntu container at the mounted path var:
echo "hello" > var/file1
root#2b67a89a0050:/# cat /var/file1
hello
We can see this data with cat even after deleting the container:
cat /var/lib/docker/volumes/DataVolume3/_data/file1
hello
Note: Although, we are able to access the volumes like shown above but it not a recommended practice to access volumes data like this.
Now, next time when another container uses the same volume then the data from the volume gets mounted at the container directory specified as part of -v flag.
(*) The location may vary based on OS as pointed by David and probably can be seen by the docker volume inspect command.
Docker has a concept of a named volume. By default the storage for this lives somewhere on your host system and you can't directly access it from outside Docker (*). A named volume has its own lifecycle, it can be independently docker volume rm'd, and if you start another container mounting the same volume, it will have the same persistent content.
The docker run -v option takes some unit of storage, either a named volume or a specific host directory, and mounts it (as in the mount(8) command) in a specific place in the container filesystem. This will hide what was originally in the image and replace it with the volume content.
As you note, if the thing you mount is an empty named volume, it will get populated from the image content at container initialization time. There are some really important caveats on this functionality:
Named volume initialization happens only if the volume is totally empty.
The contents of the named volume never automatically update.
If the volume isn't empty, the volume contents completely replace what's in the image, even if it's changed.
The initialization happens only on native Docker, and not for example in Kubernetes.
The initialization happens only on named volumes, and not for bind-mounted host directories.
With all of these caveats, I'd avoid relying on this functionality.
If you need to mount a volume into a container, assume it will be empty when your entrypoint or the main container command starts. If you need a particular directory layout or file structure there, an entrypoint script can create it; if you're expecting it to hold particular data, keep a copy of it somewhere else in your image and copy it in if it's not already there (or, perhaps, always).
(*) On native Linux you can find a filesystem location for it, but accessing this isn't a best practice. On other OSes this will be hidden inside a virtual machine or other opaque storage. If you need to directly access the data (or inject config files, or read log files) a docker run -v /host/path:/container/path bind mount is a better choice.
Volumes are part of neither the container nor the host. Well, technically everything resides in the host machine. But the docker directories are only accessible by users in "docker" group. The files in these directories are separately managed by docker.
"Volumes are stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux)."
Hence volumes are like the union of files under the docker container and the host itself. Any addition on either end will be added to the volume(/var/lib/docker/volumes), not hard copy, rather something like symbol link
As volumes can be shared across different containers, deleting a container does not cascade to the volumes associated with it.
To remove unused volumes:
docker volume prune .
I have the following in my docker-compose.yml file:
volumes:
- .:/var/www/app
- my_modules:/var/www/app/node_modules
I do this because I don't have node_modules on my host and everything gets installed in the image and is located at /var/www/app/node_modules.
I have two questions regarding this:
An empty folder gets created in my host named node_modules. If I add another volume (named or anonymous) to the list in my .yml file, it'll show up in my host directory in the same folder that contains my .yml file. From this answer, it seems to have to do with the fact that there's these two mappings going on simultaneously. However, why is the folder empty on my host? Shouldn't it either a) contain the files from the named volume or b) not show up at all on the host?
How does Docker know to check the underlying /var/www/app/node_modules from the image when initializing the volume rather than just saying "Oh, node_modules doesn't exist" (since I'm assuming the host bind mount happens before the named volume gets initialized, hence /var/www/app should no longer have a folder named node_modules. It seems like it even works when I create a sample node_modules folder on my host and a new volume while keeping my_modules:/var/www/app/node_modules—it seems to still use the node_modules from the image rather than from the host (which is not what I expected, although not unwanted).
As an implementation detail, Docker actually uses the Linux kernel filesystem mount facility whenever it mounts a volume. To mount a volume it has to be mounted on to a directory, so if the mount target doesn't already exist, it creates a new empty directory to be the mount point. If the mount point is itself inside a mounted volume, you'll see the empty directory get created, but the mount won't get echoed out.
(If you're on a Linux host, try running mount in a shell while the container is running.)
That is:
/container_root/app is a bind mount to /host_path/app; they are they same underlying files.
mkdir /container_root/app/node_modules creates /host_path/app/node_modules too.
Mounting something else on /container_root/app/node_modules doesn't cause anything to be mounted on /host_path/app/node_modules.
...which leaves an empty /host_path/app/node_modules directory.
The first time you start a container, and only then, if you mount an empty volume into a container, the contents from the image get copied into the volume. You're telling Docker this directory contains critical data that needs to be persisted for longer than the lifespan of the container. It is not a magic "don't use the host directory volume" knob, and if you do things like change your package.json file, Docker will not update the contents of this volume.
This experiment tries to build a container using this Docker file:
FROM lambdalinux/baseimage-amzn:2016.09-000
COPY ./bundle /opt/bundle/
VOLUME /bundle
Then inside the container, create a /opt/bundle/file.txt and put some text in it. But that file did not show up in the bundle directory on the host as I expected after reading Should I include my code with COPY/ADD or a volume last paragraph:
There may be cases where you’ll want to use both. You can have the image include the code using a COPY, and use a volume in your Compose file to include the code from the host during development. The volume overrides the directory contents of the image.
Doesn't Dockerfile VOLUME do the same as docker-compose.yml VOLUME? If so, how can this be done so that changes in the host directory is reflected inside the container directory in this case?
I also created a file on the host bundle/play.txt but that did not show up inside the container /opt/bundle/...
A VOLUME instruction in the dockerfile creates a mount point but initially only maps it to Docker's internal data directory.
In order to map the volume to the host filesystem, you need to specify which path on the host should be mapped to the volume. You can do this in the docker-compose file using the volumes parameter. (Note: you can create volumes using docker-compose without declaring them in the Dockerfile.)
Note that when mapping a directory from the host to a container, the directory contents on the host will replace the contents in the container, not vice versa.
I've searched the docs but nothing came up so time to test it. But for a quick future reference...
Is the host folder populated with the container folder contents?
Is it the opposite?
Are both folder contents merged? (In that case: What happens when a file with the same name is in both folders?)
Or does it produce an error? Is the error thrown on launch or is it thrown when you try to build an image with a VOLUME pointing to an existing populated folder on the container?
Also, another thing that isn't in the docs: Do I have to define the container path as a VOLUME in the Dockerfile in order to use -v against it when launching the container or can I create volumes on the fly?
When you run a container and mount a volume from the host, all you see in the container is what is on the host - the volume mount points at the host directory, so if there was anything in the directory in the image it gets bypassed.
With an image from this Dockerfile:
FROM ubuntu
WORKDIR /vol
RUN touch /vol/from-container
VOLUME /vol
When you run it without a host mount, the image contents get copied into the volume:
> docker run vol-test ls /vol
from-container
But mount the volume from the host and you only see the host's content:
> ls $(pwd)/host
from-host
> docker run -v $(pwd)/host:/vol vol-test ls /vol
from-host
And no, you don't need the VOLUME instruction. The behaviour is the same without it.
Whenever a Docker container is created with a volume mounted on the host, e.g.:
docker run -v /path/on/host:/data container-image
Any contents that are already in /data due to the image build process are always completely discarded, and whatever is currently at /path/on/host is used in its place. (If /path/on/host does not exist, it is created as an empty directory, though I think some aspect of that behavior may currently be deprecated.)
Pre-defining a volume in the Dockerfile with VOLUME is not necessary; all VOLUME does is cause any containers run from the image to have an implicit -v /volume/path (Note lack of host mount path) argument added to their docker run command which is ignored if an explicit -v /host/path:/volume/path is used.