I am not sure if this is a right question to ask. In the tutorial of Docker compose, https://docs.docker.com/compose/gettingstarted/#step-5-edit-the-compose-file-to-add-a-bind-mount, there is a volume key in the docker-compose.yml:
version: '3'
services:
web:
build: .
ports:
- "5000:5000"
volumes:
- .:/code
redis:
image: "redis:alpine"
And according to the tutorial, the volume key mounts the local file to the remote, and therefore we can change the code on the fly without restarting the Docker. My question is what internet protocol is used behind to transfer the updated code file.
Furthermore, I guess there would be more framework having this feature. What are the common protocols behind and why?
The tutorial doesn't say "the volume key mounts the local file to the remote". It says:
...in your project directory to add a bind mount for the web service:
[...]
The new volumes key mounts the project directory (current directory) on the host to /code inside the container, allowing you to modify the code on the fly, without having to rebuild the image.
If you click on the bind mount link, it will take you to
documentation that should answer all of your questions.
Briefly, a bind mount is way of making one directory on your system
appear in another location. For example, if I were run:
mkdir /tmp/newetc
mount -o bind /etc /tmp/newetc
Then running ls /tmp/newetc would show the same contents as /etc.
Docker uses this feature to expose host directories inside your
containers.
A bind mount only works on the same host; it cannot be used to expose
files on your local system to a remote system. It is a kernel feature and there are no internet protocols involved.
Related
Supposed I have a Docker container and a folder on my host /hostFolder. Now if I want to add this folder to the Docker container as a volume, then I can do this either by using ADD in the Dockerfile or mounting it as a volume.
So far, so good.
Now /hostFolder contains a sub-folder, /hostFolder/subFolder.
I want to mount /hostFolder into the Docker container (whether as read-write or read-only does not matter, works both for me), but I do NOT want to have it included /hostFolder/subFolder. I want to exclude this, and I also want the Docker container be able to make changes to this sub-folder, without the consequence of having it changed on the host as well.
Is this possible? If so, how?
Using docker-compose I'm able to use node_modules locally, but ignore it in the docker container using the following syntax in the docker-compose.yml
volumes:
- './angularApp:/opt/app'
- /opt/app/node_modules/
So everything in ./angularApp is mapped to /opt/app and then I create another mount volume /opt/app/node_modules/ which is now empty directory - even if in my local machine ./angularApp/node_modules is not empty.
If you want to have subdirectories ignored by docker-compose but persistent, you can do the following in docker-compose.yml:
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules
This will mount your current directory as a shared volume, but mount a persistent docker volume in place of your local node_modules directory. This is similar to the answer by #kernix, but this will allow node_modules to persist between docker-compose up runs, which is likely the desired behavior.
For those trying to get a nice workflow going where node_modules isn't overridden by local this might help.
Change your docker-compose to mount an anonymous persistent volume to node_modules to prevent your local overriding it. This has been outlined in this thread a few times.
services:
server:
build: .
volumes:
- .:/app
- /app/node_modules
This is the important bit we were missing. When spinning up your stack use docker-compose -V. Without this if you added a new package and rebuilt your image it would be using the node_modules from your initial docker-compose launch.
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
To exclude a file, use the following
volumes:
- /hostFolder:/folder
- /dev/null:/folder/fileToBeExcluded
With the docker command line:
docker run \
--mount type=bind,src=/hostFolder,dst=/containerFolder \
--mount type=volume,dst=/containerFolder/subFolder \
...other-args...
The -v option may also be used (credit to Bogdan Mart), but --mount is clearer and recommended.
First, using the ADD instruction in a Dockerfile is very different from using a volume (either via the -v argument to docker run or the VOLUME instruction in a Dockerfile). The ADD and COPY commands just take a copy of the files at the time docker build is run. These files are not updated until a fresh image is created with the docker build command. By contrast, using a volume is essentially saying "this directory should not be stored in the container image; instead use a directory on the host"; whenever a file inside a volume is changed, both the host and container will see it immediately.
I don't believe you can achieve what you want using volumes, you'll have to rethink your directory structure if you want to do this.
However, it's quite simple to achieve using COPY (which should be preferred to ADD). You can either use a .dockerignore file to exclude the subdirectory, or you could COPY all the files then do a RUN rm bla to remove the subdirectory.
Remember that any files you add to image with COPY or ADD must be inside the build context i.e. in or below the directory you run docker build from.
for the people who also had the issue that the node_modules folder would still overwrite from your local system and the other way around
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules/
This is the solution, With the trailing / after the node_modules being the fix.
Looks like the old solution doesn't work anymore(at least for me).
Creating an empty folder and mapping target folder to it helped though.
volumes:
- ./angularApp:/opt/app
- .empty:/opt/app/node_modules/
I found this link which saved me: Working with docker bind mounts and node_modules.
This working solution will create a "exclude" named volume in docker volumes manager. The volume name "exclude" is arbitrary, so you can use a custom name for the volume intead exclude.
services:
node:
command: nodemon index.js
volumes:
- ./:/usr/local/app/
# the volume above prevents our host system's node_modules to be mounted
- exclude:/usr/local/app/node_modules/
volumes:
exclude:
You can see more infos about volumes in Official docs - Use a volume with docker compose
To exclude a mounted file contained in the volume of your machine, you will have to overwrite it by allocating a volume to this same file.
In your config file:
services:
server:
build : ./Dockerfile
volumes:
- .:/app
An example in you dockerfile:
# Image Location
FROM node:13.12.0-buster
VOLUME /app/you_overwrite_file
I have the following docker-compose:
version: '3.7'
services:
db:
image: bitnami/mongodb:5.0.6
volumes:
- "/app/local-data:/data/db"
env_file: ./db/.env
The problem is data does not persist between docker-compose up/down and docker does not seem to use /app/local-data even though it creates it.
When I run docker-compose, container starts and works naturally. The directory /app/local-data is created by docker, however Mongodb does not populate it, and no r/w error is being shown on console. This makes me thing a temporary volume is assigned to container instead.. But if that is true then why docker still creates /app/local-data and not using it?
Any ideas how can I debug this?
Docker directives like volumes: don't know anything about what's actually running in the image. That directive creates the specified host and container paths if required, and bind-mounts the host path into the container path. It's up to the application code to use that directory (or not).
If you look at the bitnami/mongodb Docker Hub page under "Persisting your database", the database is configured to store data in the /bitnami/mongodb directory inside the container, and that directory needs to be the second volumes: path. Also note the requirement that the data directory needs to be writable by user ID 1001, which may or may not exist on your host (there's no specific requirement to create it).
volumes:
- "/app/local-data:/bitnami/mongodb"
# ^^^^^^^^^^^^^^^^
sudo chown -R 1001 /app/local-data
sudo docker-compose up -d
I have a very simple nextjs application where I have two folders which I like to map to the host (developer system) while I deploy this application inside docker (I use docker-desktop).
Data folder (It has some json files and also some nested folders and files)
Public folder (It has nested folders too but It contains images)
I have tested in local and also inside the docker container (without volume and all) - It's all functioning.
As a next step - I want to use the Volume with my docker-compose file so that, I can bind these directories inside the container with the source (and going forward with AKS file storage options).
I have tried with multiple approaches (also checked some of the answers in stackoverflow) but it does not help to achieve the same result.
Here is my docker-compose file for your reference.
version: '3.4'
services:
portfolio:
image: ${DOCKER_REGISTRY-}brij1111-portfolio
build:
context: ./APP-03/clientapp
dockerfile: dockerfile
volumes:
- /app/node_modules
# anonymous volume only for node_modules
- portfolio_data:/app/data
# named volume inside which the nextjs app writes content to the file
volumes:
portfolio_data:
driver: local
driver_opts:
o: bind
type: none
device: /APP-03/clientapp/data
# I have tried here to give a full path like /mnt/c/work/.../APP-03/clientapp/data but that also not working.
using docker-desktop I can see the volume indeed created for the container and it has all the files. However, It does not get reflected in my source if anything is updated inside that volume (like I add some content through nextjs application to that file it does not get reflected inside the running container).
in case, someone wants to know my folder hierarchy and where am i running docker-compose file, here is that reference image.
I had a similar problem installing Gitea on my NAS until someone (thankfully) told me a way to compromise (ie. your data will be persistent, but not in the location of your choosing).
version: '3.4'
volumes:
portfolio_data: {}
services:
portfolio:
image: ${DOCKER_REGISTRY-}brij1111-portfolio
build:
context: ./APP-03/clientapp
dockerfile: dockerfile
volumes:
- /app/node_modules
# anonymous volume only for node_modules
- portfolio_data:/app/data
In my articular case, I had to access my NAS using terminal, to the location which the container image is located, and search from there.
Hope it helps you
I am writing a docker-compose.yaml file for my project. I have checked the volumes documentation here .
I also understand the concept of volume in docker that I can mount a volume e.g. -v my-data/:/var/lib/db where my-data/ is a directory on my host machine while /var/lib/db is the path inside database container.
My confuse is with the link I put above. There it has the following sample:
version: "3.9"
services:
db:
image: db
volumes:
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
- data-volume:/var/lib/backup/data
volumes:
data-volume:
I wonder does it mean that I have to create a directory named data-volume on my host machine? What if I have a directory on my machine with path temp/my-data/ and I want to mount that path to the database container /var/lib/db ? Should I do something like below?
version: "3.9"
services:
db:
image: db
volumes:
- temp/my-data/:/var/lib/db
volumes:
temp/my-data/:
My main confusion is the volumes: section at the bottom, I am not sure whether the volume name should be the path of my directory or should be just literally a name I give & if it is the latter case then how could the given name be mapped with temp/my-data/ on my machine? The sample doesn't indicate that & is ambiguous to clarify that.
Could someone please clarify it for me?
P.S. I tried with above docker-compose I guessed, ended up with the error:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
volumes value 'temp/my-data/' does not match any of the regexes: '^[a-zA-Z0-9._-]+$'
Mapped volumes can either be files/directories on the host machine (sometimes called bind mounts in the documentation) or they can be docker volumes that can be managed using docker volume commands.
The volumes: section in a docker-compose file specify docker volumes, i.e. not files/directories. The first docker-compose in your post uses such a volume.
If you want to map a file or directory (like in your last docker-compose file), you don't need to specify anything in the volumes: section.
Docker volumes (the ones specified in the volumes: section or created using docker volume create) are of course also stored somewhere on your host computer, but docker manages that and you shouldn't normally need to know where or what the format is.
This part of the documentation is pretty good about explaining it, I think https://docs.docker.com/storage/volumes/
As #HansKilian mentions, you don't need both volumes and services.volumes. To use services.volumes, map the host directory to the container directory like this:
services:
db:
image: db
volumes:
- /host/path/lib/db:/container/path/lib/db
With that, the directory /host/path/lib/db on the host machine will be used by the container and available at /container/path/lib/db.
Now, if you're like me, I get really confused with fake examples, so let's say the real directory on your host machine is /var/lib/db and you just want to see it at /db when you run a shell in Docker (i.e., docker exec -it /bin/bash container-id).
docker-compose.yaml would look like this:
services:
db:
image: db
volumes:
- /var/lib/db:/db
Now when you run the shell, cd /logs and ls, you'll see the same results as if you'd cd /var/lib/db on the host.
If you want to use the volumes section to indicate a global volume to use, you first have to create that volume using docker volume create. The documentation Hans linked includes steps to do this. The syntax of /host/path:/container/path is replaced by volume-name:/container/path. Then, once defined, you'd alter your docker-compose.yaml to be more like this:
services:
db:
image: db
volumes:
- your-global-volume-name:/db
volumes:
your-global-volume-name:
external: true
Note that I have not tested or used the this configuration. I'm assuming it's correct based on the other method working and the few changes I can identify in the docs.
I want to add the bind mount to docker file just like I initialise a volume inside Dockefile.
Is there any way for it?
A Dockerfile defines how an image is built, not how it's used - so you can't specify the bind mount in a Dockerfile. Try using docker-compose instead. A simple docker-compose.yml that mounts a directory for you would look like this:
version: '3.1'
services:
mycontainer:
image: myimage
build: .
volumes:
- './path/on/docker/host:/path/inside/container'
The build: . is optional if you're building the image by some other means, but sometimes it's handy to do it all in one.
Run this with docker-compose up -d
In addition to what the other answers say:
Because bind mounts provide access to the host filesystem, allowing them to be embedded into an image would be a huge security risk. Consider an image that purports to be, say, a web server, but in fact bind mounts your /etc/passwd and /etc/shadow and then sends them off to a remote server.
Or one that bind mounts /lib/ld-linux.so and then overwrites it, thus breaking your entire system.
For these reasons, you cannot embed a a bind mount in your Dockerfile. Similarly, you cannot specify host port mappings, host device access, or any other similar attributes in the Dockerfile.
Simple answer is no.
A basic design principle for docker images is portablility. Bind mounts are hosts specific since the mounted folder is defined on the host machine. Thus this contradicts with the portablility requirement for Docker images.