Docker volumes newbie questions - docker

For a service I've defined a volume as (an extract of my yml file)
services:
wordpress:
volumes:
- wp_data:/var/www/html
networks:
- wpsite
networks:
wpsite:
volumes:
wp_data:
driver: local
I'm aware on a Windows 10 filesystem that the WP volumes won't be readily visible to me as they'll exist within the linux VM. Alternatively I'd have to provide a path argument to be able view my WP installation e.g.
volumes:
- ./mysql:/var/lib/mysql
But my question is what is the point of the 'driver: local' option, is this default. I've tried with & without this option and can't see the difference.
Secondly what does this do? In my yml file I've commented it out to no ill effect that I can see!?
networks:
wpsite:

First question:
The --driver or -d option defaults to local. driver: local is redundant. On Windows, the local driver does not support any options. If you were running docker on a Linux machine, you would've had some options: Official documentation here - https://docs.docker.com/engine/reference/commandline/volume_create/
Second question:
In each section networks:/volumes:/services: you basically declare the resources you need for your deployment.
In other words, creating an analogy with a virtual machine, you can think about it like this: you need to create a virtual disk named wp_data and a virtual network named wpsite.
Then, you want your wordpress service, to mount the the wp_data disk under /var/www/html and to connect to the wpsite subnet.
You can use the following docker commands to display the resources that are created behind the scenes by your compose file:
docker ps - show containers
docker volume ls - show docker volumes
docker network ls - show docker networks
Hint: once you created a network or a volume, unless you manually delete it, it will not be destroyed automatically. You can clean-up manually the resources and experiment yourself by removing/adding more resources from your compose file.
Updated to answer question in comment:
If you run your docker on a Windows host, you probably enabled hyper-v. This allowed Windows to create a Linux VM, on top of which your docker engine is running.
With the docker engine installed, docker can then create "virtual resources" such as virtual networks, virtual disks(volumes), containers(people often compare containers to VMs), services, containers etc.
Let's look at the following section from your compose file:
volumes:
wp_data:
driver: local
This will create a virtual disk managed by docker, named wp_data. The volume is not created directly on your Windows host file system, but instead it is being created inside the Linux VM that is running on top of the HyperV that is running on your Windows host. If you want to know precisely where, you can either execute docker inspect <containerID> and look for the mounts that you have on that container, or docker volume ls then docker volume inspect <volumeID> and look for the key "Mountpoint" to get the actual location.

Related

Remote & Local Volume Paths on Docker Machine

I have three nodes with docker in swarm mode. I deploy from my machine using contexts to target remote host.
I have the following docker-compose.yml:
version: "3.9"
services:
...
nginx:
image: 'nginx:1.23.3-alpine'
ports:
- 8080:80
volumes:
- ./conf:/etc/nginx/conf.d
depends_on:
- ui
...
How can I deliver the ./conf directory to one of the docker hosts? I found an outdated and inconvenient way, but there are more recent solutions (declare directly in the docker-compose.yml)?
The simple answer is, you don't. Docker Compose does not support this directly.
However, there are options that involve varying amounts of refactoring of your deployment process and they include:
Create a root relative conf folder (/mnt/conf etc) on the remote server, and reference that. Deploy files via some other process (scp etc.)
Create a "conf" volume remotely, and populate it using a docker image that you build that carries the files. (There is a syntax to mount a filesystem from another container, IDK if its compose compatible, but you could just mount a container that you build with the contents of ./conf You will need a registry to store the image so you can build it locally, but reference it remotely. registry:2 is easy to deploy)
If "conf" contains 1 to a few files, then enable swarm mode remotely, and mount the individual files as docker configs. This means using docker -c remote stack deploy rather than docker -c remote compose up.
Make "conf" shared on a nfs server, and declare the volume using docker local volume drivers option that supports nfs (or other fstab compatible) mounts. Alternative put the files in a s3 bucket (AWS or using a product like minio) and use the same syntax to use the "s3fs" fuse driver (if you don't use a containerised fuse driver, the fs driver will need to be installed on the remote host)
Use an actual docker volume plugin (e.g. https://rclone.org/docker/) to mount a wide variety of network shares into a compose or swarm service.

Docker bind propagation mount error "is not a shared mount"

I am trying to mount a FUSE virtual filesystem from inside a Docker container and expose the mount point to the host.
Docker is installed via snap on Ubuntu 20.04
The software is a fresh install of Seafile (a Dropbox alternative), but this problem I believe is more related to Docker, snap, and mounting file systems on Ubuntu. For what it's worth, I was following the official instructions here.
Inside the container (when it runs successfully), a script mounts a FUSE virtual filesystem to /seafile-fuse that makes the all files stored within Seafile visible.
docker-compose.yml exerpt:
version: '3.3'
services:
seafile:
image: seafileltd/seafile-mc:latest
container_name: seafile
volumes:
- /home/jonathan/seafile/seafile-data:/shared
- type: bind
source: /home/jonathan/seafile/seafile-fuse
target: /seafile-fuse
bind:
propagation: rshared
privileged: true
cap_add:
- SYS_ADMIN
This leads to:
ERROR: for seafile Cannot start service seafile: path /home/jonathan/seafile/seafile-fuse is mounted on /home but it is not a shared mount
I found this somewhat related answer which hints that the issue may to do with the docker daemon running in a different namespace. But I am unable to get his solution to work.
What do I need to do to connect the host directory /home/jonathan/seafile/seafile-fuse so that it sees the container directory /seafile-fuse?
Bonus question...
Given that this is to be an internet facing home-server, is it necessary this this becomes a privileged container? Are there better options?
Thanks!!

How to avoid the "Docker cannot link to a non running container" error when the external-linked container is actually running using docker-compose

What we want to do:
We want to use docker-compose to link one already running container (A) to another container (B) by container name. We use "external-link" as both containers are started from different docker-compose.yml files.
Problem:
Container B fails to start with the error although a container with that name is running.
ERROR: for container_b Cannot start service container_b: Cannot link to a non running container: /PREVIOUSLY_LINKED_ID_container_a_1 AS /container_b_1/container_a_1
output of "docker ps":
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
RUNNING_ID container_a "/docker-entrypoint.s" 15 minutes ago Up 15 minutes 5432/tcp container_a_1
Sample code:
docker-compose.yml of Container B:
container_b:
external_links:
- container_a_1
What differs this question from the other "how to fix"-questions:
we can't use "sudo service docker restart" (which works) as this is a production environment
We don't want to fix this every time manually but find the reason so that we can
understand what we are doing wrong
understand how to avoid this
Assumptions:
It seems like two instances of the container_a exist (RUNNING_ID and PREVIOUSLY_LINKED_ID)
This might happen because we
rebuilt the container via docker-compose build and
changed the forwarded external port of the container (80801:8080)
Comment
Do not use docker-compose down as suggested in the comments, this removes volumnes!
Docker links are deprecated so unless you need some functionality they provide or are on an extremely old version of docker, I'd recommend switching to docker networks.
Since the containers you want to connect appear to be started in separate compose files, you would create that network externally:
docker network create app_net
Then in your docker-compose.yml files, you connect your containers to that network:
version: '3'
networks:
app_net:
external:
name: app_net
services:
container_a:
# ...
networks:
- app_net
Then in your container_b, you would connect to container_a as "container_a", not "container_a_1".
As an aside, docker-compose down is not documented to remove volumes unless you pass the -v flag. Perhaps you are using anonymous volumes, in which case I'm not sure that docker-compose up would know where to find your data. A named volume is preferred. More than likely, your data was not being stored in a volume, which is dangerous and removes your ability to update your containers:
$ docker-compose down --help
By default, the only things removed are:
- Containers for services defined in the Compose file
- Networks defined in the `networks` section of the Compose file
- The default network, if one is used
Networks and volumes defined as `external` are never removed.
Usage: down [options]
Options:
--rmi type Remove images. Type must be one of:
'all': Remove all images used by any service.
'local': Remove only images that don't have a custom tag
set by the `image` field.
-v, --volumes Remove named volumes declared in the `volumes` section
of the Compose file and anonymous volumes
attached to containers.
--remove-orphans Remove containers for services not defined in the
Compose file

Docker volumes mounting on Windows 8 is not working

Context
I want to run a Docker Compose application on a Windows 8. I made it under a Ubuntu 16.04 and it's perfectly working on it.
This Docker Compose run:
nginx
php-fpm
The two containers use volumes.
Files
My .env file:
COMPOSE_CONVERT_WINDOWS_PATHS=1
APPLICATION_PATH=//C/Users/my_user/Documents/Development/my_application
My docker-compose.yml file:
version: '2'
services:
web:
build: ../application-web/
ports:
- "80:80"
tty: true
# Add a volume to link php code on the host and inside the container
volumes:
- ${APPLICATION_PATH}:/usr/share/nginx/html/application
- ${APPLICATION_PATH}/docker_files/docker-assistant:/usr/share/nginx/html/assistant
# Add hostnames to allow devs to call special url to open sites
extra_hosts:
- "localhost:127.0.0.1"
- "assistant.docker:127.0.0.1"
- "application.dev:127.0.0.1"
depends_on:
- custom-php
links:
- custom-php:custom-php
custom-php:
build: ../application-php/
ports:
- "50:50"
volumes:
- ${APPLICATION_PATH}:/usr/share/nginx/html/application
- ${APPLICATION_PATH}/docker_files/docker-assistant:/usr/share/nginx/html/assistant
Problem
When I run docker-compose up, everything goes well. Containers start.
But when I try to reach http://192.168.99.100 in my web browser, I got a 403 error.
My investigations show that there is no mounted volumes in the nginx and the php containers:
docker exec -it compose_web_1 bash
ls -la /usr/share/nginx/html/assistant/
shows
drwxr.xr.x 2 root root 80 May 18 15:30 .
drwxr.xr.x 2 root root 4096 May 18 16:10 ..
It seems that Docker cannot mount volumes. Why?
Other information
I am using the Docker Toolbox: https://www.docker.com/products/docker-toolbox
I know that's the good IP address because when I try to reach it in my web browser, I see my nginx container displaying logs.
The environment variable APPLICATION_PATH set as //C:/Users/my_user/Documents/Development/my_application cannot work because Docker use the ":" character as separator for volume declaration:
ERROR: Volume //C:/Users/my_user/Documents/Development/my_application://C:/Users/my_user/Documents/Development/my_application has incorrect format, should be external:internal[:mode]
It's not a nginx problem because when I create an index.phtml file in the folder, I am able to run it:
<?php
echo 'Hello world!';
Ok, I finally did it!
TL;DR
Follow those instructions to be able to access C:\ inside your containers.
1. Install the Docker Toolbox
Go get it here: https://www.docker.com/products/docker-toolbox
Install it.
2. Run a Hello world
Open a Docker Quickstart Terminal.
Run in it:
docker run hello-world
3. Share C:\ with Docker
Open Virtualbox
Open configuration of the default virtual machine and go to shared folders
Modify or create a new shared folder by clicking on buttons to the right. Set options to:
C:\
C
Auto mount
Permanent configuration
Then validate.
4. Activate sharing
Shutdown the default virtual machine then restart it.
5. Set your paths
e.G. if you have a .env file:
COMPOSE_CONVERT_WINDOWS_PATHS=1
APPLICATION_PATH=//C/path_from_C_to_the_folder_you_want_to_share_on_the_volume
/!\ you need to set COMPOSE_CONVERT_WINDOWS_PATHS to 1!
6. Start your Compose
In the Docker Quickstart Terminal:
Go to your Docker Compose folder, then start it:
cd /path_to_your_compose_folder
docker-compose up
Why have I to do that? It's so complicated!
The Docker technology rely on Linux namespaces. Without Linux, it can't work. To allow use of Docker on a Windows, Docker needs to install a Linux virtual machine. All the containers will run inside it.
The default virtual machine is now created and running within Virtualbox, that's why you have to share your folders using Virtualbox.
After sharing, the default virtual machine will have a mounted folder in it with a custom name (in the above example, it's C but it could be elephant or whatever).
Finally, Docker will mount volumes from the default virtual machine to the container: you have to use the name of the default machine shared folder in your volume declaration (in the above example, it's C but it could be elephant or whatever).

Empty directory when mounting volume using windows for docker

I am attempting to mount a volume from C:/Users into a container running on a docker-machine using the hyperv driver on docker for windows (win 10 pro). I am a using the lastest docker (1.13.1) and the same on the hyper vm machine. I have tried switching to using a local account, shared the drive in the docker settings menu and ive pretty much tried everything i could find on google.
Running the test volume run command in the settings menu works for me. At this point in time I presume hyperv does not support mounting volumes from the host however i cant find anywhere that explicitly says that volumes mounting will not work in hyperv.
This is my docker-compose config:
networks: {}
services:
app:
build:
context: C:\users\deep\projects\chat\app
command: sleep 3600
image: app
links:
- rethinkdb
- redis
ports:
- 4005:4005
- 4007:4007
volumes:
- /c/users/deep/projects/chat/app:/usr/src/app:rw
redis:
image: redis
rethinkdb:
image: rethinkdb:2.3.5
version: '2.0'
volumes: {}
In my Dockerfile i can see copy files into the container to usr/src/app. When i up the services with the volume specified in the compose file the directory is emptied, however i omit this volume mount i can see my files that i copied into the container from the dockerfile.
Running verbose when starting my services i can see a volumes path specified as such 'Binds': [u'/c/users/deep/projects/chat/app:/usr/src/app:rw']. However, when i inspect the container using docker-compose inspect app i see volumes set to null "Volumes": null.
I presume at this point that mounting volumes into a container running inside a hyperv VM is not supported? Can someone confirm so that I can RIP :)
I think you just need to share the volume (c:) of the folder from the Docker app settings.
See the "Shared Drives" paragraph from the getting started guide
I'm using mounted folders with a similar configuration and it works fine once the drive has been shared.
As stupid as it seems, this happens to me often. The solution is to un-check the C drive in "Docker for windows" - > Setting - > Shared Drives, apply and check it again with apply.
You should use /c/Users, with a capital "C".

Resources