I thought I can use Bound volumes as suggested for my another post
Docker-compose mariadb external volume mapping issue
But I read that containers should be stateless. So it seems my thinking is wrong?
I do not know what catastrophic failures can occur, so is there a possibility that I may lose all data, if the container is bricked? or is there a way to use external storage and recover?
How to manage this situation? I have a schema table which manages migrations, so don't want that table to be new and start from square 1
Question: Should I let the mariadb container on cloud write to wherever it likes? or write to host folder?
My docker -compose snippet
mariadb:
image: mariadb:10.4
...
environment:
..
logging:
...
networks:
- backend
restart: on-failure
volumes:
- maria_volume:/var/lib/mysql
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci
# Volumes
volumes:
maria_volume:
Another version is (./mariadb instead of maria_volume in volumes section)
networks:
- backend
restart: on-failure
volumes:
- ./mariadb:/var/lib/mysql
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci
Your application at large needs to keep data somewhere. Having a relational-database container with storage mounted is fine. In a production environment you could choose to run a non-container database or use a cloud-hosted database if that met your needs better.
I feel like the actual storage mechanisms are pretty robust, both for named volumes and bind-mounted host directories. You probably will not have data-corruption problems in either case. As always, make sure you have backups of your data if it's at all important.
There's not a clear choice between using named volumes and host directories here. Host directories are probably easier to back up and restore; on some platforms named volumes will be faster. In both cases, in normal operation, the data will survive destroying and recreating the container. It'll be a little easier to destroy a named volume's state using docker commands, which depending on your specific use case could point in either direction.
It has occasionally happened to me that Docker's internal state gets corrupted, and when this happens the easiest workaround is to delete the entire /var/lib/docker tree and start over (there is an equivalent "reset" button in the Docker Desktop application). This path would lose named volumes too. On native Linux it's been widely observed that the actual named-volume storage is just a directory, so you might be able to preserve this.
Related
I have a composer file with four services. I need to OPEN one of them to outside by settings ports.
After changing .yml file, do I need to 'rebuild the container' (docker-compose down/up) or do I just need to stop/start? (docker-compose stop/start)?
Specifically, what I neet to do accesible to outside is a Posgree Server. This is my actual postgres service definition in .yml:
mydb:
image: postgres:9.4
environment:
- POSTGRES_PASSWORD=myPassword
volumes:
- db-data:/var/lib/postgresql/data
I think I just need to change it to:
mydb:
image: postgres:9.4
ports:
- "5432:5432"
environment:
- POSTGRES_PASSWORD=myPassword
volumes:
- db-data:/var/lib/postgresql/data
I'm worried of loosing data on 'db-data' volume, or connection to the other services, if I use down/up.
Also, there are 3 other services specified in the .yml file. If it is necessary to REBUILD (without loosing data in db-data, of course), I don't want to touch these other containers. In this case, which would be the steps?:
First, rebuild 'mydb' container with 'docker run' (Could you provide me the right command, please?)
Modify the .yml as stated before, just adding the ports
Perform a simple docker-compose stop/start
Could you help me, please?
If you're only changing settings like ports:, it is enough to re-run docker-compose up -d again. Compose will figure out which things are different from the existing containers, and destroy and recreate only those specific containers.
If you're changing a Dockerfile or your application code you may specifically need to docker-compose build your application or use docker-compose up -d --build. But you don't specifically need to rebuild the images if you're only changing runtime settings like ports:.
docker-compose down tears down your entire container stack. You don't need it for routine rebuilds or container updates. You may want to intentionally shut down the container system (and free up host ports, memory, and other resources) and it's useful then.
docker-compose stop leaves the containers in an unusual state of existing but without a running process. You almost never need this. docker-compose start restarts containers in this unusual state, and you also almost never need it.
You have to rebuild it.
For that reason the best practice is to map all the mount points and resources externally, so you can recreate the container (with changed parameters) without any loss of data.
In your scenario I see that you put all the data in an external docker volume, so I think you could recreate it with changed ports in a safe way.
If someone may know, does it need to be separate Dockerfile for a database and service itself in case if you want to run an application within Docker containers?
It's not quite clear where to specify the external database and server name, is it in the .env file?
https://github.com/gurock/testrail-docker/blob/master/README.md
http://docs.gurock.com/testrail-admin/installation-docker/migrating-upgrading-testrail
Yes, you should run both application and Database in a separate container.
It's not quite clear where to specify the external database and server
name, is it in the .env file?
You have two option to speicy Environment variable
.env file
Envrionment Variables
place the .env file in the root of your docker-compose and specify this in your docker-compose file.
services:
api:
image: 'node:6-alpine'
env_file:
- .env
Using Environment
environment:
MYSQL_USER: "${DB_USER:-testrail}"
MYSQL_PASSWORD: "${DB_PWD:-testrail}"
MYSQL_DATABASE: "${DB_NAME:-testrail}"
MYSQL_ROOT_PASSWORD: "${DB_ROOT_PWD:-my-secret-password}"
MYSQL_ALLOW_EMPTY_PASSWORD: 'false'
does it need to be separate Dockerfile for a database and service
Better to use offical database image, and for service, you can customize the image, but you provided link is the better choice for you to start with docker-compose.yml.
Also, the documentation of docker-compose is already given in the link.
Theoretically you can have an application and the database running in the same container, but this will have kinds of unintended consequences for example if the database falls over the application might still be running, but docker won't notice that the database fell over if it is not aware of it.
Something to wrap your mind around when running the database in a container is data persistence, which means that data would survive even when the container is killed or deleted and that once you create the container again the container would still be able to access the databases and other data.
Here is a good article explaining volumes in docker in the context of running mysql in its own container with a volume to hold the data:
https://severalnines.com/database-blog/mysql-docker-containers-understanding-basics
In context of the repo that you linked it seems there is a separate Dockerfile for the database and that you have the option to choose to use either Mariadb or MySQL, see here:
https://github.com/gurock/testrail-docker/tree/master/Dockerfiles/testrail_mariadb
and here:
https://github.com/gurock/testrail-docker/tree/master/Dockerfiles/testrail_mysql
What's the difference between declaring in the docker-compose.yml file a volume section and just using the volumes keyword under a service?
For example, I map a volume this way for a container:
services:
mysqldb:
volumes:
- ./data:/var/lib/mysql
This will map to the folder called data from my working directory.
But I could also map a volume by declaring a volume section and use its alias for the container:
services:
mysqldb:
volumes:
- data_volume:/var/lib/mysql
volumes:
data_volume:
driver: local
In this method, the actual location of where the mapped files are stored appears to be somewhat managed by docker compose.
What are the differences between these 2 methods or are they the same? Which one should I really use?
Are there any benefits of using one method over the other?
The difference between the methods you've described is that first method is a bind mount, and the other is a volume. These are more of Docker functions (rather than Docker Compose), and there are several benefits volumes provide over mounting a path from your host's filesystem. As described in the documentation, they:
are easier to back up or migrate
can be managed with docker volumes or the API (as opposed to the raw filesystem)
work on both Linux and Windows containers
can be safely shared among multiple containers
can have content pre-populated by a container (with bind mounts sometimes you have to copy data out, then restart the container)
Another massive benefit to using volumes are the volume drivers, which you'd specify in place of local. They allow you to store volumes remotely (i.e. cloud, etc) or add other features like encryption. This is core to the concept of containers, because if the running container is stateless and uses remote volumes, then you can move the container across hosts and it can be run without being reconfigured.
Therefore, the recommendation is to use Docker volumes. Another good example is the following:
services:
webserver_a:
volumes:
- ./serving/prod:/var/www
webserver_b:
volumes:
- ./serving/prod:/var/www
cache_server:
volumes:
- ./serving/prod:/cache_root
If you move the ./serving directory somewhere else, the bind mount breaks because it's a relative path. As you noted, volumes have aliases and have their path managed by Docker, so:
you wouldn't need to find and replace the path 3 times
the volume using local stores data somewhere else on your system and would continue mounting just fine
TL;DR: try and use volumes. They're portable, and encourage practices that reduce dependencies on your host machine.
I want to use Docker MySQL.
docker run mysql
But I don't want to save data on the host machine. I want all the information to be protected inside the container. By default, this image created an unnamed volume, and attach it to the container.
Is it possible, to use the same container, (I don't want to create a new MySQL image from ground), but disable the volume?
In other words: Many Docker images in docker hub are using volumes by default. What is the easiest way to save all the data inside the container (so push, and commit will contain the data)? There is a command to stop a container, change it's Mounts settings, and start again?
I know that it is not best practice, my question is if it is possible.
EDIT: There is a tool mentioned in the comments of the below thread that can edit docker image metadata, allowing you to remove a volume.
This is currently an open issue awaiting someone with the bandwidth to code it. You can track the progress here, with this link going directly to the applicable comment:
#veqryn since reopening this issue, nobody started working on a pull-request; the existing pull request did no longer apply cleanly on the code-base so a new one has to be opened; if anyone is interested in working on this, then things can get going again.
I too would like this feature! Mounting /var/lib/mysql/ on windows hosts with NTFS gives the volume root:root permissions which can't be chown'd; I don't want to add mysql user to the root group. I would like to UNVOLUME the /var/lib/mysql directory and replace it with a symlink that does have mysql:mysql permissions, pointed at /host/ntfs/mnt which is root:root 🤷♀️
As shown in this question, you can create, name and associate a container volume easily enough to the default unnamed one of mysql
version: '2'
services:
db:
image: mysql
volumes:
- dbdata:/var/lib/mysql
volumes:
dbdata:
driver: local
See "Mount a shared-storage volume as a data volume": you can uise other drivers, like flocker, and benefit from a multi-host portable volume.
I have the following example
version: '2'
services:
proxy:
container_name: proxy
hostname: proxy
image: nginx
ports:
- 80:80
- 443:443
volumes:
- proxy_conf:/etc/nginx
- proxy_htdocs:/usr/share/nginx/html
volumes:
proxy_conf: {}
proxy_htdocs: {}
which works fine. When I run docker-compose up it creates those named volumes in /var/lib/docker/volumes and all is good. However, from the host, I can only access /var/lib/docker as root, because it's root:root (makes sense). I was wondering if there is a way of chowning the host's directories to something more sensible/safe (like, my relatively unprivileged user that I use to do most things on the host) or if I just have to suck it up and chown them manually. I'm starting to have a number of scripts already to work around other issues, so having an extra couple of lines won't be much of a problem, but I'd really like to keep my self-written automation minimal, if I can -- fewer chances for stupid mistakes.
By the way, no: if I mount host directories instead of creating volumes, they get overlaid, meaning that if they start empty, they stay empty, and I don't get the default configuration (or whatever) from inside the container.
Extra points: can I just move the volumes to a more convenient location? Say, /home/myuser/myserverstuff/volumes?
It's best to not try to access files inside /var/lib/docker directly. Those directories are meant to be managed by the docker daemon, and not to be messed with.
To access the data inside a volume, there's a number of options;
use a bind-mounted directory (you considered that, but didn't fit your use case).
use a "service" container that uses the same volume and makes it accessible through that container, for example a container running ssh (to use scp) or a SAMBA container (such as svendowideit/samba)
use a volume-driver plugin. there's various plugins around that offer all kind of options. For example, the local persist plugin is a really simple plug-in that allows you to specify where docker should store the volume data (so outside of /var/lib/docker)