I am in the process of building a simple web application using NodeJS that persists data to a MySQL database and saves images that have been uploaded to it. With my current setup, I have 4 Docker containers - 1 for the NodeJS application, 1 for the MySQL server, 1 Volume Container for the MySQL Data and 1 Volume container for the uploaded files.
What I would like to do is come up with a mechanism where I can periodically take backups of both volume containers automatically without stopping the web application.
Is it possible to do this and if so, what's the best way?
I have looked at the Docker Documentation on Volume management that covers backing up and restoring volumes, but I'm not sure that would work while the application is still writing data to the database or saving uploaded files.
To backup your database I suggest to use mysqldump, that is more safer then a simple file copy.
For volume backup you can also run a container, link the volume and tar the contents together.
In both cases you can use additional containers or process injection via docker exec.
Related
I am creating a core library for Blazor server Apps creating a core DB automatically at runtime.
Until now, I create the database in Environment.SpecialFolder.LocalApplicationData which I got working on multiple platforms (OSX, Ubunt and Windows).
As I just discovered Docker's simplicity for deploying images, I am trying to make my library compatible with it.
So I face two issues:
Determine if the app is hosted on a Docker image or not
Persist data on a different volume that is NOT on the host if running Docker.
Of course, if in Docker, I shall not use Environment.SpecialFolder.LocalApplicationData as this is not a persistent location on the image itself. I can mount a volume when starting the image as described here.
So my natural idea is to assume users will mount a volumne with a specific path when starting the image, say
docker volume create MyAppDB
and run it with
docker run -dp 3000:3000 -v MyAppDB:/app/Data/MyAppDB myBuildDockerImage
can be verified by testing the existance of folder /app/Data/MyAppDB, and once 1. is verified, 2. become trivial.
If the folder does not exist, I am for sure on non-docker image... Well am I? What if users forgot to mount the volume? Or misspelled it? Maybe the folder does not exist because I am running on a non-docker environment...!
Is there a way to tweak my docker image when building it to force the mount volumes - i.e. created by ME and not the end-user? That seems safest... Alternatively, if not possible, can I add some specific element in Docker image to make absolutely sure I am running on the docker image I built or not?
I have created a grafana docker image in aws fargate using aws ecs. The web app works well. However, I loose dashboards and user information anytime I restart the app. From my readings, this is because grafana image has no storage to keep information.
Following this link,https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-mount-efs-containers-tasks/, I have added an EFS volume to the container.
Volume configuration : Root directory = / (the default one).
On the container name: STORAGE AND LOGGING session:.
Mount points: I have added the volume name.
Container path: /usr/app.
However, I still loose all dashboards ans user information on container restart.
Is there something that I am missing ?
Thank you for your support
Instead of making container have persistent storage one alternative could be to have a custom entrypoint script that downloads the dashboards and put them in /etc/grafana/provisioning/dashboards/ (could be from s3) and then runs /run.sh this way you can keep your container stateless and not add any storage to it.
Configure custom database (e.g. MySQL, PostgreSQL - AWS RDS/Aurora) instead of default file based SQLite DB. RDBs are better for concurrent access, so you can scale out better.
AWS offers also options for backups (e.g. automated snapshosts), so I would say DB is better solution than FS volumes (+ problems with FS permissions).
I'm using named volumes to persist data on Host machine in the cloud.
I want to take backup of these volumes present in the docker environment so that I can reuse them on critical incidents.
Almost decided to write a python script to compress the specified directory on the host machine and push it to the AWS S3.
But I would like to know if there is any other approaches to this problem?
docker-volume-backup may be helpful. It allows you to back up your Docker volumes to an external location or to a S3 storage.
Why use a Docker container to back up a Docker volume instead of writing your own Python script? Ideally you don't want to make backups while the volume is being used, so having a container on your docker-compose able to properly stop your container before taking backups can effectively copy data without affecting the application performance or backup integrity.
There's also this alternative: volume-backup
I have a Python app using a SQLite database (it's a data collector that runs daily by cron). I want to deploy it, probably on AWS or Google Container Engine, using Docker. I see three main steps:
1. Containerize and test the app locally.
2. Deploy and run the app on AWS or GCE.
3. Backup the DB periodically and download back to a local archive.
Recent posts (on Docker, StackOverflow and elsewhere) say that since 1.9, Volumes are now the recommended way to handle persisted data, rather than the "data container" pattern. For future compatibility, I always like to use the preferred, idiomatic method, however Volumes seem to be much more of a challenge than data containers. Am I missing something??
Following the "data container" pattern, I can easily:
Build a base image with all the static program and config files.
From that image create a data container image and copy my DB and backup directory into it (simple COPY in the Dockerfile).
Push both images to Docker Hub.
Pull them down to AWS.
Run the data and base images, using "--volume-from" to refer to the data.
Using "docker volume create":
I'm unclear how to copy my DB into the volume.
I'm very unclear how to get that volume (containing the DB) up to AWS or GCE... you can't PUSH/PULL a volume.
Am I missing something regarding Volumes?
Is there a good overview of using Volumes to do what I want to do?
Is there a recommended, idiomatic way to backup and download data (either using the data container pattern or volumes) as per my step 3?
When you first use an empty named volume, it will receive a copy of the image's volume data where it's first used (unlike a host based volume that completely overlays the mount point with the host directory). So you can initialize the volume contents in your main image as a volume, upload that image to your registry and pull that image down to your target host, create a named volume on that host, point your image to that named volume (using docker-compose makes the last two steps easy, it's really 2 commands at most docker volume create <vol-name> and docker run -v <vol-name>:/mnt <image>), and it will be populated with your initial data.
Retrieving the data from a container based volume or a named volume is an identical process, you need to mount the volume in a container and run an export/backup to your outside location. The only difference is in the command line, instead of --volumes-from <container-id> you have -v <vol-name>:/mnt. You can use this same process to import data into the volume as well, removing the need to initialize the app image with data in it's volume.
The biggest advantage of the new process is that it clearly separates data from containers. You can purge all the containers on the system without fear of losing data, and any volumes listed on the system are clear in their name, rather than a randomly assigned name. Lastly, named volumes can be mounted anywhere on the target, and you can pick and choose which of the volumes you'd like to mount if you have multiple data sources (e.g. config files vs databases).
I am new to both docker and redis. I have configured serverA to run redis inside docker. The redis database has been pre-seeded with a thousand key/value pairs. I can confirm the data has been persisted in this container. I then created a new docker image from this container, uploaded it to my docker repository.
On serverB, I pulled the redis image "redis-preseeded" and got it started. Using the redis-cli tool when I connect and issue the 'info keyspace' command, the keyspace is empty, suggesting none of the data made it across. What am I doing wrong?
Are you is using the official image for Redis?
https://hub.docker.com/_/redis/
Dockefile for the redis:3.2.0-alpine release
It has a volume declared:
..
VOLUME /data
WORKDIR /data
..
Volumes are described in the documentation. In a nutshell what the authors are doing is configuring Redis to store data on an external disk volume that will be maintained by the docker engine. Not only is this more efficient it also allows users to keep the data separate to the container using it.
Labouring the point a snapshot of the image will contain no data (if you think about it this is a good thing).
Once you understand these concepts then it becomes more obvious how data can be moved between servers:
Backup, Restore and Migrate data volumes