Docker commit is not saving my changes to image - docker

I'm new to docker world: I'm at a point where i can deploy docker containers and do some work.
Trying to get to the next level of saving my changes and moving my containers/images to another pc/server.
Currently, I'm using docker on windows 10, but I do have access to Ubuntu 16.04 server to test my work.
This is where I'm stuck: I have Wordpress and MariaDB images deployed on Docker.
My WP is running perfectly OK.I have installed few themes and created few pages with images.
At this point, I like to save my work and send it to my friend who will deploy my image and do further work on this same Wordpress.
What I have read online is: I should run docker commit command to save and create my docker image in .tar format and then send this image file (.tar) to my friend. He will run docker load -i on my file to load it as image into his docker and then create container from it which should give him all of my work on Wordpress.
Just to clarify, I'm committing both Wordpress and Mariadb containers.
I don't have any external volumes mounted so all the work is being saved in containers.
I do remember putting check mark on drive C and D in docker settings but i don't know if that has anything to to do with volumes.
I don't get any error in my commit and moving .tar files process. Once my friend create his containers from my committed images, he gets clean Wordpress (like new installation of Wordpress starting from wp setup pages).
Another thing I noticed is that the image I create has the same file size as original image i pulled. When I run docker images, I see my image is 420MB ,as well as Wordpress image is 420MB.
I think my image should be a little bit bigger since I have installed themes, plugins and uploaded images to Wordpress. At least it should add 3 to 5 MB more then original images. Please help. Thank you.
Running docker system df gives me this.
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 5 3 1.259GB 785.9MB (62%)
Containers 3 3 58.96kB 0B (0%)
Local Volumes 2 2 311.4MB 0B (0%)
Build Cache 0 0 0B 0B

Make sure, as shown here, to commit a running container (to avoid any data cleanup)
docker commit CONTAINER_ID yourImage
After the docker commit command, you can use docker save to save your image in a tar, and docker load to import it back, as shown here.

You should never run docker commit.
To answer your immediate question, containers that run databases generally store their data in volumes; they are set up so that the data is stored in an anonymous volume even if there was no docker run -v option given to explicitly store data in a named volume or host directory. That means that docker commit never persists the data in a database, and you need some other mechanism to copy the actual data around.
At a more practical level, your colleague can ask questions like "where did this 400 MB tarball come from, why should I trust it, and how can I recreate it if it gets damaged in transit?" There are also good questions like "the underlying database has a security fix I need, so how do I get the changes I made on top of a newer base image?" If you're diligent you can write down everything you do in a text file. If you then have a text file that says "I started from mysql:5.6, then I ran ..." that's very close to being a Dockerfile. The syntax is straightforward, and Docker has a good tutorial on building and running custom images.
When you need a custom image, you should always describe what goes into it using a Dockerfile, which can be checked into source control, and can rebuild an image using docker build.
For your use case it doesn't sound like you actually need a custom image. I would probably suggest setting up a Docker Compose YAML file that described your setup and actually stored the data in local directories. The database half of it might look like
version: '3'
services:
db:
image: 'mysql:8.0'
volumes:
- './mysql:/var/lib/mysql/data'
ports:
- '3306:3306'
The data will be stored on the host, in a mysql subdirectory. Now you can tar up this directory tree and send that tar file to your colleague, who can then untar it and recreate the same environment with its associated data.

Use docker build (Changes to the images should be stored in the Dockerfile).
Now if you have multiple services, just use docker's brother docker-compose. One extra step you have to do is create docker-compose.yml (don't be afraid yet my friend, it's nothing trivial). All you're doing in this file is listing out your images (along with defining where their Dockerfile is for that image, could be in some subfolder for each image). You can also define some other properties there if you'd like.

Notice that certain directories are considered volume directories by docker, meaning that they are container specific and therefore never saved in the image. The \data directory is such an example. When docker commit my_container my_image:my_tag is executed, all of the containers filesystem is saved, except for /data. To work around it, you could do:
mkdir /data0
cp /data/* /data0
Then, outside the container:
docker commit my_container my_image:my_tag
Then you would perhaps want to copy the data on /data0 back to /data, in which case you could make a new image:
On the Dockerfile:
FROM my_image:my_tag
CMD "cp /data0 /data && my_other_CMD"
Notice that trying to copy content to /data in a RUN command will not work, since a new container is created in every layer and, in each of them, the contents of /data are discarded. After the container has been instatiated, you could also do:
docker exec -d my_container /bin/bash -c "cp /data0/* /data"

You have to use the volumes to store your data.
Here you can find the documentation: https://docs.docker.com/storage/volumes/
For example you can do somethink like this in your docker-compose.yml.
version: '3.1'
services:
wordpress:
image: wordpress:php7.2-apache
ports:
- "8080:80"
environment:
WORDPRESS_DB_HOST: databasename
WORDPRESS_DB_USER: username
WORDPRESS_DB_PASSWORD: password
WORDPRESS_DB_NAME: namedatabase
volumes:
- name_volume:/var/www/html
volumes:
- name_volume:
or
volumes:
- ./yourpath:/var/www/html

Related

Why does a file within a docker volume not get overwritten?

I'm trying to understand volumes.
When I build and run this image with docker build -t myserver . and docker run -dp 8080:80 myserver, the web server on it prints "Hallo". When I change "Hallo" to "Huhu" in the Dockerfile and rebuild & run the image/container, it shows "Huhu". So far, no surprises.
Next, I added a docker-compose.yaml file that has two volumes. One volume is mounted on an existing path of where the Dockerfile creates the index.html. The other is mounted on a new and unused path. I build and run everything with docker compose up --build.
On the first build, the web server prints "Hallo" as expected. I can also see the two volumes in Docker GUI and its contents. The index.html that was written to the image, is now present in the volume. (I guess the volume gets mounted before the Dockerfile can write to it.)
On the second build (swap "Hallo" with "huhu" and run docker compose up --build again) I was expecting the webserver to print "Huhu". But it prints "Hallo". So I'm not sure why the data on the volume was not overwritten by the Dockerfile.
Can you explain?
Here are the files:
Dockerfile
FROM nginx
# First build
RUN echo "Hallo" > /usr/share/nginx/html/index.html
# Second build
# RUN echo "Huhu" > /usr/share/nginx/html/index.html
docker-compose.yaml
services:
web:
build: .
ports:
- "8080:80"
volumes:
- html:/usr/share/nginx/html
- persistent:/persistent
volumes:
html:
persistent:
There are three different cases here:
When you build the image, it knows nothing about volumes. Whatever string is in that RUN echo line, it is stored in the image. Volumes are not mounted when you run the docker-compose build step, and the Dockerfile cannot write to a volume at all.
The first time you run a container with the volume mounted, and the first time only, if the volume is empty, Docker copies content from the mount point in the image into the volume. This only happens with named volumes and not bind mounts; it only happens on native Docker and not Kubernetes; the volume content is never updated at all after this happens.
The second time you run a container with the volume mounted, since the volume is already populated, the content from the volume hides the content in the image.
You routinely see various cases that uses named volumes to "pass through" to the image (especially Node applications) or to "share files" with another container (frequently an Nginx server). These only work because Docker (and only Docker) automatically populates empty named volumes, and therefore they only work the first time. If you change your package.json, your Node application that mounts a volume over node_modules won't see updates; if you change your static assets that you're sharing with a Web server, the named volume will hide those changes in both the application and HTTP-server containers.
Since the named-volume auto-copy only happens in this one very specific case, I'd try to avoid using it, and more generally try to avoid mounting anything over non-empty directories in your image.

Create Docker instance .tar including volumes

I have a local Gitlab docker image running and added a bunch of projects. This project/repository data seems to end up inside of the 3 volumes that have been created by the image.
I want to create a single .tar of the Gitlab instance which includes the complete image + all data found in the volumes. It's okay that the .tar becomes huge.
I tried to accomplish this by using docker commit and docker save but I have been unable to save the volumes along with the image.
How can I create such a single .tar export.
If I was going to set this up, I'd have a single Docker Compose file that contained all of the relevant pieces in a single directory tree.
version: '3'
services:
db:
image: 'postgres:11'
volumes:
- './postgres:/var/lib/postgresql/data'
gitlab:
image: gitlab-community-edition
# Details below here made up
ports:
- 8080:80
env:
PGHOST: db
volumes:
- './gitlab:/data'
The important thing here is that every piece of persisted data is in the same directory tree on the host. The actual contents of the container filesystem aren't important (every piece of persisted data is in these directories) and the images aren't important (they can be pulled from Docker Hub).
You may use docker cp to achieve this.
docker cp my_container:/path/to/gitlab /destination/folder
and then tar the contents of the destination folder.
You’ll need to use docker export to create the tar of the image. Docker import then to install it back. Essentially you’re looking for an installer from the sounds of it. You can write a bash script which copies whatever files you need and exports the images.

Wordpress image with mysql data

Is there any image available that contain wordpress along with mysql data?
When I commit and take backup of the image, mysql data is not included. I will prefer a single image for both.
I tried to create such image using this Dockerfile:
FROM tutum/lamp:latest
RUN rm -fr /app && git clone https://github.com/WordPress/WordPress.git /app
EXPOSE 80
CMD ["/run.sh"]
I can initiate a fresh installation using a command like this...
docker run -p 88:80 shantanuo/wp
But the container can not be moved to another server "as is". I need to take data backup using mysql-dump command and that is something I am trying to avoid. Is it possible?
If I do not volumanize the container, then I am able to copy the wordpress image along with it's data.
https://hub.docker.com/r/shantanuo/lamp/~/dockerfile/
But it does not work on the new server. Adding wordpress tag.
Is there any image available that contain wordpress along with mysql data?
Short answer: not recommended.
An image usually deals with one service (so two images would be involved here: wordpress and MySQL)
And the persistent data would not be "in" the image, but on the host in a volume / bind mount.
For instance, the tutumcloud/lamp image does declare volumes:
# Add volumes for MySQL
VOLUME ["/etc/mysql", "/var/lib/mysql" ]
The docker run command initializes the newly created volume with any data that exists at the specified location within the base image.
Making your own image without those lines might work as you expect (ie, commit a container with its data).
But if the server reboot at any time, or you have to docker run your original container again, it will start anew, without the data.
A typical docker wordpress image would use a mysql one
version: '3.1'
services:
wordpress:
image: wordpress
restart: always
ports:
- 8080:80
environment:
WORDPRESS_DB_PASSWORD: example
mysql:
image: mysql:5.7
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
And in turn, that mysql container would use a local host mounted volume in order to persists the database.
docker run --name some-mysql -v /my/own/datadir:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag
See for instance "Quickstart: Compose and WordPress"
So not only should you commit your Wordpress image, but your Mysql one as well, and your volume.
However, committing a volume is not supported: see "Commit content of mounted volumes as well" in order to backup that volume with your WordPress database in it.
With those three backups, you then migrate them to your other server.
However, this seems overly complex, and a fresh WordPress/MySQL docker project on the second server is easier to start.
You would then need, yes, your database dump file.
And some other Wordpress folders (like themes)
See "Easy WordPress Migration with Docker".
That would be the recommended way over trying to commit existing containers form one server and "transplant" them onto another server.
If you want to export your workig dataset to another server, docker has the commit command. This command creates a new image from a running container.
$ docker commit c3f279d17e0a svendowideit/testimage:version3
Documentation.

Overwrite files with `docker run`

Maybe I'm missing this when reading the docs, but is there a way to overwrite files on the container's file system when issuing a docker run command?
Something akin to the Dockerfile COPY command? The key desire here is to be able to take a particular Docker image, and spin several of the same image up, but with different configuration files. (I'd prefer to do this with environment variables, but the application that I'm Dockerizing is not partial to that.)
You have a few options. Using something like docker-compose, you could automatically build a unique image for each container using your base image as a template. For example, if you had a docker-compose.yml that look liked:
container0:
build: container0
container1:
build: container1
And then inside container0/Dockerfile you had:
FROM larsks/thttpd
COPY index.html /index.html
And inside container0/index.html you had whatever content you
wanted, then running docker-compose build would generate unique
images for each entry (and running docker-compose up would start
everything up).
I've put together an example of the above
here.
Using just the Docker command line, you can use host volume mounts,
which allow you to mount files into a container as well as
directories. Using my thttpd as an example again, you could use the
following -v argument to override /index.html in the container
with the content of your choice:
docker run -v index.html:/index.html larsks/thttpd
And you could accomplish the same thing with docker-compose via the
volume entry:
container0:
image: larsks/thttpd
volumes:
- ./container0/index.html:/index.html
container1:
image: larsks/thttpd
volumes:
- ./container1/index.html:/index.html
I would suggest that using the build mechanism makes more sense if you are trying to override many files, while using volumes is fine for one or two files.
A key difference between the two mechanisms is that when building images, each container will have a copy of the files, while using volume mounts, changes made to the file within the image will be reflected on the host filesystem.

docker postgres with initial data is not persisted over commits

I created a rails app in a docker environment and it links to a postgres instance. I edited the
postgres container to add initial data (by running rake db:setup from the rails app). Now I commited the postgres database, but it doesn't seem to remember my data when I create a new container (of the commited postgres image).
Isn't it possible to save data in a commit and then reuse it afterwards?
I used the postgres image: https://registry.hub.docker.com/_/postgres/
The problem is that the postgres Dockerfile declares "/var/lib/postgresql/data" as a volume. This is a just a normal directory that lives outside of the Union File System used by images. Volumes live until no containers link to them and they are explicitly deleted.
You have a few choices:
Use the --volumes-from command to share data with new containers. This will only work if there is only one running postgres image at a time, but it is the best solution.
Write your own Dockerfile which creates the data before declaring the volume. This data will then be copied into the volume when the container is created.
Write an entrypoint or cmd script which populates the database at run time.
All of these suggestions require you to use Volumes to manage the data once the container is running. Alternatively, you could write your own Dockerfile and simply not declare a volume. You could then use docker commit to create a new image after adding data. This will probably work in the short term, but is definitely not how you should work with containers - it isn't repeatable and you will eventually run out of layers in the Union File System.
Have a look at the official Docker docs on managing data in containers for more info.
Create a new Dockerfile and change PGDATA:
FROM postgres:9.2.10
RUN mkdir -p /var/lib/postgresql-static/data
ENV PGDATA /var/lib/postgresql-static/data
You should be all set with the following command. The most important part is the PGDATA location, which should be anything but the default.
docker run -e PGDATA=/var/lib/postgresql/pgdata -e POSTGRES_PASSWORD=YourPa$$W0rd -d postgres
It is not possible to save data during a commit since the data resides on a mount which is specific for that container and will get removed once you run docker rm <container ID> but you can use data volumes to share and reuse data between container and the changes made are directly on the volume.
You can use docker run -v /host/path:/Container/path to mount the volume to the new container.
Please refer to: https://docs.docker.com/userguide/dockervolumes/
For keeping permanent data such as databases, you should define these data volumes as external, therefore it will not be removed or created automatically every time you run docker-compose up or down commands, or redeploy your stack to the swarm.
...
volumes:
db-data:
external: true
...
then you should create this volume:
docker volume create db-data
and use it as data volume for your databse:
...
db:
image: postgres:latest
volumes:
- db-data:/var/lib/postgresql/data
ports:
- 5432:5432
...
In production, there are many factors to consider when using docker for keeping permanent data safely, specially in swarm mode, or in kubernetes cluster.

Resources