I have a local Gitlab docker image running and added a bunch of projects. This project/repository data seems to end up inside of the 3 volumes that have been created by the image.
I want to create a single .tar of the Gitlab instance which includes the complete image + all data found in the volumes. It's okay that the .tar becomes huge.
I tried to accomplish this by using docker commit and docker save but I have been unable to save the volumes along with the image.
How can I create such a single .tar export.
If I was going to set this up, I'd have a single Docker Compose file that contained all of the relevant pieces in a single directory tree.
version: '3'
services:
db:
image: 'postgres:11'
volumes:
- './postgres:/var/lib/postgresql/data'
gitlab:
image: gitlab-community-edition
# Details below here made up
ports:
- 8080:80
env:
PGHOST: db
volumes:
- './gitlab:/data'
The important thing here is that every piece of persisted data is in the same directory tree on the host. The actual contents of the container filesystem aren't important (every piece of persisted data is in these directories) and the images aren't important (they can be pulled from Docker Hub).
You may use docker cp to achieve this.
docker cp my_container:/path/to/gitlab /destination/folder
and then tar the contents of the destination folder.
You’ll need to use docker export to create the tar of the image. Docker import then to install it back. Essentially you’re looking for an installer from the sounds of it. You can write a bash script which copies whatever files you need and exports the images.
Related
I pull an image from Docker Hub (say) Ghost CMS and after reading the documentation, I see that the default mount point is /var/lib/ghost/content
Now, when I make my own application with Ghost as the base image, I map some folder (say) CMS-Content and mount it on /var/lib/ghost/content written like this -
volumes:
- CMS-Content: /var/lib/ghost/content
The path /var/lib/ghost/content are System Level paths. However, CMS-Content is a folder I created to host my files (persistent data).
Finally, I decide to publish my application as an image in Docker Hub, so what will be the mount point now?
If you want to make a pesistent data for the container :
Using command-line :
docker run -it --name <WHATEVER> -p <LOCAL_PORT>:<CONTAINER_PORT> -v <LOCAL_PATH>:<CONTAINER_PATH> -d <IMAGE>:<TAG>
Using docker-compose.yaml :
version: '2'
services:
cms:
image: <IMAGE>:<TAG>
ports:
- <LOCAL_PORT>:<CONTAINER_PORT>
volumes:
- <LOCAL_PATH>:<CONTAINER_PATH>
Assume :
IMAGE: ghost-cms
TAG: latest
LOCAL_PORT: 8080
CONTAINER_PORT: 8080
LOCAL_PATH: /persistent-volume
CONTAINER_PATH: /var/lib/ghost/content
Examples :
First create /persistent-volume.
$ mkdir -p /persistent-volume
docker-compose -f docker-compose.yaml up -d
version: '2'
services:
cms:
image: ghost-cms:latest
ports:
- 8080:8080
volumes:
- /persistent-volume:/var/lib/ghost/content
Each container has its own isolated filesystem. Whoever writes an image's Dockerfile gets to decide what filesystem paths it uses, but since it's isolated from other containers and the host it's very normal to use "system" paths. As another example, the standard Docker Hub database images use /var/lib/mysql and /var/lib/postgresql/data. For a custom application you might choose to use /app/data or even just /data, if that makes sense for you.
If you're creating an image FROM a pre-existing image like this, you'll usually inherit its filesystem layout, so the mount point for your custom image would be the same as in the base image.
Flipping through the Ghost Tutorials, it looks like most things you could want to do either involve using the admin UI or making manual changes in the content directory. The only files that changes are in the CMS-Content named volume in your example (and even if you didn't name a volume, the Docker Hub ghost image specifies an anonymous volume there). That means you can't create a derived image with a standard theme or other similar setup: you can't change the content directory in a derived image, and if you experiment with docker commit (not recommended) the image it produces won't have the content from the volume.
So I just need a simply thing which is copying a script whose relative path (from the working dir.) is influxdb/init.iql (InfluxDB initialization script) to the path /docker-entrypoint-initdb.d/ which is a way to initialize an InfluxDB database according to InfluxDB Docker image doc:
Initialization Files
If the Docker image finds any files with the extensions .sh or .iql inside of the /docker-entrypoint-initdb.d folder, it will execute them
Right now, my docker-compose.yml is:
version: "3.3"
services:
influxdb:
image: influxdb:latest
container_name: influxdb
ports:
- "8087:8086"
volumes:
- influxdb-data:/var/lib/influxdb
volumes:
influxdb-data:
I need the script to be copied before the image gets built as if it finds the script in the specified path, it will execute it when building the image.
How can I do this? I thought about implementing a Makefile but I would rather prefer to use Docker to accomplish this if it is possible to not add an unnecessary extra piece to the project.
Thanks in advance.
The docker-compose file tells how to RUN an image, not how to BUILD it. These two are completely separate concepts. Also I'm not sure what are you trying to do. If you need to initialize your container with data, just mount a script (or an iql file) to the /docker-entrypoint-initdb.d location inside volumes of the docker-compose file, eg.:
volumes:
- influxdb-data:/var/lib/influxdb
- project_import.sh:/docker-entrypoint-initdb.d/import.sh:ro
The script(s) (or iql file(s)) will be executed when the container starts, not when the image is built. If you don't believe me check out the entrypoint script of the image to see how this process works.
Just remember that those scripts will get executed each time the container starts.
I'm new to docker world: I'm at a point where i can deploy docker containers and do some work.
Trying to get to the next level of saving my changes and moving my containers/images to another pc/server.
Currently, I'm using docker on windows 10, but I do have access to Ubuntu 16.04 server to test my work.
This is where I'm stuck: I have Wordpress and MariaDB images deployed on Docker.
My WP is running perfectly OK.I have installed few themes and created few pages with images.
At this point, I like to save my work and send it to my friend who will deploy my image and do further work on this same Wordpress.
What I have read online is: I should run docker commit command to save and create my docker image in .tar format and then send this image file (.tar) to my friend. He will run docker load -i on my file to load it as image into his docker and then create container from it which should give him all of my work on Wordpress.
Just to clarify, I'm committing both Wordpress and Mariadb containers.
I don't have any external volumes mounted so all the work is being saved in containers.
I do remember putting check mark on drive C and D in docker settings but i don't know if that has anything to to do with volumes.
I don't get any error in my commit and moving .tar files process. Once my friend create his containers from my committed images, he gets clean Wordpress (like new installation of Wordpress starting from wp setup pages).
Another thing I noticed is that the image I create has the same file size as original image i pulled. When I run docker images, I see my image is 420MB ,as well as Wordpress image is 420MB.
I think my image should be a little bit bigger since I have installed themes, plugins and uploaded images to Wordpress. At least it should add 3 to 5 MB more then original images. Please help. Thank you.
Running docker system df gives me this.
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 5 3 1.259GB 785.9MB (62%)
Containers 3 3 58.96kB 0B (0%)
Local Volumes 2 2 311.4MB 0B (0%)
Build Cache 0 0 0B 0B
Make sure, as shown here, to commit a running container (to avoid any data cleanup)
docker commit CONTAINER_ID yourImage
After the docker commit command, you can use docker save to save your image in a tar, and docker load to import it back, as shown here.
You should never run docker commit.
To answer your immediate question, containers that run databases generally store their data in volumes; they are set up so that the data is stored in an anonymous volume even if there was no docker run -v option given to explicitly store data in a named volume or host directory. That means that docker commit never persists the data in a database, and you need some other mechanism to copy the actual data around.
At a more practical level, your colleague can ask questions like "where did this 400 MB tarball come from, why should I trust it, and how can I recreate it if it gets damaged in transit?" There are also good questions like "the underlying database has a security fix I need, so how do I get the changes I made on top of a newer base image?" If you're diligent you can write down everything you do in a text file. If you then have a text file that says "I started from mysql:5.6, then I ran ..." that's very close to being a Dockerfile. The syntax is straightforward, and Docker has a good tutorial on building and running custom images.
When you need a custom image, you should always describe what goes into it using a Dockerfile, which can be checked into source control, and can rebuild an image using docker build.
For your use case it doesn't sound like you actually need a custom image. I would probably suggest setting up a Docker Compose YAML file that described your setup and actually stored the data in local directories. The database half of it might look like
version: '3'
services:
db:
image: 'mysql:8.0'
volumes:
- './mysql:/var/lib/mysql/data'
ports:
- '3306:3306'
The data will be stored on the host, in a mysql subdirectory. Now you can tar up this directory tree and send that tar file to your colleague, who can then untar it and recreate the same environment with its associated data.
Use docker build (Changes to the images should be stored in the Dockerfile).
Now if you have multiple services, just use docker's brother docker-compose. One extra step you have to do is create docker-compose.yml (don't be afraid yet my friend, it's nothing trivial). All you're doing in this file is listing out your images (along with defining where their Dockerfile is for that image, could be in some subfolder for each image). You can also define some other properties there if you'd like.
Notice that certain directories are considered volume directories by docker, meaning that they are container specific and therefore never saved in the image. The \data directory is such an example. When docker commit my_container my_image:my_tag is executed, all of the containers filesystem is saved, except for /data. To work around it, you could do:
mkdir /data0
cp /data/* /data0
Then, outside the container:
docker commit my_container my_image:my_tag
Then you would perhaps want to copy the data on /data0 back to /data, in which case you could make a new image:
On the Dockerfile:
FROM my_image:my_tag
CMD "cp /data0 /data && my_other_CMD"
Notice that trying to copy content to /data in a RUN command will not work, since a new container is created in every layer and, in each of them, the contents of /data are discarded. After the container has been instatiated, you could also do:
docker exec -d my_container /bin/bash -c "cp /data0/* /data"
You have to use the volumes to store your data.
Here you can find the documentation: https://docs.docker.com/storage/volumes/
For example you can do somethink like this in your docker-compose.yml.
version: '3.1'
services:
wordpress:
image: wordpress:php7.2-apache
ports:
- "8080:80"
environment:
WORDPRESS_DB_HOST: databasename
WORDPRESS_DB_USER: username
WORDPRESS_DB_PASSWORD: password
WORDPRESS_DB_NAME: namedatabase
volumes:
- name_volume:/var/www/html
volumes:
- name_volume:
or
volumes:
- ./yourpath:/var/www/html
I am working on a docker container that is being created from a generic image. The entry point of this container is dependent on a file in the local file system and not in the generic image. My docker-compose file looks something like this:
service_name:
image: base_generic_image
container_name: container_name
entrypoint:
- "/user/dlc/bin"
- "-p"
- "localFolder/fileName.ext"
- more parameters
The challenge that I am facing is removing this dependency and adding it to the base_generic_image at run time so that I can deploy it independently. Should I add this file to the base generic image and then proceed(this file is not required by others) or should this be done when creating the container, if so then what is the best way of going about it.
You should create a separate image for each part of your application. These can be based on the base image if you'd like; the Dockerfile might look like
FROM base_generic_image
COPY dlc /usr/bin
CMD ["dlc"]
Your Docker Compose setup might have a subdirectory for each component and could look like
servicename:
image: my/servicename
build:
context: ./servicename
command: ["dlc", "-p", ...]
In general Docker volumes and bind-mounts are good things to use for persistent data (when absolutely required; stateless containers with external databases are often easier to manage), getting log files out of containers, and pushing complex configuration into containers. The actual program that's being run generally should be built into the base image. The goal is that you can take the image and docker run it on a clean system without any of your development environment on it.
Maybe I'm missing this when reading the docs, but is there a way to overwrite files on the container's file system when issuing a docker run command?
Something akin to the Dockerfile COPY command? The key desire here is to be able to take a particular Docker image, and spin several of the same image up, but with different configuration files. (I'd prefer to do this with environment variables, but the application that I'm Dockerizing is not partial to that.)
You have a few options. Using something like docker-compose, you could automatically build a unique image for each container using your base image as a template. For example, if you had a docker-compose.yml that look liked:
container0:
build: container0
container1:
build: container1
And then inside container0/Dockerfile you had:
FROM larsks/thttpd
COPY index.html /index.html
And inside container0/index.html you had whatever content you
wanted, then running docker-compose build would generate unique
images for each entry (and running docker-compose up would start
everything up).
I've put together an example of the above
here.
Using just the Docker command line, you can use host volume mounts,
which allow you to mount files into a container as well as
directories. Using my thttpd as an example again, you could use the
following -v argument to override /index.html in the container
with the content of your choice:
docker run -v index.html:/index.html larsks/thttpd
And you could accomplish the same thing with docker-compose via the
volume entry:
container0:
image: larsks/thttpd
volumes:
- ./container0/index.html:/index.html
container1:
image: larsks/thttpd
volumes:
- ./container1/index.html:/index.html
I would suggest that using the build mechanism makes more sense if you are trying to override many files, while using volumes is fine for one or two files.
A key difference between the two mechanisms is that when building images, each container will have a copy of the files, while using volume mounts, changes made to the file within the image will be reflected on the host filesystem.