I'd like to keep some less used images on an external disk.
Is that possible?
Or should I move all images to an external disk changing some base path?
All of the Docker images are stored in an opaque, backend-specific format inside the /var/lib/docker directory. You can't move some of the images to a different location, only the entire Docker storage tree. See for example How to change the docker image installation directory?.
If you have images you only rarely use, you can docker rmi them for now and then docker pull them again from Docker Hub or another repository when you need them.
Related
We have requirement in order to fasten the deployment, which is, something like this :
pull docker image
start docker container
copy 2 files from docker container to host
stop and delete docker container
execute 1st file with 2nd file as input, will check and download bigger archived files to a path (skips if bigger archived files already downloaded with integrity check passed)
start docker container with volume mapping.
This is meant to cut down the size of the docker image (say from 30GB) and fasten the deployment. However is there something alternative way , which is, can we do something :
pull docker image
depending on imageId, we can find the detail of files under /var/lib/docker
copy 2 files from /var/lib/docker path to another specific path
question mark - is above step possible in order to get the files according to imageid (suppose two or more docker images were there).
I am using a machine with os ubuntu-18.05 in arm platform. It has some space issue and I want to work with docker images.
Usually, when I work on this machine I mount a directory and perform memory expensive operations there.
Is there any way I can pull image in host machine and make it work?
Example: I have mounted directory /home/test-mount, now instead of storing docker image and it's graph in location mentioned here Where are Docker images stored on the host machine? I want to efficiently pull, store and use image at path /home/test-mount, such that it can be easily switched to actual path.
Stop docker and make sure the following can be found in in /etc/docker/daemon.json:
{
"data-root": "/home/test-mount"
}
Now restart docker
Why does docker have to create an image from a dockerfile then create a container from the image instead of creating a container directly from a Dockerfile?
What is the purpose/benefit of creating the image first from the Dockerfile then from that create a container?
-----EDIT-----
This question What is the difference between a Docker image and a container?
Does not answer my question.
My question is: Why do we need to create a container from an image and not a dockerfile? What is the purpose/benefit of creating the image first from the Dockerfile then from that create a container?
the Dockerfile is the recipe to create an image
the image is a virtual filesystem
the container is the a running process on a host machine
You don't want every host to build its own image based on the recipe. It's easier for some hosts to just download an image and work with that.
Creating an image can be very expensive. I have complicated Dockerfiles that may take hours to build, may download 50 GB of data, yet still only create a 200 MB image that I can send to different hosts.
Spinning up a container from an existing image is very cheap.
If all you had was the Dockerfile in order to spin up image-containers, the entire workflow would become very cumbersome.
Images and Containers are two different concepts.
Basically, images are like a snapshot of a filesystem, along with some meta-data.
A container is one of several process that are actually running (and which is based on an image). As soon as the processes end, your container do not exist anymore (well, it is stopped to be exact)
You can view the image as the base that you will make your container run on.
Thus, you Dockerfile will create an image (which is static) which you can store locally or push on a repository, to be able to use it later.
The container cannot be "stored" because it is a "living" thing.
You can think of Images vs Containers similar to Classes vs Objects or the Definition vs Instance. The image contains the filesystem and default settings for creating the container. The container contains the settings for a specific instance, and when running, the namespaces and running process.
As for why you'd want to separate them, efficiency and portability. Since we have separate images, we also have inheritance, where one image extends another. The key detail of that inheritance is that filesystem layers in the image are not copied for each image. Those layers are static, and you can them by creating a new image with new layers. Using the overlay filesystem (or one of the other union filesystem drivers) we can append additional changes to that filesystem with our new image. Containers do the same when the run the image. That means you can have a 1 Gig base image, extend it with a child image with 100 Megs of changes, and run 5 containers that each write 1 Meg of files, and the overall disk space used on the docker host is only 1.105 Gigs rather than 7.6 Gigs.
The portability part comes into play when you use registries, e.g. Docker Hub. The image is the part of the container that is generic, reusable, and transferable. It's not associated with an instance on any host. So you can push and pull images, but containers are tightly bound to the host they are running on, named volumes on that host, networks defined on that host, etc.
I have a private registry with a set of images. It can be visualized as a store of applications.
My app can take these applications and run them on other machines.
To achieve this, my app first pull the image from the private registry and then copies it to a local registry for later use.
Step as are follow:
docker pull privateregistry:5000/company/app:tag
docker tag privateregistry:5000/company/app:tag localregistry:5000/company/app:tag
docker push localregistry:5000/company/app:tag
Then later on a different machine in my network:
docker pull localregistry:5000/company/app:tag
Is there a way to efficiently copy an image from a repository to another without using a docker client in between ?
you can use docker save to save the images to tar archive and then copy the tar to new host and use docker load to untar it.
read below links for more
https://docs.docker.com/engine/reference/commandline/save/
Is there a way to efficiently copy an image from a repository to another without using a docker client in between?
Yes, there's a variety of tools that implement this today. RedHat has been pushing their skopeo, Google has crane, and I've been working on my own with regclient. Each of these tools talks directly to the registry server without needing a docker engine. And at least with regclient (I haven't tested the others), these will only copy the layers that are not already in the target registry, avoiding the need to pull layers again. Additionally, you can move a multi-platform image, retaining all of the available platforms, which you would lose with a docker pull since that dereferences the image to a single platform.
I need to create a Docker image (and consequently containers from that image) that use large files (containing genomic data, thus reaching ~10GB in size).
How am I supposed to optimize their usage? Am I supposed to include them in the container (such as COPY large_folder large_folder_in_container)? Is there a better way of referencing such files? The point is that it sounds strange to me to push such container (which would be >10GB) in my private repository. I wonder if there is a way of attaching a sort of volume to the container, without packing all those GBs together.
Thank you.
Is there a better way of referencing such files?
If you already have some way to distribute the data I would use a "bind mount" to attach a volume to the containers.
docker run -v /path/to/data/on/host:/path/to/data/in/container <image> ...
That way you can change the image and you won't have to re-download the large data set each time.
If you wanted to use the registry to distribute the large data set, but want to manage changes to the data set separately, you could use a data volume container with a Dockerfile like this:
FROM tianon/true
COPY dataset /dataset
VOLUME /dataset
From your application container you can attach that volume using:
docker run -d --name dataset <data volume image name>
docker run --volumes-from dataset <image> ...
Either way, I think https://docs.docker.com/engine/tutorials/dockervolumes/ are what you want.
Am I supposed to include them in the container (such as COPY large_folder large_folder_in_container)?
If you do so, that would include them in the image, not the container: you could launch 20 containers from that image, the actual disk space used would still be 10 GB.
If you were to make another image from your first image, the layered filesystem will reuse the layers from the parent image, and the new image would still be "only" 10GB.
I have was having trouble with a 900MB json file and changing the Memory limit in the preferences and it fixed it.