I want to distribute a Docker Compose system as a single archive. For this I want to run docker save on all the images. But how do I get the list of images from Docker Compose?
This is what I do at the moment:
IMAGES=$(cat docker-compose.yml | sed -n 's/image:\(.*\)/\1/p')
docker save -o images.tar $IMAGES
Do the following:
# Save Compressed Images
IMAGES=`grep '^\s*image' docker-compose.yml | sed 's/image://' | sort | uniq`
docker save $IMAGES | gzip > images.tar.gz
# Load Compressed Images
gunzip -c images.tar.gz | docker load
Related
Suppose there is a container image published to a Docker registry, and I want to rebuild the image with modifications while re-using as many of the published layers as possible.
Concretely, suppose the image foo/bar is built from this Dockerfile, and I only want to modify the layer that contains script.sh:
FROM ubuntu:focal
COPY script.sh .
Even though pulling the image downloads the layers for ubuntu:focal, when I re-build my local machine may resolve the ubuntu:focal tag to another version, producing a new image with no common layers with the one I pulled.
6a9e8d7 <foo/bar:new>
<foo/bar:old> c7632a5 |
| |
+----+----+
|
3b15784 <ubuntu:focal (then)>
|
...
DESIRABLE
<foo/bar:old> c7632a5 48fead0 <foo/bar:new>
| |
| |
<ubuntu:focal (then)> 3b15784 9a634f5 <ubuntu:focal (now)>
| |
... ...
UNDESIRABLE
The desired outcome could possibly be achieved by looking at the pulled layers and tagging the correct one (3b15784) as ubuntu:focal before building. But I'm not sure if Docker exposes enough information to do this in an automatic way.
As a workaround, I could explicitly include the base image's digest as a label on the built image:
FROM ${UBUNTU_IMAGE_ID:-ubuntu:focal}
COPY script.sh .
Then I would build with:
# First build
UBUNTU_IMAGE_ID=$( \
docker inspect \
--format '{{ index .RepoTags 0 }}#{{ index . "Id" }}' \
ubuntu:focal \
)
# Subsequent builds
docker pull foo/bar:old
UBUNTU_IMAGE_ID=$( \
docker inspect \
--format '{{ index .Config.Labels "ubuntu_image_id" }}' \
foo/bar:old \
)
docker build . \
--build-arg "UBUNTU_IMAGE_ID=${UBUNTU_IMAGE_ID}" \
--label "ubuntu_image_id=${UBUNTU_IMAGE_ID}" \
--tag foo/bar:new
However, a more elegant solution would definitely be welcome, especially one that does not involve any specific assumptions about how the original pulled image was built.
Maybe Docker supports this built in with --cache-from:
docker pull foo/bar:old
docker build . \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from foo/bar:old
--tag foo/bar:new
There are around 10 container image files on the current directory, and I want to load them to my Kubernetes cluster that is using containerd as CRI.
[root#test tmp]# ls -1
test1.tar
test2.tar
test3.tar
...
I tried to load them at once using xargs but got the following result:
[root#test tmp]# ls -1 | xargs nerdctl load -i
unpacking image1:1.0 (sha256:...)...done
[root#test tmp]#
The first tar file was successfully loaded, but the command exited and the remaining tar files were not processed.
I have confirmed the command nerdctl load -i succeeded with exit code 0.
[root#test tmp]# nerdctl load -i test1.tar
unpacking image1:1.0 (sha256:...)...done
[root#test tmp]# echo $?
0
Does anyone know the cause?
Your actual ls command piped to xargs is seen as a single argument where file names are separated by null bytes (shortly said... see for example this article for a better in-depth analyze). If your version of xargs supports it, you can use the -0 option to take this into account:
ls -1 | xargs -0 nerdctl load -i
Meanwhile, this is not really safe and you should see why it's not a good idea to loop over ls output in your shell
I would rather transform the above to the following command:
for f in *.tar; do
nerdctl load -i "$f"
done
I have 20 images TARed, now I want to load those images on another system. However, loading itself is taking 30 to 40 minutes. All images are independent of each other so all images loading should happen in parallel, I believe.
I tried solution like running load command in background(&) and wait till loading finishes, but observed that it is taking even more time. Any help here is highly appreciated.
Note:- not sure about the option -i to docker load command.
Try
find /path/to/image/archives/ -iname "*.tar" -o -iname "*.tar.xz" |xargs -r -P4 -i docker load -i {}
This will load Docker image archives in parallel (adjust -P4 to the desired number of parallel loads or set to -P0 for unlimited concurrency).
For speeding up the pulling/saving processes, you can use ideas from the snippet below:
#!/usr/bin/env bash
TEMP_FILE="docker-compose.image.pull.yaml"
image_name()
{
local name="$1"
echo "$name" | awk -F '[:/]' '{ print $1 }'
}
pull_images_file_gen()
{
local from_file="$1"
cat <<EOF >"$TEMP_FILE"
version: '3.4'
services:
EOF
while read -r line; do
cat <<EOF >>"$TEMP_FILE"
$(image_name "$line"):
image: $line
EOF
done < "$from_file"
}
save_images()
{
local from_file="$1"
while read -r line; do
docker save -o /tmp/"$(image_name "$line")".tar "$line" &>/dev/null & disown;
done < "$from_file"
}
pull_images_file_gen "images"
docker-compose -f $TEMP_FILE pull
save_images "images"
rm -f $TEMP_FILE
images - contains needed Docker images names list line by line.
Good luck!
I have list of .tar docker image files , I have tried loading docker images using below commands
docker load -i *.tar
docker load -i alldockerimages.tar
where alldockerimages.tar contains all individual tar files .
Let me know how we can load multiple tar files.
Using xargs:
ls -1 *.tar | xargs --no-run-if-empty -L 1 docker load -i
(A previous revision left off the -i flag to "docker load".)
First I attempted to use the glob expression approach you first described:
# download some images to play with
docker pull alpine
docker pull nginx:alpine
# stream the images to disk as tarballs
docker save alpine > alpine.tar
docker save nginx:alpine > nginx.tar
# delete the images so we can attempt to load them from scratch
docker rmi alpine nginx:alpine
# issue the load command to try and load all images at once
cat *.tar | docker load
Unfortunately this only resulted in alpine.tar being loaded. It was my (presumably faulty) understanding that the glob expression would be expanded and ultimately cause the docker load command to be run for every file into which the glob expression expanded.
Therefore, one has to use a shell for loop to load all tarballs sequentially:
for f in *.tar; do
cat $f | docker load
done
Use the script described in save-load-docker-images.sh to save or load the images. For your case it would be
./save-load-docker-images.sh load -d <directory-location>
You can try the next option using find:
find -type f -name "*.tar" -exec docker load --input "{}" \;
Docker caching is not yet available on travis: https://github.com/travis-ci/travis-ci/issues/5358
I'm trying to write a workaround by doing:
`docker save -o file.tar $(docker history -q image_name | grep -v missing)`
`docker load -i file.tar
Which works great, gives me all the image layers back. My only problem now is the saving takes a long time, and most of the time I'm actually changing one layer, so I don't need to rewrite all the rest. Is there a way of telling the docker save command to skip layers already in file.tar?
In the manifest.json file inside the tar you have the information you need.
tar -xOf file.tar manifest.json
Check the value of the Config keys. The first 12 characters are the image id. You can use the command above, extract the image ids that you already have, and exclude them in your docker save command.
I'm not very good with bash scripting, but this works on my mac
tar -xOf file.tar manifest.json | tr , '\n' | grep -o '"Config":".*"' | awk -F ':' '{print $2}' | awk '{print substr($0,2,12)}'
Using this outputs everything
docker history -q IMAGE_HERE | grep -v missing && tar -xOf file.tar manifest.json | tr , '\n' | grep -o '"Config":".*"' | awk -F ':' '{print $2}' | awk '{print substr($0,2,12)}'
After this you only need to get the unique values. This could be done with sort and uniq -u, but for some reason, sort doesn't work as expected. This command assumes the presence of file.tar so take that into consideration too.
I couldn't find anything about append in the docker save command. The above strategy could work with multiple file tars that are all different with each other.