There are around 10 container image files on the current directory, and I want to load them to my Kubernetes cluster that is using containerd as CRI.
[root#test tmp]# ls -1
test1.tar
test2.tar
test3.tar
...
I tried to load them at once using xargs but got the following result:
[root#test tmp]# ls -1 | xargs nerdctl load -i
unpacking image1:1.0 (sha256:...)...done
[root#test tmp]#
The first tar file was successfully loaded, but the command exited and the remaining tar files were not processed.
I have confirmed the command nerdctl load -i succeeded with exit code 0.
[root#test tmp]# nerdctl load -i test1.tar
unpacking image1:1.0 (sha256:...)...done
[root#test tmp]# echo $?
0
Does anyone know the cause?
Your actual ls command piped to xargs is seen as a single argument where file names are separated by null bytes (shortly said... see for example this article for a better in-depth analyze). If your version of xargs supports it, you can use the -0 option to take this into account:
ls -1 | xargs -0 nerdctl load -i
Meanwhile, this is not really safe and you should see why it's not a good idea to loop over ls output in your shell
I would rather transform the above to the following command:
for f in *.tar; do
nerdctl load -i "$f"
done
Related
I have a Docker image which contains a file, say /usr/bin/foo. What's the easiest way to find out which step of the Dockerfile added that path? (Which I thought was equivalent to the question, of which layer of the Docker image does that path come from?)
I wrote a script which prints out all the paths in the image, prefixed by layer ID. It appears to work, but is quite slow:
#!/bin/bash
die() { echo 1>&2 "ERROR: $*"; exit 1; }
dir=$(mktemp -d)
trap "rm -rf $dir" EXIT
img="$1"
[[ -n "$img" ]] || die "wrong arguments"
docker image save "$img" | (cd $dir && tar xf -) ||
die "failed extracting docker image $img"
(cd $dir && find . -name '*.tar' | while read f; do layer=$(echo $f | cut -d/ -f2); tar tf $f | sed -e "s/^/$layer:/"; done) ||
die "failed listing layers"
(It could be made faster if it didn't write anything to disk. The problem is while tar tf - prints the paths in the TAR, it doesn't do the same for the nested layer.tar files. I am thinking I could use the Python tarfile module - but surely somebody else out there has done this already?)
However, I don't know how to translate the layer ID it gives me to a step in the Docker image. I thought I'd correlate it with the layer IDs reported by docker inspect:
docker image inspect $IMAGE | jq -r '.[].RootFS.Layers[]' | nl
But the layer ID which my script reports as containing the path, I can't find in the output of the above command. (Is that a consequence of BuildKit???)
In the end, I gave up on this whole approach. Instead I just made some educated guesses as to which Dockerfile line was probably creating that path, tested each guess by commenting it out (and all the lines after it), and soon I found the answer. Still, there must be a better way, surely? Ideally, what I'd like is something like a --contains-path= option to docker image history – which doesn't exist, but maybe there is something else which does the equivalent?
While dlayer does not have any searching function built-in, it is straight-forward to implement by combining it with a Perl one-liner:
docker image save $IMAGE |
dlayer -n 999999 |
perl -ne 'chomp;$query=quotemeta("usr/bin/foo");$cmd=$_ if $_ =~ m/ [\$] /;print "$cmd\n\t$_\n" if m/ $query/;'
This will print something like:
13 MB $ /opt/bar/install.sh # buildkit
637 B usr/bin/foo
-n 999999 is to increase limit of number of file names output from the default 100, otherwise the path will only be found if it is in the first 100 from that layer.
(I submitted a PR to add a built-in search function to dlayer, which removes the need for this one-line Perl script.)
I have 20 images TARed, now I want to load those images on another system. However, loading itself is taking 30 to 40 minutes. All images are independent of each other so all images loading should happen in parallel, I believe.
I tried solution like running load command in background(&) and wait till loading finishes, but observed that it is taking even more time. Any help here is highly appreciated.
Note:- not sure about the option -i to docker load command.
Try
find /path/to/image/archives/ -iname "*.tar" -o -iname "*.tar.xz" |xargs -r -P4 -i docker load -i {}
This will load Docker image archives in parallel (adjust -P4 to the desired number of parallel loads or set to -P0 for unlimited concurrency).
For speeding up the pulling/saving processes, you can use ideas from the snippet below:
#!/usr/bin/env bash
TEMP_FILE="docker-compose.image.pull.yaml"
image_name()
{
local name="$1"
echo "$name" | awk -F '[:/]' '{ print $1 }'
}
pull_images_file_gen()
{
local from_file="$1"
cat <<EOF >"$TEMP_FILE"
version: '3.4'
services:
EOF
while read -r line; do
cat <<EOF >>"$TEMP_FILE"
$(image_name "$line"):
image: $line
EOF
done < "$from_file"
}
save_images()
{
local from_file="$1"
while read -r line; do
docker save -o /tmp/"$(image_name "$line")".tar "$line" &>/dev/null & disown;
done < "$from_file"
}
pull_images_file_gen "images"
docker-compose -f $TEMP_FILE pull
save_images "images"
rm -f $TEMP_FILE
images - contains needed Docker images names list line by line.
Good luck!
I have list of .tar docker image files , I have tried loading docker images using below commands
docker load -i *.tar
docker load -i alldockerimages.tar
where alldockerimages.tar contains all individual tar files .
Let me know how we can load multiple tar files.
Using xargs:
ls -1 *.tar | xargs --no-run-if-empty -L 1 docker load -i
(A previous revision left off the -i flag to "docker load".)
First I attempted to use the glob expression approach you first described:
# download some images to play with
docker pull alpine
docker pull nginx:alpine
# stream the images to disk as tarballs
docker save alpine > alpine.tar
docker save nginx:alpine > nginx.tar
# delete the images so we can attempt to load them from scratch
docker rmi alpine nginx:alpine
# issue the load command to try and load all images at once
cat *.tar | docker load
Unfortunately this only resulted in alpine.tar being loaded. It was my (presumably faulty) understanding that the glob expression would be expanded and ultimately cause the docker load command to be run for every file into which the glob expression expanded.
Therefore, one has to use a shell for loop to load all tarballs sequentially:
for f in *.tar; do
cat $f | docker load
done
Use the script described in save-load-docker-images.sh to save or load the images. For your case it would be
./save-load-docker-images.sh load -d <directory-location>
You can try the next option using find:
find -type f -name "*.tar" -exec docker load --input "{}" \;
I have container which in logs sometimes write key word which is for me important, and I want to highlight this word in color in my terminal, but also important is still see all content logs in real time (--follow). I just tried command
docker logs -f my_app --tail=100 | grep --color -E '^myWord'
but not working.
So exist some way to do this ?
I use ccze. as #aimless said, grc is the great utility also. It easy to install by sudo apt install ccze for debian/ubuntu-like OS
But if you want to colorize stderr, you need to redirect stderr output to stdout. For example:
docker logs -f my-app 2>&1 | ccze -m ansi
arg -m ansi helps if you want to scroll output normally
UPD:
ccze can be very slow. If you encounter this, try running ccze with the nolookups option: ccze -o nolookups.
originally answered - https://unix.stackexchange.com/a/461390/83391
Try this.
docker logs -f my_app --tail=100 | grep --color=always -E '^myWord'
Note the "--color=always" argument.
Another option would be to use something like https://github.com/jlinoff/colorize. I wrote it to specifically address situations like this. For example it has the ability to specify different colors for each pattern (see the help for details).
Here is an example of how to use it for your case.
$ curl -L https://github.com/jlinoff/colorize/releases/download/v0.8.1/colorize-linux-amd64 --out colorize
$ chmod a+x colorize
$ ./colorize -h
$ docker logs -f my_app --tail=100 | ./colorize '^myWord'
$ # really make it standout.
$ docker logs -f my_app --tail=100 | ./colorize -c red+greenB+bold '^myWord'
try grc. Follow the instruction to install and just pipe the logs output:
docker logs -app | grc
Docker caching is not yet available on travis: https://github.com/travis-ci/travis-ci/issues/5358
I'm trying to write a workaround by doing:
`docker save -o file.tar $(docker history -q image_name | grep -v missing)`
`docker load -i file.tar
Which works great, gives me all the image layers back. My only problem now is the saving takes a long time, and most of the time I'm actually changing one layer, so I don't need to rewrite all the rest. Is there a way of telling the docker save command to skip layers already in file.tar?
In the manifest.json file inside the tar you have the information you need.
tar -xOf file.tar manifest.json
Check the value of the Config keys. The first 12 characters are the image id. You can use the command above, extract the image ids that you already have, and exclude them in your docker save command.
I'm not very good with bash scripting, but this works on my mac
tar -xOf file.tar manifest.json | tr , '\n' | grep -o '"Config":".*"' | awk -F ':' '{print $2}' | awk '{print substr($0,2,12)}'
Using this outputs everything
docker history -q IMAGE_HERE | grep -v missing && tar -xOf file.tar manifest.json | tr , '\n' | grep -o '"Config":".*"' | awk -F ':' '{print $2}' | awk '{print substr($0,2,12)}'
After this you only need to get the unique values. This could be done with sort and uniq -u, but for some reason, sort doesn't work as expected. This command assumes the presence of file.tar so take that into consideration too.
I couldn't find anything about append in the docker save command. The above strategy could work with multiple file tars that are all different with each other.