I have pulled a couple of images from my private repo. am able to see the size of the layers using docker history <image-id> i don't see actual layer sha256 id for the layers instead it shows missing. So am not sure how I can get the size of each layer.
Actually, I want the size of each layer in the image.
am able to get layers details from docker-inspect command docker inspect <image-id> | jq .[].RootFS.Layers
docker history f183414e30ab
IMAGE CREATED CREATED BY SIZE COMMENT
f183414e30ab 16 months ago /bin/sh -c apt-get update && apt-get install… 317MB
<missing> 16 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 16 months ago /bin/sh -c mkdir -p /run/systemd && echo 'do… 7B
<missing> 16 months ago /bin/sh -c set -xe && echo '#!/bin/sh' > /… 745B
<missing> 16 months ago /bin/sh -c [ -z "$(apt-get indextargets)" ] 987kB
<missing> 16 months ago /bin/sh -c #(nop) ADD file:3ddd02d976792b6c6… 63.2MB
Problem trying to solve: am trying to get the size of docker image layers which also been shared with other images.
You are looking for command docker system df
Related
Docker version 19.03.12, build 48a66213fe
OS: Red Hat Enterprise Linux Server release 7.9 (Maipo)
Please See: Dockerfile have minimal steps and it builds successfully.
To create docker image, I ran (this command will capture stdout/stderr in a .log file):
DOCKER_BUILDKIT=1 docker build \
--network=host \
-t project-opensuse-docker-image:15.2 . \
|& tee -a /tmp/project-opensuse-docker-image.log
When I use DOCKER_BUILDKIT=1 during docker build ....... command, it makes Image ID of the subsequent image layers listed as <missing>; If I re-run docker build again as soon as the first run completes successfully, it is NOT able to use any layer cache concept to go/build fast even though there's no change done to Dockerfile or any folder/file on the file system (i.e if used during COPY / ADD etc steps).
It builds everytime from scratch and I think, the reason is because there's NO valid IMAGE ID for those subsequent layers under the final Image ID: a07eebd0a9ba.
See Image ID in docker history <image> output showing for subsequent layer's IDs.
WHY am I getting under Image ID for these layers?
[gigauser#rh79maipo_machine opensuse-x]$ sudo docker history a07eebd0a9ba
IMAGE CREATED CREATED BY SIZE COMMENT
a07eebd0a9ba 3 hours ago CMD ["ls -l"] 0B buildkit.dockerfile.v0
<missing> 3 hours ago WORKDIR /home/nonroot_user 0B buildkit.dockerfile.v0
<missing> 3 hours ago VOLUME [/home/nonroot_user/git] 0B buildkit.dockerfile.v0
<missing> 3 hours ago RUN /bin/sh -c ls -l /home/nonroot_user /home/rayd… 174MB buildkit.dockerfile.v0
<missing> 3 hours ago COPY ./boost_1_68_0.tar.gz /home/nonroot_user/tool… 109MB buildkit.dockerfile.v0
<missing> 3 hours ago RUN /bin/sh -c ls -l /home/nonroot_user /home/rayd… 253MB buildkit.dockerfile.v0
<missing> 3 hours ago COPY ./wxWidgets-3.1.3.tar.bz2 /home/nonroot_user/… 21.3MB buildkit.dockerfile.v0
<missing> 3 hours ago RUN /bin/sh -c echo -e "\n-- Installing Zypper… 1.71GB buildkit.dockerfile.v0
<missing> 26 hours ago RUN /bin/sh -c useradd -r -m -l … 953kB buildkit.dockerfile.v0
<missing> 26 hours ago COPY ./CompanyCertBundle/PEM/*.cer /usr/sha… 34.7kB buildkit.dockerfile.v0
<missing> 26 hours ago ENV http_proxy=http://company.proxy.com:80/ … 0B buildkit.dockerfile.v0
<missing> 26 hours ago LABEL Project=aPROJECT IM CentOS_Version=CentOS… 0B buildkit.dockerfile.v0
<missing> 4 weeks ago KIWI 9.23.20 109MB
[gigauser#rh79maipo_machine opensuse-x]$
I have a docker image.
When I use the docker history command on the image, I can see
85d9bf810d44 9 days ago /bin/sh -c apk add vim 26.9MB
<missing> 9 days ago /bin/sh -c apk update 1.78MB
<missing> 9 days ago /bin/sh -c rm -f file.txt 0B
<missing> 9 days ago /bin/sh -c a=$(base64 -d < file.txt) && echo $a … 49B
<missing> 9 days ago /bin/sh -c #(nop) COPY file:98f5646751cb4985… 68B
<missing> 6 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 6 weeks ago /bin/sh -c #(nop) ADD file:f17f65714f703db90… 5.57MB
So there was a file.txt at some point in the image, but it was later removed. I would like to know if there is a way to retrieve the content of that file from the image layers.
I have looked into Dive and all sorts of stuff. Also navigating through Docker's overlay files (as indicated here) seemed promising, but I am using macOS and I couldn't find the corresponding directories...
docker image save will export a tarball that contains a tarball per layer.
https://docs.docker.com/engine/reference/commandline/image_save/
This Dockerfile hangs after the download has completed:
FROM ubuntu:18.04
MAINTAINER Dean Schulze
ENV JS_CE_VERSION 7.1.0
ENV JS_CE_HOME /opt/jasperreports-server-cp-${JS_CE_VERSION}
ENV PATH $PATH:${JS_CE_HOME}
RUN apt-get update && apt-get install -y wget \
&& wget --progress=bar:force:noscroll -O TIB_js-jrs-cp_${JS_CE_VERSION}_linux_x86_64.run https://sourceforge.net/projects/jasperserver/files/JasperServer/JasperReports%20Server%20Community%20Edition%20${JS_CE_VERSION}/TIB_js-jrs-cp_${JS_CE_VERSION}_linux_x86_64.run \
&& chmod a+x TIB_js-jrs-cp_${JS_CE_VERSION}_linux_x86_64.run \
&& /TIB_js-jrs-cp_${JS_CE_VERSION}_linux_x86_64.run --mode unattended --jasperLicenseAccepted yes --postgres_password Postgres1 --prefix ${JS_CE_HOME} \
&& rm TIB_js-jrs-cp_${JS_CE_VERSION}_linux_x86_64.run \
&& rm -rf ${JS_CE_HOME}/apache-ant ${JS_CE_HOME}/buildomatic \
${JS_CE_HOME}/docs ${JS_CE_HOME}/samples ${JS_CE_HOME}/scripts \
&& apt-get clean
EXPOSE 8080
CMD ctlscript.sh start && tail -f /dev/null
The last few lines of output are
Resolving superb-dca2.dl.sourceforge.net (superb-dca2.dl.sourceforge.net)... 209.61.193.20
Connecting to superb-dca2.dl.sourceforge.net (superb-dca2.dl.sourceforge.net)|209.61.193.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 343517555 (328M) [application/x-makeself]
Saving to: 'TIB_js-jrs-cp_7.1.0_linux_x86_64.run'
TIB_js-jrs-cp_7.1.0 100%[===================>] 327.60M 1.60MB/s in 3m 52s
2018-07-28 03:15:28 (1.41 MB/s) - 'TIB_js-jrs-cp_7.1.0_linux_x86_64.run' saved [343517555/343517555]
How do I diagnose a docker build that hangs like this?
Edit
Here's the requested output:
$ docker image history d5d47e51eafc
IMAGE CREATED CREATED BY SIZE COMMENT
d5d47e51eafc 23 minutes ago /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:… 0B
831a3a551fa7 23 minutes ago /bin/sh -c #(nop) ENV JS_CE_HOME=/opt/jaspe… 0B
e8361426e492 23 minutes ago /bin/sh -c #(nop) ENV JS_CE_VERSION=6.3.0 0B
7af364f52b1b 23 minutes ago /bin/sh -c #(nop) MAINTAINER JS Minet 0B
735f80812f90 30 hours ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 30 hours ago /bin/sh -c mkdir -p /run/systemd && echo 'do… 7B
<missing> 30 hours ago /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$… 2.76kB
<missing> 30 hours ago /bin/sh -c rm -rf /var/lib/apt/lists/* 0B
<missing> 30 hours ago /bin/sh -c set -xe && echo '#!/bin/sh' > /… 745B
<missing> 30 hours ago /bin/sh -c #(nop) ADD file:4bb62bb0587406855… 83.5MB
It looks like the lines are in reverse order and the last one executed is the ENV PATH. The next line would be the RUN line which has multiple commands separated by &&.
This is a modification of a Dockerfile on DockerHub. I changed ubuntu versions and the version of the app being installed. That shouldn't have broken anything.
Should I file a bug report?
First list the "layers" of your finished or incomplete image. Each layer typically corresponds to an instruction in your Dockerfile.
Identify the image ID using
docker image ls
then list the layers inside the image using
docker image history <ID>
You will see something like this:
IMAGE CREATED CREATED BY SIZE COMMENT
6c32fe3da793 2 days ago /bin/sh -c #(nop) COPY file:c25ef1dcc737cb59… 635B
4c1309db9b9c 2 days ago /bin/sh -c #(nop) COPY dir:30506cf0fc0cdb096… 8.64kB
5f5ae40b5fd5 3 weeks ago /bin/sh -c apk update && apk add --no-cache … 164MB
989d78741d0b 3 weeks ago /bin/sh -c #(nop) ENV DOCKER_VERSION=18.03.… 0B
6160911711fc 3 weeks ago /bin/sh -c #(nop) CMD ["python3"] 0B
... etc
Then create a container from any point inside your image. From there you can perform the next instruction that would cause a problem in your Dockerfile.
Eg:
docker run -it --rm 4c1309db9b9c sh
At some point in the last couple of years, Buildkit has become the default Docker backend. Buildkit does not write out intermediate layers as images, as a performance optimization. Therefore, if you need to debug a hanging docker build, comment out the line that the build is hanging at and all subsequent lines. You now have a Dockerfile that you can execute $ docker build . with. Once the build is finished, you can launch the image, bash into the container, run the command that is causing the build to hang and then inspect the state of the container to debug the situation.
I got this to work by rolling back to ubuntu 16.04. There must be some change in 18.04 that causes this Dockerfile to fail, or maybe the ubuntu 18.04 image omitted something.
When you create your own docker image, you usually start a Docker file with FROM, and base your image off something that already exists on docker hub. How can I learn more about what is actually in the image I am referencing?
For example, I'm interested in starting with this image:
https://hub.docker.com/_/swift/
Besides what's listed in the description fields on that webpage, how can I verify what is actually getting installed? Is there a way to view a Dockerfile for an existing image on docker hub?
Thanks
Dockerfile links
Many images, especially "official" images, will contain Dockerfile links. You'll find them in the description on Docker Hub. For instance, right now at the link you posted in your question, you'll find a few image tags and a couple of links to Dockerfile.
3.1.0, 3.1, 3, latest (Dockerfile)
Simply click on "Dockerfile" and it will take you to the Dockerfile that was used to build that version of the image.
It should be noted that this is metadata associated with the Docker Hub account. You can't completely trust that it is correct, because it's just a link. (To GitHub, in this case, but it can be anywhere.)
Since you can't completely trust that, you may want to look also at...
Docker history
If you docker pull swift to fetch the image, you can then use the docker history command to take a closer look at it. Currently, that looks like this:
IMAGE CREATED CREATED BY SIZE COMMENT
d505ae70cb39 2 weeks ago /bin/sh -c swift --version 0B
<missing> 2 weeks ago /bin/sh -c SWIFT_URL=https://swift.org/bui... 403MB
<missing> 2 weeks ago /bin/sh -c #(nop) ENV SWIFT_PLATFORM=ubun... 0B
<missing> 2 weeks ago /bin/sh -c #(nop) ARG SWIFT_VERSION=swift... 0B
<missing> 2 weeks ago /bin/sh -c #(nop) ARG SWIFT_BRANCH=swift-... 0B
<missing> 2 weeks ago /bin/sh -c #(nop) ARG SWIFT_PLATFORM=ubun... 0B
<missing> 2 weeks ago /bin/sh -c apt-get -q update && apt-ge... 626MB
<missing> 2 weeks ago /bin/sh -c #(nop) MAINTAINER Haris Amin <... 0B
<missing> 2 weeks ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 2 weeks ago /bin/sh -c mkdir -p /run/systemd && echo '... 7B
<missing> 2 weeks ago /bin/sh -c sed -i 's/^#\s*\(deb.*universe\... 2.76kB
<missing> 2 weeks ago /bin/sh -c rm -rf /var/lib/apt/lists/* 0B
<missing> 2 weeks ago /bin/sh -c set -xe && echo '#!/bin/sh' >... 745B
<missing> 2 weeks ago /bin/sh -c #(nop) ADD file:5aff8c59a707833... 118MB
You'll notice that the commands used to build each layer of the image are truncated for display. That makes this display not especially useful, but you can use the --no-trunc flag to get a much more verbose output.
docker history --no-trunc swift:latest
Then you will get a lot of output (more than I will paste here), but here is a one-entry sample:
<missing> 2 weeks ago /bin/sh -c SWIFT_URL=https://swift.org/builds/$SWIFT_BRANCH/$(echo "$SWIFT_PLATFORM" | tr -d .)/$SWIFT_VERSION/$SWIFT_VERSION-$SWIFT_PLATFORM.tar.gz && curl -fSsL $SWIFT_URL -o swift.tar.gz && curl -fSsL $SWIFT_URL.sig -o swift.tar.gz.sig && export GNUPGHOME="$(mktemp -d)" && set -e; for key in 7463A81A4B2EEA1B551FFBCFD441C977412B37AD 1BE1E29A084CB305F397D62A9F597F4D21A56D5F A3BAFD3556A59079C06894BD63BC1CFE91D306C6 ; do gpg --quiet --keyserver ha.pool.sks-keyservers.net --recv-keys "$key"; done && gpg --batch --verify --quiet swift.tar.gz.sig swift.tar.gz && tar -xzf swift.tar.gz --directory / --strip-components=1 && rm -r "$GNUPGHOME" swift.tar.gz.sig swift.tar.gz 403MB
Most of the text is simply the commands executed by Dockerfile RUN statements. You will also see the other Dockerfile commands like ARG, CMD, ADD, COPY, etc.
Since this is encoded into the image layers, it is probably more reliable (if less readable) than the Dockerfile links found in the Docker Hub readme file.
Let's say there is a docker image, someone makes changes to it and then pushes it up to a docker repository. I then pull down the image. Is there a way to then take that image and run a container from a previous layer? Run the version before the changes were made.
If I run docker history it will look something like this:
docker history imagename:tag
IMAGE CREATED CREATED BY SIZE COMMENT
3e23a5875458 8 days ago /bin/sh -c #(nop) ENV LC_ALL=C.UTF-8 0 B
<missing> 8 days ago /bin/sh -c dpkg-reconfigure locales && loc 1.245 MB
<missing> 8 days ago /bin/sh -c apt-get update && apt-get install 338.3 MB
<missing> 6 weeks ago /bin/sh -c #(nop) ADD jessie.tar.xz in / 121 MB
<missing> 6 weeks ago /bin/sh -c #(nop) MAINTAINER ssss <ad 0 B
<missing> 9 months ago 0 B
It seems as if I could run an earlier version if I figure out a way to somehow tag or identify previous layers of the image.
You can, by tagging build layers of the image if you have access to them. As described here.
In your case what could be happening is that from version v1.10.0 forward they've changed the way that the docker engine handles content addressability. This is being heavily discussed here.
What it means is that you won't have access to the build layers unless you built this image in the current machine or exported and loaded by combining:
docker save imagename build-layer1 build-layer2 build-layer3 > image-caching.tar
docker load -i image-caching.tar
A user has posted a handy way to save that cache in the discussion I've mentioned previously:
docker save imagename $(sudo docker history -q imagename | tail -n +2 | grep -v \<missing\> | tr '\n' ' ') > image-caching.tar
This should collect all the build layers of the given image and save them to a cache tar file.