Related
So I have a private repository at docker hub
and I am trying to download image (blobs) manually using HTTP API.
Now there are some issues
I have tried https://hub.docker.com/support/doc/how-do-i-authenticate-with-the-v2-api
and this script works and I can see my tags.
But there is no API in docker HUB api to get list of blobs from a tag and then download it.
There is a docker registry api, but there my username password does not work.
What to do?
For an image, you first pull the manifest and parse that for the list of blobs that you need to pull. All of these API's need the same authorization headers used to list the tags. I'm going to be using regctl from my regclient project to query a local registry, but you could also use curl against Hub which I show below.
$ regctl tag ls localhost:5000/library/alpine
3
3-bkup-20210904
3.10
3.11
3.12
3.13
3.14
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
latest
$ regctl manifest get localhost:5000/library/alpine:latest --format body | jq .
{
"manifests": [
{
"digest": "sha256:14b55f5bb845c7b810283290ce057f175de87838be56f49060e941580032c60c",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "amd64",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:40f396779ba29da16f29f780963bd4ad5b7719e3eb5dec04516d583713256aa8",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v6"
},
"size": 528
},
{
"digest": "sha256:392d9d85dff31e34d756be33579f05ef493cb1b0edccc36a11b3295365553bfd",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v7"
},
"size": 528
},
{
"digest": "sha256:4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm64",
"os": "linux",
"variant": "v8"
},
"size": 528
},
{
"digest": "sha256:e8d9cf28250078f08e890a3466efbefda68a8feac03cc4076d3ada3397370d6e",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "386",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:d860569a59af627dafee0b0f2b8069e31b07fbdaebe552904dbaec28047ccf64",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "ppc64le",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:6640b198347e5bf1e9a9dc5fc864e927154275dc31f3d26193b74350a5c94c9f",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "s390x",
"os": "linux"
},
"size": 528
}
],
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"schemaVersion": 2
}
$ regctl manifest get localhost:5000/library/alpine#sha256:14b55f5bb845c7b810283290ce057f175de87838be56f49060e941580032c60c --format body | jq .
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1472,
"digest": "sha256:e9adb5357e84d853cc3eb08cd4d3f9bd6cebdb8a67f0415cc884be7b0202416d"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2812636,
"digest": "sha256:3d243047344378e9b7136d552d48feb7ea8b6fe14ce0990e0cc011d5e369626a"
}
]
}
$ regctl blob get localhost:5000/library/alpine sha256:e9adb5357e84d853cc3eb08cd4d3f9bd6cebdb8a67f0415cc884be7b0202416d | jq .
{
"architecture": "amd64",
"config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh"
],
"Image": "sha256:e211ac20c5c7aaa4ed30d5553654d4679082ec48efcb4d164bac6d50d62653fd",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"container": "b6ba94212561a8075e1d324fb050db160e25035ffcfbbe5b410e411e2b7000e2",
"container_config": {
"Hostname": "b6ba94212561",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"/bin/sh\"]"
],
"Image": "sha256:e211ac20c5c7aaa4ed30d5553654d4679082ec48efcb4d164bac6d50d62653fd",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"created": "2022-03-17T04:01:59.188838147Z",
"docker_version": "20.10.12",
"history": [
{
"created": "2022-03-17T04:01:58.883733237Z",
"created_by": "/bin/sh -c #(nop) ADD file:cf4b631a115c2bbfbd81cad2d3041bceb64a8136aac92ba8a63b6c51d60af764 in / "
},
{
"created": "2022-03-17T04:01:59.188838147Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/sh\"]",
"empty_layer": true
}
],
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:5e03d8cae8773cb694fff1d55da34a40d23c2349087ed15ce68476395d33753c"
]
}
}
$ regctl blob get localhost:5000/library/alpine sha256:3d243047344378e9b7136d552d48feb7ea8b6fe14ce0990e0cc011d5e369626a | tar -tvzf - | head
drwxr-xr-x 0/0 0 2022-03-16 16:15 bin/
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/arch -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/ash -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/base64 -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/bbconfig -> /bin/busybox
-rwxr-xr-x 0/0 824984 2022-02-02 13:21 bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/cat -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chgrp -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chmod -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chown -> /bin/busybox
...
A few examples if you try doing this with curl:
Get a token (specific to Docker Hub, each registry may have different auth methods and servers):
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
To use a user/password rather than an anonymous login, call curl with the -u user:pass option:
token=$(curl -u "$user:$pass" -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
Tags:
curl -H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/tags/list" | jq .
Manifests:
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
curl -H "Accept: ${api}" -H "Accept: ${apil}" \
-H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}" | jq .
And blobs:
curl -H "Authorization: Bearer $token" \
-s -L -o - "https://registry-1.docker.io/v2/${repo}/blobs/${digest}"
If you want to export everything to be able to later import, regctl image export will convert the image to a tar of manifests and blobs. This is output in the OCI Layout format, and if you only pull a single image manifest (and not a multi-platform manifest) it will also include the needed files for docker load to import the tar.
There is a docker registry api, but there my username password does not work. What to do?
It probably depends on how you supply them. I didn't try it with a private repository on Docker Hub, but here's a script that downloads public images from Docker Hub, and images from private registries:
#!/usr/bin/env bash
set -eu
image=$1
creds=${2-}
# https://github.com/moby/moby/blob/v20.10.18/vendor/github.com/docker/distribution/reference/normalize.go#L29-L57
# https://github.com/moby/moby/blob/v20.10.18/vendor/github.com/docker/distribution/reference/normalize.go#L88-L105
registry=${image%%/*}
if [ "$registry" = "$image" ] \
|| { [ "`expr index "$registry" .:`" = 0 ] && [ "$registry" != localhost ]; }; then
registry=docker.io
else
image=${image#*/}
fi
if [ "$registry" = docker.io ] && [ "`expr index "$image" /`" = 0 ]; then
image=library/$image
fi
if [ "`expr index "$image" :`" = 0 ]; then
tag=latest
else
tag=${image#*:}
image=${image%:*}
fi
if [ "$registry" = docker.io ]; then
registry=https://registry-1.docker.io
elif ! [[ "$registry" =~ ^localhost(:[0-9]+)$ ]]; then
registry=https://$registry
fi
r=`curl -sS "$registry/v2/" \
-o /dev/null \
-w '%{http_code}:%header{www-authenticate}'`
http_code=`echo "$r" | cut -d: -f1`
curl_args=(-sS -H 'Accept: application/vnd.docker.distribution.manifest.v2+json')
if [ "$http_code" = 401 ]; then
if [ "$registry" = https://registry-1.docker.io ]; then
header_www_authenticate=`echo "$r" | cut -d: -f2-`
header_www_authenticate=`echo "$header_www_authenticate" | sed -E 's/^Bearer +//'`
split_into_lines() {
sed -Ee :1 -e 's/^(([^",]|"([^"]|\")*")*),/\1\n/; t1'
}
header_www_authenticate=`echo "$header_www_authenticate" | split_into_lines`
extract_value() {
sed -E 's/^[^=]+="(([^"]|\")*)"$/\1/; s/\\(.)/\1/g'
}
realm=$(echo "$header_www_authenticate" | grep '^realm=' | extract_value)
service=$(echo "$header_www_authenticate" | grep '^service=' | extract_value)
scope=repository:$image:pull
token=`curl -sS "$realm?service=$service&scope=$scope" | jq -r .token`
curl_args+=(-H "Authorization: Bearer $token")
else
curl_args+=(-u "$creds")
fi
fi
manifest=`curl "${curl_args[#]}" "$registry/v2/$image/manifests/$tag"`
config_digest=`echo "$manifest" | jq -r .config.digest`
config=`curl "${curl_args[#]}" -L "$registry/v2/$image/blobs/$config_digest"`
layers=`echo "$manifest" | jq -r '.layers[] | .digest'`
echo "$layers" | \
while IFS= read -r digest; do
curl "${curl_args[#]}" -L "$registry/v2/$image/blobs/$digest" | wc -c
done
Usage:
$ ./download-image.sh hello-world
$ ./download-image.sh library/hello-world
$ ./download-image.sh docker.io/library/hello-world
$ ./download-image.sh myregistry.com/hello-world testuser:testpassword
$ ./download-image.sh localhost:5000/hello-world
Or to be more precise it retrieves the image config and the layers. The thing is, what are you going to do with them? You can try to create a tar archive that looks like the one produced by docker save. Which is what docker-drag basically does. But ideally for that you should know what exactly docker pull and docker save do.
To give you some links to the source code, docker pull (the server part) more or less starts here:
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image.go#L37
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image_routes.go#L78
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_pull.go#L54
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_pull.go#L130
https://github.com/moby/moby/blob/v20.10.18/distribution/pull.go#L52
Most of the relevant code is in distribution/pull_v2.go (the high-level part).
The code that does the http requests is in vendor/github.com/docker/distribution/registry/client/auth/session.go and vendor/github.com/docker/distribution/registry/client/repository.go.
docker save:
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image.go#L31
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image_routes.go#L160
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_exporter.go#L16
https://github.com/moby/moby/blob/v20.10.18/image/tarexport/save.go#L187
Most of the code is in image/tarexport/save.go.
But well, apparently docker-drag's developer didn't concern himself with it, and it (docker-drag) seems to work.
In case of private Docker Hub repositories you probably need to add -u user:password to the $realm?service=$service&scope=$scope request.
A couple of remarks about using the API:
A registry is a collection of repositories. Think GitHub. Generally repository names are of the form user/name (e.g. nginxproxy/nginx-proxy). But with official repositories (that all in fact start with library/) the first segment may be omitted (library/ruby -> ruby). Also there might be just one segment (e.g. in private registries), and occasionaly the first part alone is called a repository (example). Or the second.
A repository is a collection of images. Some of them are tagged. Untagged images usually come into being when you push a new version of an image using the same tag.
An old, but maybe somewhat up-to-date relevant article (at least the beginning).
For the Docker Hub's registry URL see this answer.
Don't expect everything in the spec to work. E.g. Docker Hub doesn't implement the /v2/_catalog route.
I cant start my docker-compose environment, because the port is allocated by something.
ERROR: for ***** Cannot start service *****: driver failed programming external connectivity on endpoint ***** (4314ec13837d41ca8ef1b7e1d8446ab8cfa96136539a75ac763ba1cf538ffdc1): Bind for 0.0.0.0:8881 failed: port is already allocated
Encountered errors while bringing up the project.
However, there is not such a container
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
But when I inspect the docker-compose network, I can see a container in the network
$ docker network inspect *****
[
{
"Name": "*****",
"Id": "56d09f51e8fe5a9ad11dd6cdaff7e89983f519fd1ecff08116c2e48a3b34ff32",
"Created": "2018-08-01T08:17:30.704352821Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"08a5defa5ceae7a5a5ac7ec85f5ecd0924ecb4e5eacca4f5752049e32a0f190c": {
"Name": "08a5defa5cea_*****",
"EndpointID": "5d081a19b5ffe4e893965a4306668059900812532cc4995e168ce017f2765e12",
"MacAddress": "02:42:ac:12:00:06",
"IPv4Address": "172.18.0.6/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "*****",
"com.docker.compose.project": "docker"
}
}
]
Yet it is impossible to remove this container
docker rm -f *****
Error: No such container: *****
How to proceed?
I run this script to shut down all stuck containers which sometimes linger
#!/bin/bash
# ... bring down all containers
force_it="" # normal
# force_it=" -f " # when you see "device or resource busy"
for curr_container_id in $( docker ps -q ); do
echo curr_container_id $curr_container_id
echo
echo "docker stop $curr_container_id "
docker stop $curr_container_id
echo
echo "docker rm $force_it $curr_container_id "
docker rm $force_it $curr_container_id
done
for curr_container_id in $( docker ps -q -a ); do
echo curr_container_id $curr_container_id
echo
echo "docker stop $curr_container_id "
docker stop $curr_container_id
echo
echo "docker rm $force_it $curr_container_id "
docker rm $force_it $curr_container_id
done
it works 99% of the time ... if not near the top you can toggle the force parm and re-run it ... this allows you to avoid having to bounce any docker server
The newest docker registry/engine have supported "manifest list" feature to allow user referencing images with different CPU architectures, OSes and other characteristics, by solo entry in registry.
So saying I have a legacy x86-only image in my repository ,and unfortunately it'd been altered during previous running as container (successive docker commits) which means no Dockerfile available. Is there a way I can convert this x86-only image to support manifest list without rebuilding it?
Docker can export the contents of a container started from the image. Then a new image can be created FROM scratch that ADDs the contents back.
Steps
docker export will create a tar file of the complete contents of a container created from the image.
$ CID=$(docker create myimage)
$ docker export -o myimage.tar $CID
$ docker rm $CID
Build a new Dockerfile FROM scratch that ADDs the exported contents tar file back.
FROM scratch
ADD myimage.tar /
Any extended meta data for Dockerfile, like ENTRYPOINT, CMD or VOLUMES, can be queried via inspect or history:
$ docker image inspect myimage -f '{{json .Config}}' | jq
{
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"27017/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"GOSU_VERSION=1.10",
"JSYAML_VERSION=3.10.0",
"GPG_KEYS=2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5",
"MONGO_PACKAGE=mongodb-org",
"MONGO_REPO=repo.mongodb.org",
"MONGO_MAJOR=3.6",
"MONGO_VERSION=3.6.3"
],
"Cmd": [
"mongod"
],
"ArgsEscaped": true,
"Image": "sha256:bac19e2cfd49108534b108c101a68a2046090d25da581ae04dc020aac93b4e31",
"Volumes": {
"/data/configdb": {},
"/data/db": {}
},
"WorkingDir": "",
"Entrypoint": [
"docker-entrypoint.sh"
],
"OnBuild": [],
"Labels": null
}
or
docker image history myimage --no-trunc
I want to get some basic information about the published versions/tags of a docker image, to know what image:tag's I can pull. I would also like to see the time that each tag was most recently published.
Is there a way to do this on the command line?
Docker version 1.10.2, build c3959b1
Basically looking for the equivalent of npm info {pkg} for a docker image.
Not from the command line. You have docker search but it only returns a subset of the data you want, and only for the image with the :latest tag:
> docker search sixeyed/hadoop-dotnet
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
sixeyed/hadoop-dotnet Hadoop with .NET Core installed 1 [OK]
If you want more detail, you'll need to use the registry API, but it only has a catalog endpoint for listing repositories, the issue for search is still open.
Assuming you know the repository name, you can navigate the API - first you need an auth token:
> curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:sixeyed/hadoop-dotnet:pull"
{"token":"eyJhbG...
Then you pass the token to subsequent requests, e.g. to list the tags:
> curl --header "Authorization: Bearer eyJh..." https://index.docker.io/v2/sixeyed/hadoop-dotnet/tags/list
{"name":"sixeyed/hadoop-dotnet","tags":["2.7.2","latest"]}
And then get all the information about one image by its repository name and tag:
> curl --header "Authorization: Bearer eyJh..." https://index.docker.io/v2/sixeyed/hadoop-dotnet/manifests/latest
I want to get some basic information about the published versions/tags of a docker image, to know what image:tag's I can pull.
This is from the tags/list API. Here's a small script that does it:
#!/bin/sh
ref="${1:-library/ubuntu:latest}"
repo="${ref%:*}"
tag="${ref##*:}"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
curl -H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/tags/list" | jq .
Running that looks like:
$ ./tags-v2.sh library/ubuntu:latest
{
"name": "library/ubuntu",
"tags": [
"10.04",
"12.04",
"12.04.5",
"12.10",
"13.04",
"13.10",
"14.04",
"14.04.1",
...
I would also like to see the time that each tag was most recently published.
You likely want to pull the image config for this. First you need to pull the manifest, parse the config descriptor, then pull that blob for the config json. Here's a script that will do that for docker images:
#!/bin/sh
ref="${1:-library/ubuntu:latest}"
repo="${ref%:*}"
tag="${ref##*:}"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
digest=$(curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
-H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/manifests/${tag}" \
| jq -r .config.digest)
curl -H "Accept: application/vnd.docker.container.image.v1+json" \
-H "Authorization: Bearer $token" \
-s -L "https://registry-1.docker.io/v2/${repo}/blobs/${digest}" | jq .
And an example of running that script:
$ ./get-config-v2.sh library/ubuntu:latest
{
"architecture": "amd64",
"config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"bash"
],
"Image": "sha256:6c18a628d47eacf574eb93da2324293a0e6c845084cca2ea13efaa3cee4d0799",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"container": "249e88be79ad9986a479c71c15a056946ae26b0c54c1f634f115be6d5f9ba1c8",
"container_config": {
"Hostname": "249e88be79ad",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"bash\"]"
],
"Image": "sha256:6c18a628d47eacf574eb93da2324293a0e6c845084cca2ea13efaa3cee4d0799",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"created": "2021-10-16T00:37:47.578710012Z",
"docker_version": "20.10.7",
"history": [
{
"created": "2021-10-16T00:37:47.226745473Z",
"created_by": "/bin/sh -c #(nop) ADD file:5d68d27cc15a80653c93d3a0b262a28112d47a46326ff5fc2dfbf7fa3b9a0ce8 in / "
},
{
"created": "2021-10-16T00:37:47.578710012Z",
"created_by": "/bin/sh -c #(nop) CMD [\"bash\"]",
"empty_layer": true
}
],
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:9f54eef412758095c8079ac465d494a2872e02e90bf1fb5f12a1641c0d1bb78b"
]
}
}
Note that this will only give you details from the build tooling, the registry itself does not track when an image was uploaded to that registry from any OCI APIs (some registries may add their own metadata, but this would be registry specific).
To handle more authentication types, different kinds of images (OCI vs Docker), and similar, I've packaged these commands and more into regctl in the regclient project. Similar projects exist from Google's container registry crane command, and RedHat's skopeo, each giving access to registries from the command line:
$ regctl tag ls ubuntu
10.04
12.04
12.04.5
12.10
13.04
13.10
14.04
14.04.1
14.04.2
14.04.3
14.04.4
14.04.5
...
$ regctl image config ubuntu
{
"created": "2021-10-16T00:37:47.578710012Z",
"architecture": "amd64",
"os": "linux",
"config": {
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"bash"
]
},
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:9f54eef412758095c8079ac465d494a2872e02e90bf1fb5f12a1641c0d1bb78b"
]
},
"history": [
{
"created": "2021-10-16T00:37:47.226745473Z",
"created_by": "/bin/sh -c #(nop) ADD file:5d68d27cc15a80653c93d3a0b262a28112d47a46326ff5fc2dfbf7fa3b9a0ce8 in / "
},
{
"created": "2021-10-16T00:37:47.578710012Z",
"created_by": "/bin/sh -c #(nop) CMD [\"bash\"]",
"empty_layer": true
}
]
}
If you don't want to use a token:
curl -L -s 'https://registry.hub.docker.com/v2/repositories/<$repo_name>/tags?page=<$page_nb>&page_size=<$page_size>' | jq '."results"[]["name"]'
Where you can get <$repo_name> with:
docker search <$expression>
For example:
$ docker search piwigo
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
linuxserver/piwigo A Piwigo container, brought to you by LinuxS… 152
mathieuruellan/piwigo Easy deployment of piwigo for my personal us… 19 [OK]
lsioarmhf/piwigo 3
hg8496/piwigo 2 [OK]
[…]
$ curl -L -s 'https://registry.hub.docker.com/v2/repositories/linuxserver/piwigo/tags?page=1&page_size=10' | jq '."results"[]["name"]'
"latest"
"12.0.0"
"version-12.0.0"
"12.0.0-ls137"
"arm64v8-12.0.0"
"arm32v7-12.0.0"
"amd64-12.0.0"
"arm64v8-version-12.0.0"
"arm32v7-version-12.0.0"
"amd64-version-12.0.0"
Source: Démarrer avec les containers Docker
I know that I can use this command $ docker images --tree docker history to view the layers of a Docker image, but how do I do that for images on Docker Hub without pulling it? This is so that I know what is on an image before I download it.
E.g., for the Tomcat repo, https://registry.hub.docker.com/_/tomcat/, the webpage doesn't seem to show what is on the image. I have to look at the Dockerfile on Github to find out.
Update
I see this repo https://registry.hub.docker.com/u/tutum/tomcat/ has more tabs. The "Dockerfile" tab shows how it is created, but only seems to show the latest version. Is there no choice to view the file for other tags?
Docker Hub is quite limited at the moment and does not offer the feature you asked for.
When an image is configured to build from source at Docker Hub (an Automated Build) you can see what went into it, but when it is uploaded pre-built you have no information.
This is now possible by clicking into the image tag on Docker Hub.
Use microbadger, which comes with a few badges as well, eg:
So your tomcat image on https://microbadger.com/images/tomcat would look like:
You can get this for any registry by using the registry API. Within the image manifest is a pointer to the image config, and that config is a json blob that includes the history steps. Note that this history is not authoritative, and depending on the build tooling, could report different commands than what were actually used to build the image layers.
A shell script version of this using the auth for Docker Hub looks like:
#!/bin/sh
ref="${1:-library/ubuntu:latest}"
repo="${ref%:*}"
tag="${ref##*:}"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
digest=$(curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
-H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/manifests/${tag}" \
| jq -r .config.digest)
curl -H "Accept: application/vnd.docker.container.image.v1+json" \
-H "Authorization: Bearer $token" \
-s -L "https://registry-1.docker.io/v2/${repo}/blobs/${digest}" | jq .history
Behind the scenes, what's happening is the registry first resolves any manifest list to a single manifest for the default architecture, but you could pull any architecture manually:
$ regctl image manifest --list localhost:5000/library/alpine:latest --format '{{jsonPretty .}}'
{
"manifests": [
{
"digest": "sha256:69704ef328d05a9f806b6b8502915e6a0a4faa4d72018dc42343f511490daf8a",
"mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "amd64",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:18c29393a090ba5cde8a5f00926e9e419f47cfcfd206cc3f7f590e91b19adfe9",
"mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v6"
},
"size": 528
},
...
Then that platform specific manifest includes the config and layers:
$ regctl image manifest localhost:5000/library/alpine#sha256:69704ef328d05a9f806b6b8502915e6a0a4faa4d72018dc42343f511490daf8a --format '{{jsonPretty .}}'
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1471,
"digest": "sha256:14119a10abf4669e8cdbdff324a9f9605d99697215a0d21c360fe8dfa8471bab"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2814446,
"digest": "sha256:a0d0a0d46f8b52473982a3c466318f479767577551a53ffc9074c9fa7035982e"
}
]
}
And pulling the config blob shows all of the contents you may recognize from the history and inspect commands:
$ regctl blob get localhost:5000/library/alpine sha256:14119a10abf4669e8cdbdff324a9f9605d99697215a0d21c360fe8dfa8471bab | jq .
{
"architecture": "amd64",
"config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh"
],
"Image": "sha256:d3e0b6258ec2f725c19668f11ae5323c3b0245e197ec478424ec6a87935690eb",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"container": "330289c649db86f5fb1ae5bfef18501012b550adb0638b9193d4a3a4b65a2f9b",
"container_config": {
"Hostname": "330289c649db",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"/bin/sh\"]"
],
"Image": "sha256:d3e0b6258ec2f725c19668f11ae5323c3b0245e197ec478424ec6a87935690eb",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"created": "2021-08-27T17:19:45.758611523Z",
"docker_version": "20.10.7",
"history": [
{
"created": "2021-08-27T17:19:45.553092363Z",
"created_by": "/bin/sh -c #(nop) ADD file:aad4290d27580cc1a094ffaf98c3ca2fc5d699fe695dfb8e6e9fac20f1129450 in / "
},
{
"created": "2021-08-27T17:19:45.758611523Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/sh\"]",
"empty_layer": true
}
],
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68"
]
}
}
The regctl commands above come from my regclient project, but I believe you can do similar commands with skopeo and crane. The advantage of these tools over curl commands is handling the authentication that varies between registries (the bearer token from Hub won't work for other registries).
Lastly, the one-line answer using regctl is a formatted image config command, which performs all the above steps without needing jq:
$ regctl image config localhost:5000/library/alpine:latest --format '{{jsonPretty .History}}'
[
{
"created": "2021-08-27T17:19:45.553092363Z",
"created_by": "/bin/sh -c #(nop) ADD file:aad4290d27580cc1a094ffaf98c3ca2fc5d699fe695dfb8e6e9fac20f1129450 in / "
},
{
"created": "2021-08-27T17:19:45.758611523Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/sh\"]",
"empty_layer": true
}
]