I need to reach manifests of a lot of images on Docker hub, but every GET request for getting manifest is counted as a pull, as a result, I am restricted by the rate limits of docker hub. Is there a way to get the manifest with a HEAD request instead of GET from the API?
Edit: From Docker registry API documentation :
GET /v2//manifests/: Fetch the manifest identified by name and reference where reference can be a tag or digest. A HEAD request can also be issued to this endpoint to obtain resource information without receiving all data.
So I assume we can get related information with an HEAD request.
A shell script to do that looks like:
#!/bin/sh
ref="${1:-library/ubuntu:latest}"
sha="${ref#*#}"
if [ "$sha" = "$ref" ]; then
sha=""
fi
wosha="${ref%%#*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
tag="latest"
fi
apio="application/vnd.oci.image.index.v1+json"
apiol="application/vnd.oci.image.manifest.v1+json"
apid="application/vnd.docker.distribution.manifest.v2+json"
apidl="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
curl -H "Accept: ${apio}" -H "Accept: ${apiol}" -H "Accept: ${apid}" -H "Accept: ${apidl}" \
-H "Authorization: Bearer $token" \
-I -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}"
Note that the head request only shows the headers, and the most useful one is docker-content-digest so you can get the digest for a specific tag:
HTTP/1.1 200 OK
content-length: 1416
content-type: application/vnd.docker.distribution.manifest.list.v2+json
docker-content-digest: sha256:20fa2d7bb4de7723f542be5923b06c4d704370f0390e4ae9e1c833c8785644c1
docker-distribution-api-version: registry/2.0
etag: "sha256:20fa2d7bb4de7723f542be5923b06c4d704370f0390e4ae9e1c833c8785644c1"
date: Thu, 08 Sep 2022 17:45:42 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 100;w=21600
docker-ratelimit-source: 68.100.24.47
If you want to do this for other registries, you'll need to adjust the authentication. Both go-containerregistry's crane and regclient's regctl tools have image digest commands that handle the authentication and return just the digest.
I have a container image called troubleshooting. There are 5 different versions of the image in the my private repository. (Meaning 5 different image digests listed.)
The most recent image has a tag on it called latest. I need a way to get the other tags on that same image. I am doing this in a build script, so I would prefer to keep it through the REST API that docker has.
I tried used the /v2/myrepository/myimage/tags/list endpoint, but it lists all the tags without breaking them up by digest.
Is there a way to get tags broken up by digest? (Using the REST API?)
At present, you would need to perform a HEAD request on each tag, with the proper accept headers, to get the digest. There are suggestions to extend OCI's distribution-spec with a better tag API, but that will take some time to get approved and then implemented by the various registries.
Here's an example script for querying Docker Hub for either a docker manifest or manifest list (there are additional media types available for things like OCI image/index):
#!/bin/sh
ref="${1:-library/ubuntu:latest}"
sha="${ref#*#}"
if [ "$sha" = "$ref" ]; then
sha=""
fi
wosha="${ref%%#*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
tag="latest"
fi
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
curl -H "Accept: ${api}" -H "Accept: ${apil}" \
-H "Authorization: Bearer $token" \
-I -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}"
And the resulting headers that includes the docker-content-digest field:
$ ./manifest-v2-head.sh
HTTP/1.1 200 OK
content-length: 1416
content-type: application/vnd.docker.distribution.manifest.list.v2+json
docker-content-digest: sha256:a0d9e826ab87bd665cfc640598a871b748b4b70a01a4f3d174d4fb02adad07a9
docker-distribution-api-version: registry/2.0
etag: "sha256:a0d9e826ab87bd665cfc640598a871b748b4b70a01a4f3d174d4fb02adad07a9"
date: Fri, 08 Oct 2021 18:04:26 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 100;w=21600
docker-ratelimit-source: 68.100.24.47
I have dockerified httpie via this Dockerfile:
FROM alpine:3.6
MAINTAINER Philipp Kretzschmar <philipp.kretzschmar#gmail.com>
ENV LAST_UPDATED=2017-06-01
ARG XDG_CACHE_HOME=/tmp/cache/
RUN apk update && \
apk add py-pip && pip install httpie && \
rm -rf /var/cache/apk/*
CMD ["--help"]
ENTRYPOINT ["/usr/bin/http"]
You can also pull the image directly:
docker pull k0pernikus/httpie-docker-alpine:1.0.0
Using httpie as a standalone program, I know it is recommended to pipe in a nested json payload via echo (simplified example):
$ echo '{"property":"value"}' | http PUT https://jsonplaceholder.typicode.com -vvv
And through the verbose output flag -vvv httpie shows that the payload was indeed sent in the request:
PUT /posts HTTP/1.1
Accept: application/json, */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 21
Content-Type: application/json
Host: jsonplaceholder.typicode.com
User-Agent: HTTPie/0.9.9
{
"property": "value"
}
(The jsonplaceholder URL will return 404, which is fine. This question's scope is only about sending the proper request.)
Now I thought that it would be equivalent to execute the command through docker via:
$ echo '{"property":"value"}' | docker run k0pernikus/httpie-docker-alpine:1.0.0 PUT https://jsonplaceholder.typicode.com -vvv
But the verbose flag reports that the request was send without the payload:
PUT / HTTP/1.1
Content-Length: 0
Accept-Encoding: gzip, deflate
Host: jsonplaceholder.typicode.com
Accept: */*
User-Agent: HTTPie/0.9.9
Connection: keep-alive
How can I pipe in the data to the docker run command?
You need to use the -i flag to docker run in order to keep stdin open. Compare:
$ echo hello | docker run --rm alpine cat
$
To:
$ echo hello | docker run -i --rm alpine cat
hello
$
Similar to the question "What´s the sha256 code of a docker image?", I would like to find the digest of a Docker image. I can see the digest when I download an image:
$ docker pull waisbrot/wait:latest
latest: Pulling from waisbrot/wait
Digest: sha256:6f2185daa4ab1711181c30d03f565508e8e978ebd0f263030e7de98deee5f330
Status: Image is up to date for waisbrot/wait:latest
$
Another question, What is the Docker registry v2 API endpoint to get the digest for an image has an answer suggesting the Docker-Content-Digest header.
I can see that there is a Docker-Content-Digest header when I fetch the manifest for the image:
$ curl 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:waisbrot/wait:pull' -H "Authorization: Basic ${username_password_base64}"
# store the resulting token in DT
$ curl -v https://registry-1.docker.io/v2/waisbrot/wait/manifests/latest -H "Authorization: Bearer $DT" -XHEAD
* Trying 52.7.141.30...
* Connected to registry-1.docker.io (52.7.141.30) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.docker.io
* Server certificate: RapidSSL SHA256 CA - G3
* Server certificate: GeoTrust Global CA
> GET /v2/waisbrot/wait/manifests/latest HTTP/1.1
> Host: registry-1.docker.io
> User-Agent: curl/7.43.0
> Accept: */*
> Authorization: Bearer LtVRw-etc-etc-etc
>
< HTTP/1.1 200 OK
< Content-Length: 4974
< Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
< Docker-Content-Digest: sha256:128c6e3534b842a2eec139999b8ce8aa9a2af9907e2b9269550809d18cd832a3
< Docker-Distribution-Api-Version: registry/2.0
< Etag: "sha256:128c6e3534b842a2eec139999b8ce8aa9a2af9907e2b9269550809d18cd832a3"
< Date: Wed, 07 Sep 2016 16:37:15 GMT
< Strict-Transport-Security: max-age=31536000
However, this header isn't the same. The pull command got me 6f21 and the header shows 128c. Further, the pull command doesn't work for that digest:
$ docker pull waisbrot/wait#sha256:128c6e3534b842a2eec139999b8ce8aa9a2af9907e2b9269550809d18cd832a3
Error response from daemon: manifest unknown: manifest unknown
whereas things work as I want when I have the correct digest:
$ docker pull waisbrot/wait#sha256:6f2185daa4ab1711181c30d03f565508e8e978ebd0f263030e7de98deee5f330 12:46 waisbrot#influenza
sha256:6f2185daa4ab1711181c30d03f565508e8e978ebd0f263030e7de98deee5f330: Pulling from waisbrot/wait
Digest: sha256:6f2185daa4ab1711181c30d03f565508e8e978ebd0f263030e7de98deee5f330
Status: Image is up to date for waisbrot/wait#sha256:6f2185daa4ab1711181c30d03f565508e8e978ebd0f263030e7de98deee5f330
What I'm looking for is a way to translate the latest tag (which changes all the time) into a fixed digest that I can reliably pull. But I don't want to actually pull it down in order to do this translation.
edit 2022-10-04:
# INPUT
REPO=waisbrot/wait
user=my-user
password=my-password
# Get TOKEN
username_password_base64=$(echo -n $user:$password | base64)
TOKEN=$(curl -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
-H "Authorization: Basic ${username_password_base64}" \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:waisbrot/wait:pull' \
| jq -r .token)
# GET Digest from v2 API
curl -s -D - -H "Authorization: Bearer $TOKEN" \
https://registry-1.docker.io/v2/waisbrot/wait/manifests/latest 2>&1 \
| grep docker-content-digest \
| cut -d' ' -f2
original answer:
For newer versions of Docker, the inspect command provides the correct value (requires the image to have been pulled as Jan Hudec has pointed out in the comments):
docker inspect --format='{{index .RepoDigests 0}}' waisbrot/wait
For older versions, fetch the value from the repository following this example with the main Docker repo:
curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
-H "Authorization: Basic ${username_password_base64}" \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:waisbrot/wait:pull'
Naive attempts to fetch that value fail because the default content-type being selected by the server is application/vnd.docker.distribution.manifest.v1+prettyjws (a v1 manifest) and you need to v2 manifest. Therefore, you need to set the Accept header to application/vnd.docker.distribution.manifest.v2+json.
This is how you do it today using a V2 manifest.
docker manifest inspect <REMOTE IMAGE>:<TAG> -v
Your output is JSON:
{
...
"Descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:d13e102941a9f7bd417440f62f9cb29de35f6acb13a26cbf6a34f4c7340f0b63",
"size": 3255,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
...
}
With 2 http requests, you can get it. The first one to get an authentication token, and the second to get the image digest list by architecture and variant:
token=$(curl --silent "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" | jq -r '.token')
curl -s --header "Accept: application/vnd.docker.distribution.manifest.list.v2+json" --header "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/$image/manifests/$tag" | jq -r '.manifests|.[]| "\(.digest) \(.platform.architecture) \(.platform.variant)"'
Example with:
image=library/nginx
tag=stable-alpine
sha256:8853c7e938c2aa5d9d7439e698f0e700f058df8414a83134a09fcbb68bb0707a amd64 null
sha256:dbcd23f95b94018fe72bfdb356e40f4ae8b95063883f3456fedaed1c02204ed4 arm v6
sha256:d3670edcd50bb07cae303767426adf9bc7ba0219736148d30e6f30dd4e08695c arm v7
sha256:0bcd76faa141e4fa37e875834b3994261e0cfc94b7233ac84896381315b845ca arm64 v8
sha256:da8e62ddb3fab89ff4fa0271dbe230f849ab53402a71338503952437dcda1026 386 null
sha256:269bf99e100294b6b75fbdecf7b4ddbef8b29ea0a953e2e904452a50dbc923ab ppc64le null
sha256:103da50956034c157abeffbc869e2e38a4fabbf913bed8ae6ae7c59e646b28a1 s390x null
I encountered a task recently that required viewing the sha256 digest without necessarily pulling the image. The tool skopeo makes the registry API calls so you don't need to pull the image.
For example,
$ skopeo inspect --creds "username:password" docker://waisbrot/wait:latest
You could then pipe this to jq if you want to get just the digest value.
$ skopeo inspect --creds "username:password" \
docker://waisbrot/wait:latest | jq -r '.Digest'
sha256:6f2185daa4ab1711181c30d03f565508e8e978ebd0f263030e7de98deee5f330
I realise this issue is answered however either I am missing something or the current version of AWS ECR registry service does not work as expected.
When trying to get the digest from AWS ECR using either HEAD and also trying to switch the content-type does not return a digest value that I can use to pull an image using the registry Api.
To get this digest you have to get the manifest for the tag you are interested in and calculate the sha256 of the response Json as is, including the formatting, without the signature section
I struggled with this also. Here is a C# (dotnet core 5.0) implementation if anyone is intersted:
/**
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:waisbrot/wait:pull" | jq -r .token)
curl -s -D - -H "Authorization: Bearer $TOKEN" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" https://index.docker.io/v2/waisbrot/wait/manifests/latest
*/
private string GetRemoteImageDigest(string image, string tag) {
using HttpClient client = new ();
var url = string.Format($"https://auth.docker.io/token?service=registry.docker.io&scope=repository:{image}:pull");
//var response = client.Send(new HttpRequestMessage(HttpMethod.Get, url));
var result = client.GetStringAsync(url);
var drt = JsonSerializer.Deserialize<DockerRegistryToken>(result.Result);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", drt.Token);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/vnd.docker.distribution.manifest.v2+json"));
var response = client.GetAsync(string.Format($"https://index.docker.io/v2/{image}/manifests/{tag}"));
var headers = response.Result.Headers;
IEnumerable<string> values;
headers.TryGetValues("Docker-Content-Digest", out values);
return values.FirstOrDefault();
}
DockerRegistryToken is defined as:
public class DockerRegistryToken{
[JsonPropertyName("token")]
public string Token { get; set; }
/// always null
[JsonPropertyName("access_token")]
public string AccessToken {get; set; }
[JsonPropertyName("expires_in")]
public int ExpiresInSeconds { get; set; }
[JsonPropertyName("issued_at")]
public DateTime IssuedAt { get; set; }
}
As mentioned in other answers, this digest did not match because you attempted to curl without an Accept header, and so the registry triggered a fallback to an older v1 image manifest:
< Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
< Docker-Content-Digest: sha256:128c6e3534b842a2eec139999b8ce8aa9a2af9907e2b9269550809d18cd832a3
You can query Hub with curl using a script like:
#!/bin/sh
ref="${1:-library/ubuntu:latest}"
sha="${ref#*#}"
if [ "$sha" = "$ref" ]; then
sha=""
fi
wosha="${ref%%#*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
tag="latest"
fi
cto="application/vnd.oci.image.index.v1+json"
ctol="application/vnd.oci.image.manifest.v1+json"
ctd="application/vnd.docker.distribution.manifest.v2+json"
ctdl="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -sL "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
curl -H "Accept: ${cto}" -H "Accept: ${ctol}" -H "Accept: ${ctd}" -H "Accept: ${ctdl}" \
-H "Authorization: Bearer $token" \
-I -sL "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}"
However, that's limited since it's specific to Hub, and you still need to parse the headers. From the skopeo output, it's still pulling the entire manifest rather than a HEAD request, which will count against Hub rate limits.
Instead my two preferred tools for this are go-containerregistry/crane and regclient/regctl (I'm the author of the latter). Each has a digest command which automatically handles auth to different registries, includes the needed Accept headers, and parses the output to just the digest which is useful for scripting:
$ regctl image digest busybox
sha256:3b3128d9df6bbbcc92e2358e596c9fbd722a437a62bafbc51607970e9e3b8869
$ crane digest busybox
sha256:3b3128d9df6bbbcc92e2358e596c9fbd722a437a62bafbc51607970e9e3b8869
To avoid installing other tools, docker now has docker buildx imagetools inspect, but similar to skopeo, this is pulling the entire manifest rather than a HEAD request:
$ docker buildx imagetools inspect busybox
Name: docker.io/library/busybox:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest: sha256:3b3128d9df6bbbcc92e2358e596c9fbd722a437a62bafbc51607970e9e3b8869
...
$ docker buildx imagetools inspect busybox --format '{{json .}}' | jq -r .manifest.digest
sha256:3b3128d9df6bbbcc92e2358e596c9fbd722a437a62bafbc51607970e9e3b8869
example for reg which requires redirect (curl follow 302)
REGISTRY_ADDRESS='registry.access.redhat.com'
image='ubi8/openjdk-17-runtime'
curl --silent -L \
--header "Accept: application/vnd.docker.distribution.manifest.v2+json" \
"https://$REGISTRY_ADDRESS/v2/$image/manifests/latest" |
jq -r '.config.digest'
Following up on ByteFlinger's suggestion, which did not have an example, I tried this, and this is how to calculate it:
$ docker-ls tag -registry https://myregistry.net:5000
spicysomtam/zookeeper:latest
requesting manifest . done
repository: spicysomtam/zookeeper
tagName: latest
digest: sha256:bd5dd80253171e4dffccbea7c639c90a63d5424aa2d7fe655aea766405c83036
$ curl -ns -H "Accept:
application/vnd.docker.distribution.manifest.v2+json" -X GET
https://myregistry.net:5000/v2/spicysomtam/zookeeper/manifests/latest|sha256sum
bd5dd80253171e4dffccbea7c639c90a63d5424aa2d7fe655aea766405c83036 -
$ docker images --digests |grep zookeeper
myregistry.net:5000/spicysomtam/zookeeper latest sha256:bd5dd80253171e4dffccbea7c639c90a63d5424aa2d7fe655aea766405c83036 a983e71ca22d 29 hours ago 584MB
You can get this using docker inspect:
docker inspect --format='{{index .RepoDigests 0}}' ${IMAGE_NAME}
Docs: https://docs.docker.com/engine/reference/commandline/inspect/
This has been in place since at least v1.9.
I wanted to put a small library code to pub.dartlang.org
After calling pub publish, it asked me to Allow accessing my google account for sending files. After I allowed by going to the given browser link, it gave me this error in the command line:
Waiting for your authorization...
Authorization received, processing...
ProcessException: No such file or directory
Command: curl --dump-header /tmp/temp_dir1_CzMqQG/curl-headers --cacert /home/afsina/apps/dart/dart-sdk/util/pub/curl/ca-certificates.crt --request POST --location --max-redirs 5 --data-binary #- --header accept: --header user-agent: --header Content-Type: application/x-www-form-urlencoded; charset=UTF-8 --header content-length: 265 https://accounts.google.com/o/oauth2/token
here is the --verbose output
Looks great! Are you ready to upload your package (y/n)? y
IO : Read line: y
FINE: Loading OAuth2 credentials.
IO : Seeing if file /home/afsina/.pub-cache/credentials.json exists.
IO : File /home/afsina/.pub-cache/credentials.json does not exist.
FINE: No credentials found at /home/afsina/.pub-cache/credentials.json.
MSG : Pub needs your authorization to upload packages on your behalf.
| In a web browser, go to https://accounts.google.com/o/oauth2/auth?access_type=offline&approval_prompt=force&response_type=code&client_id=818368855108-8grd2eg9tj9f38os6f1urbcvsq399u8n.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A59097&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email
| Then click "Allow access".
|
| Waiting for your authorization...
MSG : Authorization received, processing...
FINE: Sending Curl request POST https://accounts.google.com/o/oauth2/token
IO : Begin create temp directory .
IO : End create temp directory .
IO : Spawning curl --dump-header /tmp/temp_dir1_RpLNu6/curl-headers --cacert /home/afsina/apps/dart/dart-sdk/util/pub/curl/ca-certificates.crt --request POST --location --max-redirs 5 --data-binary #- --header accept: --header user-agent: --header Content-Type: application/x-www-form-urlencoded; charset=UTF-8 --header content-length: 265 https://accounts.google.com/o/oauth2/token
IO : Spawning curl --dump-header /tmp/temp_dir1_RpLNu6/curl-headers --cacert /home/afsina/apps/dart/dart-sdk/util/pub/curl/ca-certificates.crt --request POST --location --max-redirs 5 --data-binary #- --header accept: --header user-agent: --header Content-Type: application/x-www-form-urlencoded; charset=UTF-8 --header content-length: 265 https://accounts.google.com/o/oauth2/token
ERR : ProcessException: No such file or directory
| Command: curl --dump-header /tmp/temp_dir1_RpLNu6/curl-headers --cacert /home/afsina/apps/dart/dart-sdk/util/pub/curl/ca-certificates.crt --request POST --location --max-redirs 5 --data-binary #- --header accept: --header user-agent: --header Content-Type: application/x-www-form-urlencoded; charset=UTF-8 --header content-length: 265 https://accounts.google.com/o/oauth2/token
I am guessing it cannot find the ca-certificates.crt file but not sure why.
Version Info: Dart SDK version 0.2.10.1_r16761
ubuntu linux 12.10 64 bit
My bad, this was because pub has "curl" dependency. After installing curl I could publish the lib.
If you run pub on Linux or Mac, it will assume curl is in your PATH and try to spawn it without an explicit path. On Windows, it will use the included curl binary. Our assumption here was that most *NIX users would have curl installed already.
This curl dependency should be going away soon anyway, but I'm glad you found the solution.