I'm trying to release a web process into a container by using only the API and not the Heroku CLI.
I know that i can do something like heroku container:release web -a <<myApp>>, but use the Heroku CLI is not an option since i'm trying to create a GH Action to do it.
After pushing correctly the image in the heroku registry, i tried to release the container doing as stated from the Heroku docs in the following way:
curl -X PATCH https://api.heroku.com/apps/$APP/formation \
-d '{
"updates": [
{
"type": "web",
"docker_image": "$(docker inspect $IMAGE --format={{.Id}})"
}
]
}' \
-H "Content-Type: application/json" \
-H "Accept: application/vnd.heroku+json; version=3.docker-releases" \
-H "Authorization: Bearer $HEROKU_API_KEY"
However it keeps giving me the same message:
{
"resource": "docker_image",
"id": "not_found",
"message": "Couldn't find that docker image."
}
What am i doing wrong?
NOTES:
In my GH Action i already tried doing heroku login -i and passing username and access token, but it keeps giving me Password: invalid option -s.
Even if i copy the HEROKU_API_KEY environment variable, doing docker push ... fails.
Copying a valid .netrc file fails too with the same error of using the HEROKU_API_KEY.
If someone gets stuck like me in this i'm posting another solution here.
Create the most simple heroku.yml in your repository.
Put the following content inside it:
build:
docker:
web: Dockerfile
For me, there's no need for run section, since the CMD into the Dockerfile is already what i need to get executed.
In this way, heroku platform understand that must build the image and runs the container after the build process has completed
There's a nexus setup running for docker registry. I'm struggling to delete old/unnecessary images from nexus setup using the APIs.So far I'm aware of below available APIs. There are 2 requirements:
Delete images older than 30 days.
Keep at least 5 tags of each image.
The delete api can only delete using the digest of the images but I"m not sure how to find exact one for the tags of images. Search api don't seem to work for docker images. Can someone please help?
## Search api
https://help.sonatype.com/repomanager3/integrations/rest-and-integration-api/search-api?_ga=2.253346826.2007475959.1640178248-1042170715.1640178248#SearchAPI-SearchComponents
## Find all catalog images under docker registery
curl -u admin:adminPass -X "GET" nexus.example.com/v2/_catalog | jq
## Get all tags of an image
curl -u admin:adminPass -X "GET" nexus.example.com/v2/abc-web-service-prod/tags/list
## Get manifests
curl -u admin:adminPass -X "GET" "nexus.example.com/v2/abc-web-service-stage-2/manifests/5.2.6_1" | jq
## Delete by digest
curl -i -u admin:adminPass -X "DELETE" "nexus.example.com/v2/abc-web-service/manifests/sha256:8829ce7278c1151f61438dcfea20e3694fee2241a75737e3a8de31a27f0014a5"
Two things are missing from the "get manifests" example. First, if you include the http headers, you'll likely get the digest field, or you can skip the jq and pipe the result into a sha256sum to get the digest. But you also need to add an "Accept" header for the various media types of a manifest, otherwise the registry will convert it to an older schema v1 syntax which will not have the same digest. Here's an example that does the two v2 docker media types:
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
curl -H "Accept: ${api}" -H "Accept: ${apil}" \
-u admin:adminPass \
-I -s "nexus.example.com/v2/abc-web-service-stage-2/manifests/5.2.6_1"
The next issue you'll run into with your policy is the 30 day requirement. You can get the creation time on many images by pulling their image config blob (it's listed in the manifest), but that date will be when the image was created, not when it was pushed or last pulled. There have been suggestions to add API's to OCI to handle more metadata, but we're still a ways off from that, and further still to get registry providers to implement them. So you'd end up deleting things that are likely being used. Even the 5 tag rule can be problematic if several new tags are created working through bugs in CI and you age out the image currently deployed in production.
With that all said, some tooling that I work on called regclient may help. The regctl command gives you a way to script this in a shell, e.g.:
#!/bin/sh
registry="nexus.example.com"
cutoff="$(date -d -30days '+%s')"
for repo in $(regctl repo ls "$registry"); do
# The "head -n -5" ignores the last 5 tags, but you may want to sort that list first.
for tag in $(regctl tag ls "$registry/$repo" | head -n -5); do
# This is the most likely command to fail since the created timestamp is optional, may be set to 0,
# and the string format might vary.
# The cut is to remove the "+0000" that breaks the "date" command.
created="$(regctl image config "$registry/$repo:$tag" --format '{{.Created}}' | cut -f1,2,4 -d' ')"
createdSec="$(date -d "$created" '+%s')"
# both timestamps are converted to seconds since epoc, allowing numeric comparison
if [ "$createdSec" -lt "$cutoff" ]; then
# next line is prefixed with echo for debugging, delete the echo to run the tag delete command
echo regctl tag rm "$registry/$repo:$tag"
fi
done
done
Note that I'm using "regctl tag rm" above, which is different from an image manifest delete you're seeing in the API. This will attempt to do an OCI tag delete API first, which likely isn't supported by your registry. It falls back to pushing a dummy manifest and deleting that. The alternative of deleting the current manifest the tag points to is you may delete more tags than intended (you could have 5 tags all pointing to the same manifest).
If you want to further automate this, regbot in that same repo lets you build a policy and run it on a schedule to constantly cleanup old images according to your rules.
In addition to regclient, there's also crane and skopeo that may also help in this space, but the features of each of these will vary.
I found a great solution to this.
https://github.com/andrey-pohilko/registry-cli
1. Create a docker image [name: registry-cli:1.0.1] using below Dockerfile
ADD requirements-build.txt /
RUN pip install -r /requirements-build.txt
ADD registry.py /
ENTRYPOINT ["/registry.py"]
2. Use below command to list down all images:tags in your private nexus registry.
docker run --rm registry-cli:1.0.1 -l admin:adminPass -r http://nexus.example.com
3. To get all tags of a particular image.
docker run --rm registry-cli:1.0.1 -l admin:adminPass -r http://nexus.example.com-i <name-of-the-image1> <name-of-the-image2>
4. To delete all old tags of a particular image but keep latest 10 tags.
docker run --rm registry-cli:1.0.1 -l admin:adminPass -r http://nexus.example.com -i <name-of-the-image1> --delete
5. To delete all the old tags of all the images in the repository but keep 10 latest tags of each image
docker run --rm registry-cli:1.0.1 -l admin:adminPass -r http://nexus.example.com --delete
6. If you wish to keep 20 images instead of 10 then use --num
docker run --rm registry-cli:1.0.1 -l admin:adminPass -r http://nexus.example.com --delete --num 20
7. Once you're done deleting the older tags of the images, run task "delete unused manifests and docker images"
8. Post step:7, run compaction task to reclaim the storage.
https://docs.docker.com/docker-hub/builds/#remote-build-triggers
Docker hub now has a build system in place. One of the ways to trigger a container to be built is using Remote build triggers. COmmands such as the following:
$ curl --data build=true -X POST https://registry.hub.docker.com/u/svendowideit/testhook/trigger/be579c82-7c0e-11e4-81c4-0242ac110020/
Their website shows a few paramters that can be passed in. But does not explain their meaning, nor do they provide a list all possible parameters.
What are all the possible parameters and what are their meanings?
it works for branches for sure, not sure about tags:
curl -H "Content-Type: application/json" \
--data '{"source_type": "Branch", "source_name": "develop"}' \
-X POST "$DOCKERHUB_TRIGGER";
Try source_type = Tag
the Docker Registry v2 has an API endpoint to delete an image
DELETE /v2/<name>/manifests/<reference>
https://github.com/docker/distribution/blob/master/docs/spec/api.md#deleting-an-image
However the doc says:
For deletes, reference must be a digest or the delete will fail.
Indeed, using a tag does not work and returns a 405 Operation Not Supported
The problem is, there doesn't seem to be any endpoint to get the digest of an image.
The endpoints to list images, and tags only list those.
Trying to get the manifest with
GET /v2/<name>/manifests/<reference>
using the tag as <reference>I see that a Docker-Content-Digest header is set with a digest which the doc says is
Docker-Content-Digest: Digest of the targeted content
for the request.
while the body contains a bunch of blobSum: <digest>
If I try using the Header digest value, with
GET /v2/<name>/manifests/<reference>
and the digest as <reference>, I get a 404.
the digest looks like: sha256:6367f164d92eb69a7f4bf4cab173e6b21398f94984ea1e1d8addc1863f4ed502
and I tried with and without the sha256 prefix. but no luck
So how am I supposed to get the digest of the image I want to delete, to delete it?
curl -u login:password -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://registry.private.com/v2/<name>/manifests/<tag>
json > config > digest
Not a trivial operation in Docker API right now but I hope this procedure helps:
Create a file and give it a name, for me it will be delete-image.sh:
#!/bin/bash
# Inspired by: https://gist.github.com/jaytaylor/86d5efaddda926a25fa68c263830dac1
set -o errexit
if [ -z "$1" ]
then
echo "Error: The image name arg is mandatory"
exit 1
fi
registry='localhost:5000'
name=$1
curl -v -sSL -X DELETE "http://${registry}/v2/${name}/manifests/$(
curl -sSL -I \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
"http://${registry}/v2/${name}/manifests/$(
curl -sSL "http://${registry}/v2/${name}/tags/list" | jq -r '.tags[0]'
)" \
| awk '$1 == "Docker-Content-Digest:" { print $2 }' \
| tr -d $'\r' \
)"
Give the permission to that file so that it can be executed;
sudo chmod u+x ./delete-image.sh
./delete-image.sh <your-image-name>
After deleting the image, collect the garbage;
docker exec -it registry.localhost bin/registry \
garbage-collect /etc/docker/registry/config.yml
Now delete the folder for that image (and I'm assuming that you created a volume previously);
sudo rm -rf ${HOME}/registry/docker/registry/v2/repositories/<your-image-name>
If you have not created a volume, you may have to enter the container to delete that folder. But, in any case, it's a good idea to restart the container;
docker restart registry.localhost
Procedure not recommended for production environments.
I hope that we will have better support for these operations natively in the Docker API in the future.
I'm using docker registry v1 and I'm interested in migrating to the newer version, v2. But I need some way to get a list of images present on registry; for example with registry v1 I can execute a GET request to http://myregistry:5000/v1/search? and the result is:
{
"num_results": 2,
"query": "",
"results": [
{
"description": "",
"name": "deis/router"
},
{
"description": "",
"name": "deis/database"
}
]
}
But I can't find on official documentation something similar to get a list of image on registry. Anybody knows a way to do it on new version v2?
For the latest (as of 2015-07-31) version of Registry V2, you can get this image from DockerHub:
docker pull distribution/registry:master
List all repositories (effectively images):
curl -X GET https://myregistry:5000/v2/_catalog
> {"repositories":["redis","ubuntu"]}
List all tags for a repository:
curl -X GET https://myregistry:5000/v2/ubuntu/tags/list
> {"name":"ubuntu","tags":["14.04"]}
If the registry needs authentication you have to specify username and password in the curl command
curl -X GET -u <user>:<pass> https://myregistry:5000/v2/_catalog
curl -X GET -u <user>:<pass> https://myregistry:5000/v2/ubuntu/tags/list
you can search on
http://<ip/hostname>:<port>/v2/_catalog
Get catalogs
Default, registry api return 100 entries of catalog, there is the code:
When you curl the registry api:
curl --cacert domain.crt https://your.registry:5000/v2/_catalog
it equivalents with：
curl --cacert domain.crt https://your.registry:5000/v2/_catalog?n=100
This is a pagination methond.
When the sum of entries beyond 100, you can do in two ways:
First: give a bigger number
curl --cacert domain.crt https://your.registry:5000/v2/_catalog?n=2000
Second: parse the next linker url
curl --cacert domain.crt https://your.registry:5000/v2/_catalog
A link element contained in response header:
curl --cacert domain.crt https://your.registry:5000/v2/_catalog
response header:
Link: </v2/_catalog?last=pro-octopus-ws&n=100>; rel="next"
The link element have the last entry of this request, then you can request the next 'page':
curl --cacert domain.crt https://your.registry:5000/v2/_catalog?last=pro-octopus-ws
If the response header contains link element, you can do it in a loop.
Get Images
When you get the result of catalog, it like follows:
{
"repositories": [
"busybox",
"ceph/mds"
]
}
you can get the images in every catalog:
curl --cacert domain.crt https://your.registry:5000/v2/busybox/tags/list
returns:
{"name":"busybox","tags":["latest"]}
The latest version of Docker Registry available from https://github.com/docker/distribution supports Catalog API. (v2/_catalog). This allows for capability to search repositories
If interested, you can try docker image registry CLI I built to make it easy for using the search features in the new Docker Registry distribution (https://github.com/vivekjuneja/docker_registry_cli)
This has been driving me crazy, but I finally put all the pieces together. As of 1/25/2015, I've confirmed that it is possible to list the images in the docker V2 registry ( exactly as #jonatan mentioned, above. )
I would up-vote that answer, if I had the rep for it.
Instead, I'll expand on the answer. Since registry V2 is made with security in mind, I think it's appropriate to include how to set it up with a self signed cert, and run the container with that cert in order that an https call can be made to it with that cert:
This is the script I actually use to start the registry:
sudo docker stop registry
sudo docker rm -v registry
sudo docker run -d \
-p 5001:5001 \
-p 5000:5000 \
--restart=always \
--name registry \
-v /data/registry:/var/lib/registry \
-v /root/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
-e REGISTRY_HTTP_DEBUG_ADDR=':5001' \
registry:2.2.1
This may be obvious to some, but I always get mixed up with keys and certs. The file that needs to be referenced to make the call #jonaton mentions above**, is the domain.crt listed above. ( Since I put domain.crt in /root, I made a copy into the user directory where it could be accessed. )
curl --cacert ~/domain.crt https://myregistry:5000/v2/_catalog
> {"repositories":["redis","ubuntu"]}
**The command above has been changed: -X GET didn't actually work when I tried it.
Note: https://myregistry:5000 ( as above ) must match the domain given to the cert generated.
We wrote a CLI tool for this purpose: docker-ls It allows you to browse a docker registry and supports authentication via token or basic auth.
Here is a nice little one liner (uses JQ) to print out a list of Repos and associated tags.
If you dont have jq installed you can use: brew install jq
# This is my URL but you can use any
REPO_URL=10.230.47.94:443
curl -k -s -X GET https://$REPO_URL/v2/_catalog \
| jq '.repositories[]' \
| sort \
| xargs -I _ curl -s -k -X GET https://$REPO_URL/v2/_/tags/list
Install registry:2.1.1 or later (you can check the last one, here) and use GET /v2/_catalog to get list.
https://github.com/docker/distribution/blob/master/docs/spec/api.md#listing-repositories
Lista all images by Shell script example:
https://gist.github.com/OndrejP/a2386d08e5308b0776c0
I had to do the same here and the above works except I had to provide login details as it was a local docker repository.
It is as per the above but with supplying the username/password in the URL.
curl -k -X GET https://yourusername:yourpassword#theregistryURL/v2/_catalog
It comes back as unformatted JSON.
I piped it through the python formatter for ease of human reading, in case you would like to have it in this format.
curl -k -X GET https://yourusername:yourpassword#theregistryURL/v2/_catalog | python -m json.tool
Here's an example that lists all tags of all images on the registry. It handles a registry configured for HTTP Basic auth too.
THE_REGISTRY=localhost:5000
# Get username:password from docker configuration. You could
# inject these some other way instead if you wanted.
CREDS=$(jq -r ".[\"auths\"][\"$THE_REGISTRY\"][\"auth\"]" .docker/config.json | base64 -d)
curl -s --user $CREDS https://$THE_REGISTRY/v2/_catalog | \
jq -r '.["repositories"][]' | \
xargs -I #REPO# curl -s --user $CREDS https://$THE_REGISTRY/v2/#REPO#/tags/list | \
jq -M '.["name"] + ":" + .["tags"][]'
Explanation:
extract username:password from .docker/config.json
make a https request to the registry to list all "repositories"
filter the json result to a flat list of repository names
for each repository name:
make a https request to the registry to list all "tags" for that "repository"
filter the stream of result json objects, printing "repository":"tag" pairs for each tag found in each repository
Using "/v2/_catalog" and "/tags/list" endpoints you can't really list all the images. If you pushed a few different images and tagged them "latest" you can't really list the old images! You can still pull them if you refer to them using digest "docker pull ubuntu#sha256:ac13c5d2...". So the answer is - there is no way to list images you can only list tags which is not the same
I wrote an easy-to-use command line tool for listing images in various ways (like list all images, list all tags of those images, list all layers of those tags).
It also allows you to delete unused images in various ways, like delete only older tags of a single image or from all images etc. This is convenient when you are filling your registry from a CI server and want to keep only latest/stable versions.
It is written in python and does not need you to download bulky big custom registry images.
If some on get this far.
Taking what others have already said above. Here is a one-liner that puts the answer into a text file formatted, json.
curl "http://mydocker.registry.domain/v2/_catalog?n=2000" | jq . - > /tmp/registry.lst
This looks like
{
"repositories": [
"somerepo/somecontiner",
"somerepo_other/someothercontiner",
...
]
}
You might need to change the `?n=xxxx' to match how many containers you have.
Next is a way to automatically remove old and unused containers.
This threads dates back a long time, the most recents tools that one should consider are skopeo and crane.
skopeo supports signing and has many other features, while crane is a bit more minimalistic and I found it easier to integrate with in a simple shell script.
Docker search registry v2 functionality is currently not supported at the time of this writing. See discussion since Feb 2015: "propose registry search functionality #206" https://github.com/docker/distribution/issues/206
I wrote a script, view-private-registry, that you can find: https://github.com/BradleyA/Search-docker-registry-v2-script.1.0
It is not pretty but it gets the information needed from the private registry.
Example of output from view-private-registry:
$ view-private-registry`
busybox:latest
gcr.io/google_containers/etcd:2.0.9
gcr.io/google_containers/hyperkube:v0.21.2
gcr.io/google_containers/pause:0.8.0
google/cadvisor:latest
jenkins:latest
logstash:latest
mongo:latest
nginx:latest
python:2.7
redis:latest
registry:2.1.1
stackengine/controller:latest
tomcat:7
tomcat:latest
ubuntu:14.04.2
Number of images: 16
Disk space used: 1.7G /mnt/three/docker-registry/registry-data
One liner bash to list all images with their tags:
curl --user user:pass https://myregistry.com/v2/_catalog | jq .repositories | sed -n 's/[ ",]//gp' | xargs -L1 -IIMAGE curl -s --user user:pass https://myregistry.com/v2/IMAGE/tags/list | jq '. as $parent | .tags[] | $parent.name + ":" + . '
Two lines to search for something in the image name:
search=my_container_part_name
curl --user user:pass https://registry.medworx.io/v2/_catalog | jq .repositories | sed -n '/'"$search"'/{s/[ ",]//gp;}' | xargs -L1 -IIMAGE curl -s --user user:pass https://registry.medworx.io/v2/IMAGE/tags/list | jq '. as $parent | .tags[] | $parent.name + ":" + . '
replace: user, pass and myregistry.com accordingly
uses curl, sed, xargs and jq and is hard to understand... but it does the job. It produces one call per image + 1.
If you can ssh or attach to the docker registry container, just browse the filesystem to look for things you want, like:
kubectl exec -it docker-registry-0 -- /bin/sh
ls /var/lib/registry/docker/registry/v2/repositories
ls /var/lib/registry/docker/registry/v2/repositories/busybox/_manifests/tags/
Since each registry runs as a container the container ID has an associated log file ID-json.log this log file contains the vars.name=[image] and vars.reference=[tag]. A script can be used to extrapolate and print these. This is perhaps one method to list images pushed to registry V2-2.0.1.
If your use-case is identifying only SIGNED and TRUSTED images for production, then this method is handy.
It parses a docker image repo for all SIGNED tags and strips away all the JSON formatting, puking-out only clean image tags. Which of course can be processed further according to your requirements.
Format of Command:
docker trust inspect imageName | grep "SignedTag" | awk -F'"' '{print $4}'
Examples using the nginx & Bitnami Docker repos:
docker trust inspect nginx | grep "SignedTag" | awk -F'"' '{print $4}'
docker trust inspect bitnami/java | grep "SignedTag" | awk -F'"' '{print $4}'
If there are no signed images then No signatures or cannot access imageName will be returned.
Example of a repo WITHOUT signed images (at the time of this writing) using the Wordpress Docker repo:
docker trust inspect wordpress | grep "SignedTag" | awk -F'"' '{print $4}'
If you want a nice web interface to your registry you can use this registry-browser docker image. This is useful if you just want to look around your registry, different repositories and tags.
If, the accepted answer here only returns a blank line, it is likely because of your ssl/tls cert on your registry server. Use the --insecure flag:
curl --insecure https://<registryHostnameOrIP>:5000/v2/_catalog