I am running a script to build a Docker image. I need to push this image to 2 different repositories with different tags. For which I need to capture and store the build ID of the image.
Docker build documentation talks about a way to store it in a file using --iidfile.
Do we have any examples around the same?
Thank you.
Actually, you don't need image ID. Just use docker tag existing image in your build pipeline to create other "cloned" images. Example:
docker build -t repo1/image:tag1 .
docker tag repo1/image:tag1 repo2/image:tag2
docker push repo1/image:tag1
docker push repo2/image:tag2
Related
On a private registry (myregistry.com), say I have an image tagged with 'v1.2.3'. I then push it by:
docker push myregistry.com/myimage:v1.2.3
If I want to associate another tag, say 'staging', and push that tag to my registry, I can:
docker tag myregistry.com/myimage:v1.2.3 myregistry.com/myimage:staging
docker push myregistry.com/myimage:staging
Though this works, the second docker push still runs through each image, attempting to push it (albeit skipping upload). Is there a better way just to add a remote tag?
The way you've stated, docker tag ...; docker push ... is the best way to add a tag to an image and share it.
In the specific example you've given, both tags were in the same repo (myregistry.com/myimage). In this case you can just docker push myregistry.com/myimage and by default the docker daemon will push all the tags for the repo at the same time, saving the iteration over layers for shared layers.
You can use the same process (docker tag ...; docker push ...) to tag images between repositories as well, e.g.
docker tag myregistry.com/myimage:v1.2.3 otherregistry.com/theirimage:v2
docker push otherregistry.com/theirimage
You can achieve this with docker buildx imagetools create
docker buildx imagetools create myregistry.com/myimage:v1.2.3 --tag myregistry.com/myimage:staging
this will simply download the image manifest of myregistry.com/myimage:v1.2.3 and re-tag (and push) it as myregistry.com/myimage:staging
NOTE: this will also retain the multi-platform manifest list when you "re-tag" (e.g. when your image is build for both linux/arm64 and linux/amd64). Where as the conventional docker pull/push strategy will only retain the image manifest for the platform/architecture of the system you do the pull/push from.
pull/tag/push method will have time&network costs, you can just remotely tag your image with:
only for changing TAG the answer https://stackoverflow.com/a/38362476/8430173 works , but I wanted to change the repository name too.
by many thanks to this, I changed the repoName too!
(by help of his Github project):
1- get manifests (in v2 schema)
2- post every layer.digest in the new repo
3- post config.layer
4- put whole manifest to new repo
details:
1- GET manifest from reg:5000/v2/{oldRepo}/manifests/{oldtag} withaccept header:application/vnd.docker.distribution.manifest.v2+json
2- for every layer : POST reg:5000/v2/{newRepo}/blobs/uploads/?mount={layer.digest}&from={oldRepoNameWithaoutTag}
3- POST reg:5000/v2/{newRepo}/blobs/uploads/?mount={config.digest}&from={oldRepoNameWithaoutTag}
4- PUT reg:5000/v2/{newRepo}/manifests/{newTag} with content-type header:application/vnd.docker.distribution.manifest.v2+json and body from step 1 response
5- enjoy!
For single platform images, you can use
docker pull repo:oldtag
docker tag repo:oldtag repo:newtag
docker push repo:newtag
However, there are a few downsides.
You pull all the layers even if you don't need to run the image locally.
You are dereferencing multi-platform images to your local platform.
This can be done with curl, especially if you are in the same repository (when you go across repositories, you also need to copy all the blobs). However, that has two challenges of its own:
You need to accept all of the possible media types for the image you are pulling, track the media type you received, and use that same media type when pushing the manifest back to the registry. There are at least 6 media types I'm familiar with:
a signed and unsigned docker schema v1
the common manifest and manifest list in docker schema v2
the OCI image and index
If you go across repositories, you need to parse the manifest for the included blobs, which can be recursive for docker manifest lists and OCI indexes. You may be able to do a server side blob mount to avoid pulling and pushing the blob, but that will depend on the server support. And we're also going to see anonymous blob mounts come to registries, which allows a blob mount even if you don't know the source repo on that registry (useful for everyone pushing to a cloud registry with an image based on an official docker image from Docker Hub).
Authorization gets complicated, particularly if you have bearer tokens to request and maintain between commands.
The solution I'd recommend is using a tool that handles the registry API for you. I've been working on my own tooling for this, regclient, and there are other similar projects like Google's crane and RedHat's skopeo. These should each handle the media types, copying blobs when needed, and authorization issues that would complicate the curl command.
As an example with regclient's regctl command, you'd run:
regctl image copy repo:oldtag repo:newtag
With google's crane you just do
crane tag myregistry.com/myimage:v1.2.3 staging
It works with both docker images and OCI images, no image is downloaded locally and it even skips the layer verifications, as they are guaranteed to already be in the repository.
It's even available in a docker image: gcr.io/go-containerregistry/crane
Note that there are other similar tools, like regctl or skopeo
There is a simpler method with the new experimental manifest Docker commands. It only requires downloading and uploading the manifest file (JSON overview) of an image. The commands below have been tested with the GitLab registry. First build and push a Docker image in some previous stage:
docker build -t registry.gitlab.com/<group>/<project>/<image-name>:<tag-a> .
docker push registry.gitlab.com/<group>/<project>/<image-name>:<tag-a>
Then at a later stage where the image has not been pulled:
docker manifest create registry.gitlab.com/<group>/<project>/<image-name>:<tag-b> \
registry.gitlab.com/<group>/<project>/<image-name>:<tag-a>
docker manifest push registry.gitlab.com/<group>/<project>/<image-name>:<tag-b>
You can then pull the image with the new tag. The first step did not seem to work with public Docker Hub images, but any suggestions are welcome. Additionally, to see the manifest itself, run:
docker manifest inspect registry.gitlab.com/<group>/<project>/<image-name>:<tag-a>
There's a well-known approach to have docker images copied from one container registry to another. In case the original registry is dockerhub, the typical workflow would be something like this:
docker pull <image:tag>
docker tag <image:tag> <new-reg-url/uid/image:tag>
docker push <new-reg-url/uid/image:tag>
Now, how do you the above when dealing with images with multi-architecture layers?
As per the information in this link, you can rely on buildx to construct multi-arch images, and while doing that, you can also upload those to whichever repo you wish, but how do i do this without having to first build the images?
Looks like buildx cli has unnecessarily (?) coupled the uploading process with the building one. Any suggestions?
Thanks!
While the docker pull ...; docker tag ...; docker push ... syntax is the easy way to move images between registries, it has a couple drawbacks. First, as you've seen, is that it dereferences a multi-platform image to a single platform. And the second is that it pulls all layers to the docker engine even if the remote registry already has those layers, making it a bad method for ephemeral CI workers that would always need to pull every layer.
To do this, I prefer talking directly to the registry servers rather than the docker engine itself. You don't need the functionality from the engine to run the images, all you need is the registry API. Docker has documented the original registry API and OCI recently went 1.0 on the distribution-spec which should get us some standardization.
There's a variety of tooling based on those specs, from the docker engine itself and containerd, to skopeo, google's crane, and I've also been working on regclient. Doing this with regclient's regctl command looks like:
regctl image copy <source_image:tag> <target_image:tag>
And the result is the various layers, image config, manifests, and multi-platform manifest list will be copied between registries, but only for the layers that don't already exist on the target registry.
2022 (docker builtin) solution
It's posible to perform the copy using the not-well-documented built in command docker buildx imagetools create using --tag
# i.e.
OLD_TAG=registry.example.com/namespaced/repository/example-image:old-tag
NEW_TAG=registry.example.com/namespaced/repository/example-image:new-tag
# we can
docker buildx imagetools create --tag "$NEW_TAG" "$OLD_TAG"
Reference documentation
IMPORTANT NOTE: There is no support at the moment to perform this operation against different repositories. Given tags like
OLD_TAG=registry.example.com/namespaced/repository/example-image:latest
NEW_TAG=registry.example.com/other-repository/example-image:latest
You end with an error like
error: multiple repositories currently not supported
For this situation I'm going to test the actual accepted answer
As #laconbass has written, this can be done with docker buildx imagetools create. The ability to do this over multiple repos was added in this PR
docker buildx imagetools create -t <NEW-TAG> <OLD-TAG>
I'm in low cost project that we send to container registry (DigitalOcean) only latest image.
But all time, after running:
docker build .
Is generating the same digest, every time.
This is my script for build:
docker build .
docker tag {image}:latest registry.digitalocean.com/{company}/{image}:latest;
docker push registry.digitalocean.com/{company}/{image}
I tried:
BUILD_VERSION=`date '+%s'`;
docker build -t {image}:"$NOW" -t {image}:latest .
docker tag {image}:latest registry.digitalocean.com/{company}/{image}:latest;
docker push registry.digitalocean.com/{company}/{image}
but not worked.
Editing my answer, what David said is correct - the push with out the tag should pick up latest tag.
If you provide what you have in your local repository and the output of the above commands, it would shed more light to your problem.
Edit 2:
I think I have figured out on why:
Is generating the same digest, every time.
This means, although you are running your docker build - there has been no change to the underlying artifacts which are being packaged into the image and hence it results into the same digest.
Sometimes layers are cached but there are changes that aren't detected so you can delete the image or use 'docker system prune' to force clearing cache here
Usually according to docs In order to build a docker Image I need to follow these steps:
Create a Dockerfile for my application.
Run docker build . Dockerfile where the . is the context of my application
The using docker run run my image into a container.
Commit my image into a container
Then using docker push push the image into a container.
Though sometimes just launching the image into a container seems like a waste of time because I can tag my images using the parameter -t into the docker build command. So there's no need to commit a container as an image.
So is neseserily to commit a running container as an image?
You don't need to run and commit. docker commit allows you to create a new image from changes made on existing container.
You do need to build and tag your image in a way that will enable you to push it.
docker build -t [registry (defaults to docker hub)]/[your repository]:[image tag] [docker file context folder]
for example:
docker build -t my-repository/some-image:image-tag .
And then:
docker push my-repository/some-image:image-tag
This will build an image from a docker file found in the current folder (where you run the docker build command). The repository in this case is my-repository, the image name is some-image and it's tag is image-tag.
Also please note that you'll have to perform docker login with your credentials to docker hub before you are able to actually push the image.
You can also tag an existing image without rebuilding it. This is useful if you want to push an existing image to a different registry or if you want to create a different image tag. for example:
docker tag my-repository/some-image:image-tag localhost:5000/my-repository/some-image:image-tag
This will add a new tag to the image from the previous example. Note the registry part added (localhost:5000). If you call docker push on that tag (docker push localhost:5000/my-repository/some-image:image-tag) the image will be pushed to a registry found on localhost:5000 (of course you need the registry up and running before trying to push).
There's no need to do so. In order to prove that you can just tag the image and push it into the registry here's an example:
I made the following Dockerfile:
FROM alpine
RUN echo "Hello" > /usr/share/hello.txt
ENTRYPOINT cat /usr/share/hello.txt
Nothing special just generates a txt file and shows its content.
Then I can build my image using tags:
docker build . -t ddesyllas/dummy:201201241200 -t ddesyllas/dummy:201201241200
And then just push them to the registry:
$ docker push ddesyllas/dummy
The push refers to repository [docker.io/ddesyllas/dummy]
1aa99de3dbec: Pushed
6bc83681f1ba: Mounted from library/alpine
201908241504: digest: sha256:93e8407b1d52620aeadd769486ef1402b9e310018cae0972760f8c1a03377c94 size: 735
1aa99de3dbec: Layer already exists
6bc83681f1ba: Layer already exists
latest: digest: sha256:93e8407b1d52620aeadd769486ef1402b9e310018cae0972760f8c1a03377c94 size: 735
And as you can see from the output you can just build the tags and push it directly, good for your ci/cd pipeline. Though, generally speaking, you may need to launch the application into a container in order to do acceptance and other type of tests (eg. end-to-end tests).
On a private registry (myregistry.com), say I have an image tagged with 'v1.2.3'. I then push it by:
docker push myregistry.com/myimage:v1.2.3
If I want to associate another tag, say 'staging', and push that tag to my registry, I can:
docker tag myregistry.com/myimage:v1.2.3 myregistry.com/myimage:staging
docker push myregistry.com/myimage:staging
Though this works, the second docker push still runs through each image, attempting to push it (albeit skipping upload). Is there a better way just to add a remote tag?
The way you've stated, docker tag ...; docker push ... is the best way to add a tag to an image and share it.
In the specific example you've given, both tags were in the same repo (myregistry.com/myimage). In this case you can just docker push myregistry.com/myimage and by default the docker daemon will push all the tags for the repo at the same time, saving the iteration over layers for shared layers.
You can use the same process (docker tag ...; docker push ...) to tag images between repositories as well, e.g.
docker tag myregistry.com/myimage:v1.2.3 otherregistry.com/theirimage:v2
docker push otherregistry.com/theirimage
You can achieve this with docker buildx imagetools create
docker buildx imagetools create myregistry.com/myimage:v1.2.3 --tag myregistry.com/myimage:staging
this will simply download the image manifest of myregistry.com/myimage:v1.2.3 and re-tag (and push) it as myregistry.com/myimage:staging
NOTE: this will also retain the multi-platform manifest list when you "re-tag" (e.g. when your image is build for both linux/arm64 and linux/amd64). Where as the conventional docker pull/push strategy will only retain the image manifest for the platform/architecture of the system you do the pull/push from.
pull/tag/push method will have time&network costs, you can just remotely tag your image with:
only for changing TAG the answer https://stackoverflow.com/a/38362476/8430173 works , but I wanted to change the repository name too.
by many thanks to this, I changed the repoName too!
(by help of his Github project):
1- get manifests (in v2 schema)
2- post every layer.digest in the new repo
3- post config.layer
4- put whole manifest to new repo
details:
1- GET manifest from reg:5000/v2/{oldRepo}/manifests/{oldtag} withaccept header:application/vnd.docker.distribution.manifest.v2+json
2- for every layer : POST reg:5000/v2/{newRepo}/blobs/uploads/?mount={layer.digest}&from={oldRepoNameWithaoutTag}
3- POST reg:5000/v2/{newRepo}/blobs/uploads/?mount={config.digest}&from={oldRepoNameWithaoutTag}
4- PUT reg:5000/v2/{newRepo}/manifests/{newTag} with content-type header:application/vnd.docker.distribution.manifest.v2+json and body from step 1 response
5- enjoy!
For single platform images, you can use
docker pull repo:oldtag
docker tag repo:oldtag repo:newtag
docker push repo:newtag
However, there are a few downsides.
You pull all the layers even if you don't need to run the image locally.
You are dereferencing multi-platform images to your local platform.
This can be done with curl, especially if you are in the same repository (when you go across repositories, you also need to copy all the blobs). However, that has two challenges of its own:
You need to accept all of the possible media types for the image you are pulling, track the media type you received, and use that same media type when pushing the manifest back to the registry. There are at least 6 media types I'm familiar with:
a signed and unsigned docker schema v1
the common manifest and manifest list in docker schema v2
the OCI image and index
If you go across repositories, you need to parse the manifest for the included blobs, which can be recursive for docker manifest lists and OCI indexes. You may be able to do a server side blob mount to avoid pulling and pushing the blob, but that will depend on the server support. And we're also going to see anonymous blob mounts come to registries, which allows a blob mount even if you don't know the source repo on that registry (useful for everyone pushing to a cloud registry with an image based on an official docker image from Docker Hub).
Authorization gets complicated, particularly if you have bearer tokens to request and maintain between commands.
The solution I'd recommend is using a tool that handles the registry API for you. I've been working on my own tooling for this, regclient, and there are other similar projects like Google's crane and RedHat's skopeo. These should each handle the media types, copying blobs when needed, and authorization issues that would complicate the curl command.
As an example with regclient's regctl command, you'd run:
regctl image copy repo:oldtag repo:newtag
With google's crane you just do
crane tag myregistry.com/myimage:v1.2.3 staging
It works with both docker images and OCI images, no image is downloaded locally and it even skips the layer verifications, as they are guaranteed to already be in the repository.
It's even available in a docker image: gcr.io/go-containerregistry/crane
Note that there are other similar tools, like regctl or skopeo
There is a simpler method with the new experimental manifest Docker commands. It only requires downloading and uploading the manifest file (JSON overview) of an image. The commands below have been tested with the GitLab registry. First build and push a Docker image in some previous stage:
docker build -t registry.gitlab.com/<group>/<project>/<image-name>:<tag-a> .
docker push registry.gitlab.com/<group>/<project>/<image-name>:<tag-a>
Then at a later stage where the image has not been pulled:
docker manifest create registry.gitlab.com/<group>/<project>/<image-name>:<tag-b> \
registry.gitlab.com/<group>/<project>/<image-name>:<tag-a>
docker manifest push registry.gitlab.com/<group>/<project>/<image-name>:<tag-b>
You can then pull the image with the new tag. The first step did not seem to work with public Docker Hub images, but any suggestions are welcome. Additionally, to see the manifest itself, run:
docker manifest inspect registry.gitlab.com/<group>/<project>/<image-name>:<tag-a>