Team,
I am trying to upload Fluentd image to AWS ecr and getting error.
I have below images and tagged.
docker images | grep fluent
123.dkr.ecr.us-west-1.amazonaws.com/sre-tools/fluentd-ds latest b285a4690f19 5 months ago 209MB
123.dkr.ecr.us-west-1.amazonaws.com/sre-tools/fluentd-ds v1.8 b285a4690f19 5 months ago 209MB
fluentd v1.8-debian-1 b285a4690f19 5 months ago 209MB
aws ecr put-image --repository-name sre-tools/fluentd-ds:v1.8
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
aws help
aws <command> help
aws <command> <subcommand> help
aws: error: argument --image-manifest is required
from where should I get the image manifest? sorry this is my first time.
is manifest the sha256 of image? if yes I tried below
docker inspect b285a4690f19 | grep sha -i
"Id": "sha256:b285a4690f19ea48e817d0654df9d9225abb9c8ddb2fd3ffb9c1e00b3d7189ac",
"fluentd#sha256:a25d9ebbb344c40e1e84c173a466bc1f6e77a0233567fe778db1c6ffdeccf2e8"
"RUBY_DOWNLOAD_SHA256=d5d6da717fd48524596f9b78ac5a2eeb9691753da5c06923a6c31190abe01a62",
"Image": "sha256:66dc2c47c50cdf7f75ec68d1d027b4f139c4e5afcb2e0229ded10c659feafc43",
"RUBY_DOWNLOAD_SHA256=d5d6da717fd48524596f9b78ac5a2eeb9691753da5c06923a6c31190abe01a62",
"Image": "sha256:66dc2c47c50cdf7f75ec68d1d027b4f139c4e5afcb2e0229ded10c659feafc43",
"sha256:488dfecc21b1bc607e09368d2791cb784cf8c4ec5c05d2952b045b3e0f8cc01e",
"sha256:6bab58ebc554a7f95a80ee2dca3fbb05e05a5bceadab8b6d99d9eb41e0c7532f",
aws ecr put-image --repository-name sre-tools/fluentd-ds --image-manifest "sha256:a25d9ebbb344c40e1e84c173a466bc1f6e77a0233567fe778db1c6ffdeccf2e8"
An error occurred (InvalidParameterException) when calling the PutImage operation: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'
I had to provide a json file and not direct input
aws ecr put-image --repository-name sre-tools/fluentd-ds --image-manifest manifest.json"
also, got a workaround by using docker cli instead
docker push sre-tools/fluentd-ds:v1
Related
We have two URLs for our registry: one internal in our VPC and one external for customers pulling our images. We switched to a digest based reference system, so we pull images by their sha256 digests. Now we also want to give customers the option to install without internet access, so we export the images using docker save and then can load them using docker load. Unfortunately, we are having issues persisting the digest in this process.
How can I transfer a digest from one registry name to another when using docker tag?
An example might be illuminating here:
$ docker pull internal.registry.local/development/img:0.23.0#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267: Pulling from development/img
Digest: sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
Status: Image is up to date for internal.registry.local/development/img:0.23.0#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
internal.registry.local/development/img:0.23.0#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
gives me my image under the dev registry
$ docker image ls --digests
internal.registry.local/development/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
and is equipped with the digest
$ docker image inspect internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 | jq '.[] | { Id, RepoTags, RepoDigests }'
{
"Id": "sha256:dc9c4901ced19676f90c95d0f82c85ba97d15ba1c39d38ca9d692f3d3658bd43",
"RepoTags": [
"inernal.registry.local/development/img:0.23.0"
],
"RepoDigests": [
"internal.registry.local#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
]
}
If I now add another tag to that image under the same registry, the digest is transferred
$ docker tag internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 internal.registry.local/development/img:0.23.0-custom
$ docker image ls --digests
internal.registry.local/development/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
internal.registry.local/development/img 0.23.0-custom sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
$ docker image inspect internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 | jq '.[] | { Id, RepoTags, RepoDigests }'
{
"Id": "sha256:dc9c4901ced19676f90c95d0f82c85ba97d15ba1c39d38ca9d692f3d3658bd43",
"RepoTags": [
"internal.registry.local/development/img:0.23.0",
"internal.registry.local/development/img:0.23.0-custom",
],
"RepoDigests": [
"internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
]
}
But if I change the registry, the digest is lost
$ docker tag internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 external.example.org/production/img:0.23.0
$ docker image ls --digests
internal.registry.local/development/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
internal.registry.local/development/img 0.23.0-custom sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
external.example.org/production/img 0.23.0 <none> dc9c4901ced1 8 weeks ago 10.7GB
$ docker image inspect internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 | jq '.[] | { Id, RepoTags, RepoDigests }'
{
"Id": "sha256:dc9c4901ced19676f90c95d0f82c85ba97d15ba1c39d38ca9d692f3d3658bd43",
"RepoTags": [
"internal.registry.local/development/img:0.23.0",
"internal.registry.local/development/img:0.23.0-custom",
"external.example.org/production/img:0.23.0"
],
"RepoDigests": [
"internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
]
}
But mind you, the registry is actually the same, just under a different host name. So I can pull the image by digest from the external name and get the digest attached to it
$ docker pull external.example.org/production/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
external.example.org/production/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267: Pulling from production/img
Digest: sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
Status: Downloaded newer image for external.example.org/production/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
external.example.org/production/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
$ docker image ls --digests
internal.registry.local/development/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
internal.registry.local/development/img 0.23.0-custom sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
external.example.org/production/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
$ docker image inspect internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 | jq '.[] | { Id, RepoTags, RepoDigests }'
{
"Id": "sha256:dc9c4901ced19676f90c95d0f82c85ba97d15ba1c39d38ca9d692f3d3658bd43",
"RepoTags": [
"internal.registry.local/development/img:0.23.0",
"internal.registry.local/development/img:0.23.0-custom",
"external.example.org/production/img:0.23.0"
],
"RepoDigests": [
"internal.registry.local/development/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
"external.example.org/production/img#sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
]
}
How can I get the digest attached to the imgage when changing the registry by re-tagging without pulling by digest from the new registry name?
I've tried all permutations of
docker tag A/img#sha B/img:tag
docker tag A/img:tag B/img:tag
docker tag A/img:tag#sha B/img:tag
docker tag A/img#sha B/img#sha # error
no no avail.
The digest is made from json that is specific to how the image was pushed to the registry (different tools could change the whitespace or ordering of fields in the json to get a different sha256 hash). So docker doesn't know this value until you push or pull from the registry.
When you try to do a save/load, you'll see this issue again because the saved docker format has it's own manifest that doesn't have the same json format of what's pushed to the registry, so the digest will be regenerated when pushing to a registry after the docker load is run.
You can ship the images offline as a tar using the OCI Layout, which includes the original manifest, byte for byte, to avoid changing the digest. However, this won't load directly into the docker engine. The remote site will need to first import the images into their own registry. There are a few tools for importing and exporting between a registry and the OCI layout, including crane (from Google's go-containerregistry), skopeo (from RedHat), and regclient (from myself).
Using regclient's regctl command, that looks like:
regctl image export registry.example.org/repo#sha256... file.tar
# ... transport tar file to remote location ...
regctl image import client.example.com/repo#sha256... file.tar
Note, you typically still want to tag the image on the registry to avoid garbage collection of the untagged manifest. This won't affect the ability to pull by digest.
I’m encountering an error while login to the ecr from local machine (Windows).
I have used this command for login:
aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin 26224922****.dkr.ecr.ap-south-1.amazonaws.com
However I am getting the following error:
unknown flag: --password-stdin
See 'docker login --help'.
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
OSError: [Errno 22] Invalid argument
This used to work:
aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
Now, I get:
Error saving credentials: error storing credentials - err: exit status 1, out: `error storing credentials - err: exit status 1, out: `The user name or passphrase you entered is not correct.``
If I run just aws ecr get-login-password, I get a successful result, which suggests I have the right AWS access:
aws ecr get-login-password --region us-west-2 | base64 -d | jq '.'
{
"payload": "redacted",
"datakey": "redacted",
"version": "2",
"type": "DATA_KEY",
"expiration": redacted
}
This command that is failing is in a script that has been run successfully many times before and hasn't been changed. Has something changed in new versions of Docker that would break this? Or new versions of the Amazon CLI client? One possibility, is my AWS account privileges have changed, but I would have expected the above get-login-password command to have not succeeded in that case.
This login method is documented here, and I seem to be following instructions exactly:
https://docs.aws.amazon.com/cli/latest/reference/ecr/get-login-password.html
Is there anything I can do to troubleshoot this?
I am running moto server using the command
moto_server ecr -p 5000 -H 0.0.0.0
I created an ECR repo using the moto server with the command,
aws ecr create-repository --repository-name test --endpoint-url http://localhost:5000 --region us-east-1
Now can anybody please help on how to push a docker image to this ecr repo created using moto server?
After going through the code I found that the put-image command actually adds an image to the repository. In the case of ECR by AWS this command is used by AWS to add metadata of the image uploaded via the docker push command.
# aws ecr put-image --repository-name test --region us-east-1 --image-manifest test --endpoint-url http://localhost:5000 --image-tag v1
{
"image": {
"registryId": "012345678910",
"repositoryName": "test",
"imageId": {
"imageDigest": "sha256:a6698ae96409579a4f8ac96f5e5f276467b3f62d184c9e6db537daeedb9dd939",
"imageTag": "v1"
},
"imageManifest": "test"
}
}
# aws ecr list-images --repository-name test --region us-east-1 --endpoint-url http://localhost:5000
{
"imageIds": [
{
"imageDigest": "i don't know",
"imageTag": "v1"
}
]
}
Luckily my code doesn't need to push or pull image from the ECR repo. This method might not work if that is the case.
When attempting to push image to ECR, I always get 'no basic auth' error. ECR is in us-east-1.
This is login command
aws ecr get-login --region us-east-1 --no-include-email
I get response like this
docker login -u AWS -p eyJwYXl ...
I copy this response and run command, than I got this response
Login Succeeded
I assume that I logged in AWS and docker successfully, so I try to push image to ECR
docker push AWSID.dkr.ecr.us-east-1.amazonaws.com/repositoryname
Than response is like this
ed9f73170eb1: Preparing
f26c0d1885c7: Preparing
254cc70ba305: Preparing
6bfcbc08ecad: Preparing
0cb1addb8efc: Preparing
f9109426e338: Waiting
b7f99d06d826: Waiting
24d803cb9c1a: Waiting
25c4f6422338: Preparing
69b416623121: Waiting
0753f0746a0d: Waiting
a20143cd0986: Waiting
3028f693c3e6: Waiting
514a0f74b55d: Waiting
no basic auth credentials
To figure out what causes error, I checked ~/.docker/config.json. my config.json is like this.
{
"auths": {
"AWSPATH.dkr.ecr.us-east-1.amazonaws.com": {
"auth": "QVd..},
"https://index.docker.io/v1/": {}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/19.03.5 (darwin)"
}
It seems to be an Authentication issue. Reconfigure your AWS-CLI, maybe that will resolve your issue.
aws configure
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
try to login this way and push:
eval $(aws ecr get-login --no-include-email | sed ‘s|https://||’)
For me, it was simply missing permissions.
There are a number of permissions related to uploading such as
"ecr:UploadLayerPart",
"ecr:InitiateLayerUpload",
"ecr:CompleteLayerUpload"
"ecr:PutImage"
I have solved it by adding them to my agent's allowed permissions.
The problem was, jenkins server already had aws configured with different team's account. so I added region settings (we uses different region) and the problem solved.