cannot deploy docker image form AWS private registry - docker

Question

I am trying to push an app from a docker image hosted in the AWS Elastic Container Registry and am getting 500 error codes from the cloudfoundry API when trying to push. Am i doing something wrong or is there just an issue with the API currently? Any help is appreciated.
push command used (replaced real route, app and image name):
cf push dockerized-app --docker-image 300401118676.dkr.ecr.eu-central-1.amazonaws.com/my/image:latest --docker-username AWS --hostname my-dockerized-app -i 1 -m 1024M -k 1024M
cf-cli version:
cf version 6.34.1+bbdf81482.2018-01-17
This ist the standard log output i get:
Using docker repository password from environment variable CF_DOCKER_PASSWORD.
Pushing app dockerized-app to org ORG / space SPACE as someone#somewhere.ch...
Getting app info...
Creating app with these attributes...
+ name: dockerized-app
+ docker image: 300401118676.dkr.ecr.eu-central-1.amazonaws.com/my/image:latest
+ docker username: AWS
+ disk quota: 1G
+ instances: 1
+ memory: 1G
routes:
+ my-dockerized-app.scapp.io
Creating app dockerized-app...
Unexpected Response
Response code: 500
CC code: 0
CC error code:
Request ID: f0789965-19b1-4178-5cce-e42ff671a99b::6eb55c40-70de-4011-ad30-ee60aab54d82
Description: {
"error_code": "UnknownError",
"description": "An unknown error occurred.",
"code": 10001
}
FAILED
Here is the relevant log output with the -v flag set
Creating app with these attributes...
+ name: dockerized-app
+ docker image: 300401118676.dkr.ecr.eu-central-1.amazonaws.com/my/image:latest
+ docker username: AWS
+ disk quota: 1G
+ instances: 1
+ memory: 1G
routes:
+ my-dockerized-app.scapp.io
Creating app dockerized-app...
REQUEST: [2018-02-27T18:39:28+01:00]
POST /v2/apps HTTP/1.1
Host: api.lyra-836.appcloud.swisscom.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/6.34.1+bbdf81482.2018-01-17 (go1.9.2; amd64 darwin)
{
"disk_quota": 1024,
"docker_credentials": {
"password": "[PRIVATE DATA HIDDEN]",
"username": "AWS"
},
"docker_image": "300401118676.dkr.ecr.eu-central-1.amazonaws.com/my/image:latest",
"instances": 1,
"memory": 1024,
"name": "dockerized-app",
"space_guid": "07cead83-7db5-477e-83ca-f7bbee10e557"
}
RESPONSE: [2018-02-27T18:39:28+01:00]
HTTP/1.1 500 Internal Server Error
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Length: 99
Content-Type: application/json;charset=utf-8
Date: Tue, 27 Feb 2018 17:39:28 GMT
Expires: 0
Pragma: no-cache
Server: nginx
Strict-Transport-Security: max-age=16070400; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Vcap-Request-Id: 6c6acb3a-4ead-4f88-5d2c-e7d7f846b2af::0e919224-e372-46f1-8d70-19bf30f85145
X-Xss-Protection: 1; mode=block
{
"code": 10001,
"description": "An unknown error occurred.",
"error_code": "UnknownError"
}
Unexpected Response
Response code: 500
CC code: 0
CC error code:
Request ID: 6c6acb3a-4ead-4f88-5d2c-e7d7f846b2af::0e919224-e372-46f1-8d70-19bf30f85145
Description: {
"error_code": "UnknownError",
"description": "An unknown error occurred.",
"code": 10001
}
Seems to me like the docker registry username and password get picked up just fine (and yes they work).

Answer 1

From an operator perspective, it looks like you're hitting CloudFoundry's password limit of 1000 characters by using the Amazon Elastic Container Registry signed tokens (which are around 2000 chars):
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log.5.gz:
{"timestamp":1526311559.8367982,"message":"Request failed: 500:
{\"error_code\"=>\"UnknownError\", \"description\"=>\"An unknown
error occurred.\", \"code\"=>10001, \"test_mode_info\"=>
{\"description\"=>\"docker_password can be up to 1,000 characters\",
...
We filed the issue with the CC team: https://github.com/cloudfoundry/cloud_controller_ng/issues/1141

Answer 2

I'm not sure what version of Cloud Foundry your provider is running right now, but support for private docker registries (i.e. registries using HTTPS & basic auth) requires a fairly recent version of Cloud Foundry.
It definitely works in API versions 2.103 and later, as that's what we're running at Meshcloud right now and we have customer successfully using private registries ;-)
$ cf api
api endpoint: https://api.cf.eu-de-netde.msh.host
api version: 2.103.0
Disclaimer: I'm a co-founder at Meshcloud.