docker login fails with bad gateway. Potential V1 vs V2 issue? - docker

I received a docker registry location to login to and pull some images from at my organization. On attempts to login, I receive the following error:
docker login -u test-user internal.server.com:8080/test
Password:
Error response from daemon: Get https://internal.server.com:8080/v2/: Bad Gateway
I turned debug on for the docker daemon and tried again and receive the following errors:
level=debug msg="attempting v2 login to registry endpoint https://internal.server.com:8080/v2/"
level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://internal.server.com:8080/v2/: Bad Gateway"
level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: Get https://internal.server.com:8080/v2/: Bad Gateway" error...l.Error" module=api
If I am interpreting the error correctly, this means the my docker client is attempting to access the registry at v2 yet the registry doesn't seem to support v2? Or do I have this backwards (or some other issue entirely)? Thanks!

/v2 is part of the registry API. It will exist on all requests, so receiving a Bad Gateway from this indicates an error with the registry server.

Related

Login attempts to Nexus OSS Docker repo throwing 404

We are trying to set up a Docker repository in Nexus OSS (v3.3.2-02) in a Kubernetes cluster, and having issues logging in to it. We are intending to have a proxy set up for DockerHub, a private repo, and a group repo to tie the two together, using the below configurations
Hosted
Proxy
Group
giving us the following list:
But when I try to log in to the repository, it appears it's trying to forward me to a /v2 endpoint, which is throwing a 404 error:
> docker login -u <user> -p <pass> https://repo.myhost.com:443
Error response from daemon: login attempt to https://repo.myhost.com:443/v2/ failed with status: 404 Not Found
I would like to add that we have Maven and NPM repositories set up in this same instance and they're working, so it appears Nexus itself is OK, but there's something wrong with the Docker configuration.
I don't know why this request is trying to send me to the /v2 endpoint when trying to log in. What am I missing?
Docker requires very specific URL layout and does not allow for any context URL hence the need for Docker connectors to allow Docker client to connect to NXRM. Your screenshot shows you have configured Docker connector for your Docker hosted repository on port 444, but your terminal capture shows you're attempting to connect on port 443 which isn't your Docker connector port. The error message you have suggest your NXRM server indeed runs on port 443, but because of how Docker works you need to access it using port 444. Please try: docker login -u <user> -p <pass> https://repo.myhost.com:444 so it attempts to use your Docker connector port. Also, it's always a good idea to run the latest version of Nexus.
In an experiment I just ran (docker-machine, virtualbox, macOS), when the server was 1.13.1 (as was the docker cli), it made a graceful degradation from /v2 down to /v1, like so:
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.26/version"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.26/version"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.26/info"
level=debug msg="Calling POST /v1.26/auth"
level=debug msg="attempting v2 login to registry endpoint https://192.168.2.103:9999/v2/"
level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://192.168.2.103:9999/v2/: EOF"
level=debug msg="attempting v1 login to registry endpoint https://192.168.2.103:9999/v1/"
level=info msg="Error logging in to v1 endpoint, trying next endpoint: Get https://192.168.2.103:9999/v1/users/: dial tcp 192.168.2.103:9999: getsockopt: connection refused"
level=error msg="Handler for POST /v1.26/auth returned error: Get https://192.168.2.103:9999/v1/users/: dial tcp 192.168.2.103:9999: getsockopt: connection refused"
but after I upgraded the server to 17.06.0-ce (still with 1.13.1 cli), it only attempted /v2 and then quit:
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.30/version"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.30/info"
level=debug msg="Calling POST /v1.30/auth"
level=debug msg="attempting v2 login to registry endpoint https://192.168.2.103:9999/v2/"
level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://192.168.2.103:9999/v2/: tls: oversized record received with length 21584"
level=error msg="Handler for POST /v1.30/auth returned error: Get https://192.168.2.103:9999/v2/: tls: oversized record received with length 21584"
So the answer appears to be that one either needs to teach Nexus to respond correctly to the /v2 endpoints (as it really should be doing already), or downgrade the dockerd back down to a version that speaks the /v1 api if that is the behavior you're after
Not sure if this is going to help, but the browser based URL does not have port number in it, and could login with my credentials. Example browser based URL below.
https://nexus.mysite.net/
However I had to key in the following
docker login -u -p https://nexus.mysite.net/
I am greeted with the following
Error response from daemon: login attempt to https://nexus.mysite.net/v2/ failed with status: 404 Not Found
Giving the right port number did not show up the above error and I could login from the CLI as follows.
docker login -u the-user-name -p the-password https://nexus.mysite.net:7000
(in my case the correct port number was 7000).
Hope this helps.

Push docker image to Google Container Registry failure on Mac

I was trying to upload my image to Google Container Registry, but it return some error and I don't know how to troubleshooting.
$> gcloud docker -- push asia.gcr.io/dtapi-1314/web
The push refers to a repository [asia.gcr.io/dtapi-1314/web]
53ccd4e59f47: Retrying in 1 second
32ca8635750d: Retrying in 1 second
e5363ba7dd4d: Retrying in 1 second
d575d439624a: Retrying in 1 second
5c1cba20b78d: Retrying in 1 second
7198e99c156d: Waiting
6ca37046de16: Waiting
b8f2f07b3eab: Waiting
16681562a534: Waiting
92ea1d98cb79: Waiting
97ca462ad9ee: Waiting
unable to decode token response: read tcp 10.0.2.10:54718->74.125.23.82:443: read: connection reset by peer
I checked permission on my Mac.
$> gsutil acl get gs://asia.artifacts.dtapi-1314.appspot.com
It returned a list of correct permission.
I'd tested push on the cloud console, it works.
Does anyone have clue?
Thanks a lot if anyone could help. :)
Other troubleshooting
gcloud auth login
gcloud docker -- login -p $(gcloud auth print-access-token) -u _token https://asia.gcr.io
gsutil acl get gs://asia.artifacts.{%PROJECT_ID}.appspot.com
Add insecure-registry to dockerd startup command.
--insecure-registry asia.gcr.io
Might be the same cause
gcloud docker -- pull google/python
The error was
Error response from daemon: Get https://registry-1.docker.io/v2/google/python/manifests/latest: read tcp 10.0.2.15:37762->52.45.33.149:443: read: connection reset by peer
docker server log
DEBU[0499] Increasing token expiration to: 60 seconds
ERRO[0500] Error trying v2 registry: Get https://registry-1.docker.io/....../python/manifests/latest: read tcp 10.0.2.15:37762->52.45.33.149:443: read: connection reset by peer
ERRO[0500] Attempting next endpoint for pull after error: Get https://registry-1.docker.io/....../python/manifests/latest: read tcp 10.0.2.15:37762->52.45.33.149:443: read: connection reset by peer
DEBU[0500] Skipping v1 endpoint https://index.docker.io because v2 registry was detected
ERRO[0500] Handler for POST /v1.24/images/create returned error: Get https://registry-1.docker.io/....../python/manifests/latest: read tcp 10.0.2.15:37762->52.45.33.149:443: read: connection reset by peer
Environment
MacOS: 10.11.6
Docker Toolbox (on MAC)
Docker 1.12.3 (Git commit: 6b644ec, Built: Wed Oct 26 23:26:11 2016)
The root cause was stupid, but I'd like to update this for anyone who see this question. I found when I attached my computer to company's WIFI. Then It would work (Still some reset). The cable network of my company is mysterious broken to Google Container Registry. The cable network works for all other services (google/youtube/mobile services) we used but broken to Google Container Registry.
Seems like a permission issue. Try running
gcloud auth login
I remember running into a similar issue and this helped.

Error response from daemon: Unexpected status code 404

I am configuring docker registry on nexus 3 configuration. I Am running nexus behind apache and has https enabled.
On command line, when I do a docker search, I get the below error:
docker search my.nexus.net/ubantu
Error response from daemon: Unexpected status code 404
Here is the daemon log on debug mode:
DEBU[7519] Calling GET /images/search
INFO[7519] GET /v1.19/images/search?term=my.nexus.net%2Fubantu
DEBU[7519] pinging registry endpoint https://my.nexus.net/v0/
DEBU[7519] attempting v2 ping for registry endpoint https://my.nexus.net/v2/
DEBU[7519] hostDir: /etc/docker/certs.d/my.nexus.net
DEBU[7519] attempting v1 ping for registry endpoint https://my.nexus.net/v1/
DEBU[7519] hostDir: /etc/docker/certs.d/my.nexus.net
DEBU[7519] Error unmarshalling the _ping RegistryInfo: invalid character '<' looking for beginning of value
DEBU[7519] RegistryInfo.Version: ""
DEBU[7519] Registry standalone header: ''
DEBU[7519] RegistryInfo.Standalone: true
DEBU[7519] attempting v1 ping for registry endpoint https://my.nexus.net/v1/
DEBU[7519] hostDir: /etc/docker/certs.d/my.nexus.net
DEBU[7519] Error unmarshalling the _ping RegistryInfo: invalid character '<' looking for beginning of value
DEBU[7519] RegistryInfo.Version: ""
DEBU[7519] Registry standalone header: ''
DEBU[7519] RegistryInfo.Standalone: true
DEBU[7519] Endpoint https://my.nexus.net/v1/ is eligible for private registry. Enabling decorator.
DEBU[7519] Index server: https://my.nexus.net/v1/
DEBU[7519] hostDir: /etc/docker/certs.d/my.nexus.net
ERRO[7519] Handler for GET /images/search returned error: Unexpected status code 404
ERRO[7519] HTTP Error err=Unexpected status code 404 statusCode=500
If any one has any idea on it, please let me know.
From the logs it seems you try pining different versions of the registry endpoint. Did you use v1Enabled:falseoption on the repository configuration? It seems to get an error during v1 ping but still uses that endpoint. It's rather strange & unexpected behaviour.
I had the same problem.
After googling, it looks like 'docker search' uses the V1 API: see Issue https://github.com/docker/distribution/issues/206
So after I have enabled the V1 API on all docker registries of the group corresponding with the port, it works perfectly.
The docker search command use a v1 API. You have 2 alternatives :
Enable v1 API before using docker search
Use a curl like that: curl -X GET localhost:5000/v2/_catalog
To do pretty much anything in docker using NXRM3, you need to specify the port you are searching so the repository manager knows what repository you are looking for. If you just specify the root port (or in your case, looks like no port), NXRM3 has no idea which you are looking at.
So if your group is setup to use HTTPS connector 18075 try "docker search my.nexus.net:18075/ubantu"
Reference: http://books.sonatype.com/nexus-book/reference3/docker.html#docker-search
I used a VPN and it worked. Not psiphon.

Why does "docker login" fail in Docker Quickstart Terminal but work from within the default machine?

I've installed Docker Toolbox in Windows 8.1 and have been following the installation tutorial. When getting to the step where you create and push your own image, I got this error when I attempted to run docker login ... .
### VIA Docker Quickstart Terminal
### docker login --username=myuser --password="mypass" --email=myemail#gmail.com
time="2015-11-17T03:20:58.160803558Z" level=debug msg="Calling POST /v1.21/auth"
time="2015-11-17T03:20:58.160838971Z" level=info msg="POST /v1.21/auth"
time="2015-11-17T03:20:58.169033324Z" level=debug msg="hostDir: /etc/docker/certs.d/https:/registry-win-tp3.docker.io/v1"
time="2015-11-17T03:20:58.169071565Z" level=debug msg="pinging registry endpoint https://registry-win-tp3.docker.io/v1/"
time="2015-11-17T03:20:58.169084660Z" level=debug msg="attempting v1 ping for registry endpoint https://registry-win-tp3.docker.io/v1/"
time="2015-11-17T03:20:58.898542338Z" level=debug msg="Error unmarshalling the _ping PingResult: invalid character '<' looking for beginning of value"
time="2015-11-17T03:20:58.898803841Z" level=debug msg="PingResult.Version: \"\""
time="2015-11-17T03:20:58.898818084Z" level=debug msg="Registry standalone header: ''"
time="2015-11-17T03:20:58.898836197Z" level=debug msg="PingResult.Standalone: true"
time="2015-11-17T03:20:58.898853685Z" level=debug msg="attempting v1 login to registry endpoint https://registry-win-tp3.docker.io/v1/"
time="2015-11-17T03:20:59.478756938Z" level=error msg="Handler for POST /v1.21/auth returned error: Unexpected status code [403] : <html><body><h1>403 Forbidden</h1>\nRequest forbidden by administrative rules.\n</body></html>\n\n"
time="2015-11-17T03:20:59.478815334Z" level=error msg="HTTP Error" err="Unexpected status code [403] : <html><body><h1>403 Forbidden</h1>\nRequest forbidden by administrative rules.\n</body></html>\n\n" statusCode=500
Trying to solve the issue, I tried running docker login ... from within the Docker default VM. And there it works!
### VIA default virtual machine (192.168.99.100)
### docker login --username=myuser --password="mypass" --email=myemail#gmail.com https://index.docker.io/v1/
time="2015-11-17T03:20:46.053333255Z" level=debug msg="Calling POST /v1.21/auth"
time="2015-11-17T03:20:46.053404176Z" level=info msg="POST /v1.21/auth"
time="2015-11-17T03:20:46.082796012Z" level=debug msg="hostDir: /etc/docker/certs.d/https:/index.docker.io/v1"
time="2015-11-17T03:20:46.082930763Z" level=debug msg="pinging registry endpoint https://index.docker.io/v1/"
time="2015-11-17T03:20:46.082946790Z" level=debug msg="attempting v1 ping for registry endpoint https://index.docker.io/v1/"
time="2015-11-17T03:20:46.082959103Z" level=debug msg="attempting v1 login to registry endpoint https://index.docker.io/v1/"
I notice that they're using two different URLs and that the first one encounters a parsing error. The credentials are obviously correct since they work from within the VM, unless the two domains don't share users. Are the URLs or the response being mangled by MINGW64?
Update February 2016
PR 19891 "Enable cross-platforms login to Registry" is supposed to fixed the issue
Use a daemon-defined Registry URL for docker login.
This allows a Windows client interacting with a Linux daemon to properly use the default Registry endpoint instead of the Windows specific one.
It is in commit 19eaa71 (maybe for docker 1.10?)
This is reported both in docker/docker issue 15612 and docker/docker issue 18019
After some analysis of the source code I’ve detected that we have different registry URLs for Windows and UNIX.
Windows: https://registry-win-tp3.docker.io/v1/
Unix: https://index.docker.io/v1/
The Windows url comes from a recent PR 15417 with the comment:
// Currently it is a TEMPORARY link that allows Microsoft to continue
// development of Docker Engine for Windows.
So it is possible this url won't work (unless you are on a very recent Windows Server 2016)
There seems to be a workaround in docker/hub-feedback issues 473, which involves:
specifying the default index registry of docker io,
docker login --username=myuser --password=mypassword --email=myemail https://index.docker.io/v1/
WARNING: login credentials saved in C:\Users\myuser\.docker\config.json
Login Succeeded
modifying the config.json file created by the previous step, in order to add the same credentials for index.docker.io for the registry-win:
config.json:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "myhash",
"email": "myemail"
},
"https://registry-win-tp3.docker.io/v1/": {
"auth": "myhash",
"email": "mydomain"
}
}
}
After that, a docker push index.docker.io/myuser/myrepo:latest does work.

Could not authenticate with IBM Containers registry at registry.ng.bluemix.net

When I login to the IBM containers on Bluemix, I get the following error:
cf ic login
** Retrieving client certificates from IBM Containers
** Storing client certificates in /Users/triplez/.ice/certs Successfully retrieved client certificates
** Checking local docker configuration OK
** Authenticating with registry at registry.ng.bluemix.net
Could not authenticate with IBM Containers registry at registry.ng.bluemix.net
**** exit status 1
****Error response from daemon: invalid registry endpoint registry.ng.bluemix.net/v0/: unable to ping registry endpoint
registry.ng.bluemix.net/v0/
v2 ping attempt failed with error: Get registry.ng.bluemix.net/v2/: dial tcp: lookup
registry.ng.bluemix.net on 103.11.48.126:53: read udp
103.11.48.126:53: i/o timeout
v1 ping attempt failed with error: Get registry.ng.bluemix.net/v1/_ping: dial tcp: lookup registry.ng.bluemix.net on 103.11.48.126:53: read udp
103.11.48.126:53: i/o timeout. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add
--insecure-registry registry.ng.bluemix.net to the daemon's
arguments. In the case of HTTPS, if you have access to the registry's
CA certificate, no need for the flag; simply place the CA certificate
at /etc/docker/certs.d/registry.ng.bluemix.net/ca.crt
I've already uninstalled docker and reinstalled, giving me the same error. I've also deleted ~/.cf/config.json and ~/.ice/ice-cfg.ini and ~/.ice/certs/ and tried logging in again. I still receive the same error.
I've also tested this with ice cli with the same error.
EDITED
Tried this command:
ice login -a 'https://api.ng.bluemix.net' --registry 'registry.ng.bluemix.net' --host 'https://containers-api.ng.bluemix.net/v3/containers'
Got the same error:
Authentication with container cloud service at
containers-api.ng.bluemix.net/v3/containers completed
successfully
You can issue commands now to the container service
Proceeding to authenticate with the container cloud registry at
registry.ng.bluemix.net/v3
Error response from daemon: invalid registry endpoint
registry.ng.bluemix.net/v3/v0/:
unable to ping registry endpoint registry.ng.bluemix.net/v3/v0/
v2 ping attempt failed with error: Get registry.ng.bluemix.net/v3/v2/:
dial tcp: lookup registry.ng.bluemix.net on 192.168.0.1:53: read ump
192.168.0.1:53: i/o timeout
v1 ping attempt failed with error: Get registry.ng.bluemix.net/v3/v1/_ping: dial tcp: lookup registry.ng.bluemix.net on 192.168.0.1:53: read udp 192.168.0.1:53: i/o timeout. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add --insecure-registry registry.ng.bluemix.net to the daemon's arguments. In the case of
HTTPS, if you have access to the registry's CA certificate, no need
for the flag; simply place the CA certificate at
/etc/docker/certs.d/registry.ng.bluemix.net/ca.crt
docker is not available on your system or is not properly configured
Could not authenticate with cloud registry at registry.ng.bluemix.net/v3
You can still use IBM Containers but will not be able to run local docker containers, push, or pull images
It looks like the container cloud service host name is not correct.
Please try to run the ice login command as below:
ice login -a 'https://api.ng.bluemix.net' --registry 'registry.ng.bluemix.net' --host 'https://containers-api.ng.bluemix.net/v3/containers'
Based on your comment "boot2docker has been deprecated" I assume you are using Docker 1.8.
This version of Docker is not supported by IBM Containers on Bluemix yet.
Please see documentation in link below saying Docker 1.6 or 1.7 is required:
https://www.ng.bluemix.net/docs/containers/container_cli_ov.html
I know IBM Containers team is working to add support for Docker 1.8 as well. Please keep an eye on documentation for updates in the future.

Resources