Login attempts to Nexus OSS Docker repo throwing 404 - docker

We are trying to set up a Docker repository in Nexus OSS (v3.3.2-02) in a Kubernetes cluster, and having issues logging in to it. We are intending to have a proxy set up for DockerHub, a private repo, and a group repo to tie the two together, using the below configurations
Hosted
Proxy
Group
giving us the following list:
But when I try to log in to the repository, it appears it's trying to forward me to a /v2 endpoint, which is throwing a 404 error:
> docker login -u <user> -p <pass> https://repo.myhost.com:443
Error response from daemon: login attempt to https://repo.myhost.com:443/v2/ failed with status: 404 Not Found
I would like to add that we have Maven and NPM repositories set up in this same instance and they're working, so it appears Nexus itself is OK, but there's something wrong with the Docker configuration.
I don't know why this request is trying to send me to the /v2 endpoint when trying to log in. What am I missing?

Docker requires very specific URL layout and does not allow for any context URL hence the need for Docker connectors to allow Docker client to connect to NXRM. Your screenshot shows you have configured Docker connector for your Docker hosted repository on port 444, but your terminal capture shows you're attempting to connect on port 443 which isn't your Docker connector port. The error message you have suggest your NXRM server indeed runs on port 443, but because of how Docker works you need to access it using port 444. Please try: docker login -u <user> -p <pass> https://repo.myhost.com:444 so it attempts to use your Docker connector port. Also, it's always a good idea to run the latest version of Nexus.

In an experiment I just ran (docker-machine, virtualbox, macOS), when the server was 1.13.1 (as was the docker cli), it made a graceful degradation from /v2 down to /v1, like so:
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.26/version"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.26/version"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.26/info"
level=debug msg="Calling POST /v1.26/auth"
level=debug msg="attempting v2 login to registry endpoint https://192.168.2.103:9999/v2/"
level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://192.168.2.103:9999/v2/: EOF"
level=debug msg="attempting v1 login to registry endpoint https://192.168.2.103:9999/v1/"
level=info msg="Error logging in to v1 endpoint, trying next endpoint: Get https://192.168.2.103:9999/v1/users/: dial tcp 192.168.2.103:9999: getsockopt: connection refused"
level=error msg="Handler for POST /v1.26/auth returned error: Get https://192.168.2.103:9999/v1/users/: dial tcp 192.168.2.103:9999: getsockopt: connection refused"
but after I upgraded the server to 17.06.0-ce (still with 1.13.1 cli), it only attempted /v2 and then quit:
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.30/version"
level=debug msg="Calling GET /_ping"
level=debug msg="Calling GET /v1.30/info"
level=debug msg="Calling POST /v1.30/auth"
level=debug msg="attempting v2 login to registry endpoint https://192.168.2.103:9999/v2/"
level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://192.168.2.103:9999/v2/: tls: oversized record received with length 21584"
level=error msg="Handler for POST /v1.30/auth returned error: Get https://192.168.2.103:9999/v2/: tls: oversized record received with length 21584"
So the answer appears to be that one either needs to teach Nexus to respond correctly to the /v2 endpoints (as it really should be doing already), or downgrade the dockerd back down to a version that speaks the /v1 api if that is the behavior you're after

Not sure if this is going to help, but the browser based URL does not have port number in it, and could login with my credentials. Example browser based URL below.
https://nexus.mysite.net/
However I had to key in the following
docker login -u -p https://nexus.mysite.net/
I am greeted with the following
Error response from daemon: login attempt to https://nexus.mysite.net/v2/ failed with status: 404 Not Found
Giving the right port number did not show up the above error and I could login from the CLI as follows.
docker login -u the-user-name -p the-password https://nexus.mysite.net:7000
(in my case the correct port number was 7000).
Hope this helps.

Related

Watchtower can't pull public image hosted on Scaleway (401 unauthorized)

For a school project, I have set up a watchtower to automatically update the frontend on update.
However, it seems that Watchtower cannot pull the image, it throws a 401 unauthorized error.
My docker-compose.yml is the following
version: '3'
services:
about_website:
image: rg.nl-ams.scw.cloud/csc648-team01/about_website
command: "-l 3000"
ports:
- "3000:3000"
labels:
com.centurylinklabs.watchtower.enable: true
com.centurylinklabs.watchtower.scope: about_website
watchtower:
image: containrrr/watchtower:1.5.3
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: --interval 5 --cleanup --label-enable --debug --scope about_website
labels:
com.centurylinklabs.watchtower.scope: about_website
The image is public and can be pulled using
docker pull rg.nl-ams.scw.cloud/csc648-team01/about_website:latest
latest: Pulling from csc648-team01/about_website
Digest: sha256:99c55eb14bdcc3724e2ca751bb2c6e0d7633077b2657b4a69e0a30be0093c1c4
Status: Image is up to date for rg.nl-ams.scw.cloud/csc648-team01/about_website:latest
rg.nl-ams.scw.cloud/csc648-team01/about_website:latest
However, Watchtower isn't able to pull it, I can see in logs that it throws a 401 error
time="2023-02-13T22:02:34Z" level=debug msg="Retrieving running containers"
2023-02-13T22:02:34.065154177Z time="2023-02-13T22:02:34Z" level=debug msg="Trying to load authentication credentials." container=/about_website-about_website-1 image="rg.nl-ams.scw.cloud/csc648-team01/about_website:latest"
2023-02-13T22:02:34.065371344Z time="2023-02-13T22:02:34Z" level=debug msg="No credentials for rg.nl-ams.scw.cloud found" config_file=/config.json
2023-02-13T22:02:34.065384885Z time="2023-02-13T22:02:34Z" level=debug msg="Got image name: rg.nl-ams.scw.cloud/csc648-team01/about_website:latest"
2023-02-13T22:02:34.065388719Z time="2023-02-13T22:02:34Z" level=debug msg="Checking if pull is needed" container=/about_website-about_website-1 image="rg.nl-ams.scw.cloud/csc648-team01/about_website:latest"
2023-02-13T22:02:34.065391969Z time="2023-02-13T22:02:34Z" level=debug msg="Building challenge URL" URL="https://rg.nl-ams.scw.cloud/v2/"
2023-02-13T22:02:34.717871469Z time="2023-02-13T22:02:34Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://api.scaleway.com/registry-internal/v1/regions/nl-ams/tokens\",service=\"registry\"" status="401 Unauthorized"
2023-02-13T22:02:34.718049427Z time="2023-02-13T22:02:34Z" level=debug msg="Checking challenge header content" realm="https://api.scaleway.com/registry-internal/v1/regions/nl-ams/tokens" service=registry
2023-02-13T22:02:34.718054969Z time="2023-02-13T22:02:34Z" level=debug msg="Setting scope for auth token" image=rg.nl-ams.scw.cloud/csc648-team01/about_website scope="repository:rg.nl-ams.scw.cloud/csc648-team01/about_website:pull"
2023-02-13T22:02:34.718060552Z time="2023-02-13T22:02:34Z" level=debug msg="No credentials found."
2023-02-13T22:02:35.429569719Z time="2023-02-13T22:02:35Z" level=debug msg="Parsing image ref" host=rg.nl-ams.scw.cloud image=csc648-team01/about_website normalized="rg.nl-ams.scw.cloud/csc648-team01/about_website:latest" tag=latest
2023-02-13T22:02:35.429658511Z time="2023-02-13T22:02:35Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://rg.nl-ams.scw.cloud/v2/csc648-team01/about_website/manifests/latest"
2023-02-13T22:02:36.083084511Z time="2023-02-13T22:02:36Z" level=debug msg="Could not do a head request for \"rg.nl-ams.scw.cloud/csc648-team01/about_website:latest\", falling back to regular pull." container=/about_website-about_website-1 image="rg.nl-ams.scw.cloud/csc648-team01/about_website:latest"
2023-02-13T22:02:36.085633136Z time="2023-02-13T22:02:36Z" level=debug msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://api.scaleway.com/registry-internal/v1/regions/nl-ams/tokens\\\",service=\\\"registry\\\",scope=\\\"repository:csc648-team01/about_website:pull\\\",error=\\\"invalid_token\\\"\"" container=/about_website-about_website-1 image="rg.nl-ams.scw.cloud/csc648-team01/about_website:latest"
2023-02-13T22:02:36.087838428Z time="2023-02-13T22:02:36Z" level=debug msg="Pulling image" container=/about_website-about_website-1 image="rg.nl-ams.scw.cloud/csc648-team01/about_website:latest"
2023-02-13T22:02:38.074009471Z
time="2023-02-13T22:02:38Z" level=debug msg="No new images found for /about_website-about_website-1"
2023-02-13T22:02:38.074042429Z time="2023-02-13T22:02:38Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=0 notify=no
EDIT I tried to host my image on Docker Hub and it works fine.
It seems the issue come from Scaleway :(

docker login fails with bad gateway. Potential V1 vs V2 issue?

I received a docker registry location to login to and pull some images from at my organization. On attempts to login, I receive the following error:
docker login -u test-user internal.server.com:8080/test
Password:
Error response from daemon: Get https://internal.server.com:8080/v2/: Bad Gateway
I turned debug on for the docker daemon and tried again and receive the following errors:
level=debug msg="attempting v2 login to registry endpoint https://internal.server.com:8080/v2/"
level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://internal.server.com:8080/v2/: Bad Gateway"
level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: Get https://internal.server.com:8080/v2/: Bad Gateway" error...l.Error" module=api
If I am interpreting the error correctly, this means the my docker client is attempting to access the registry at v2 yet the registry doesn't seem to support v2? Or do I have this backwards (or some other issue entirely)? Thanks!
/v2 is part of the registry API. It will exist on all requests, so receiving a Bad Gateway from this indicates an error with the registry server.

In Sonatype Nexus Docker proxy, how do I debug "Bad request" error?

I've been following the instructions to set up a Proxy Repository for Docker. I am intending to set up a proxy for Docker hub, that is for https://index.docker.io/.
My setup details on Nexus
Nexus version 3.36.0-01
Installed via docker-compose
Generated/installed self-signed cert
Using built-in https/jetty, NOT reverse proxy
http listening on port 80
https listening on port 443
My setup details on Nexus docker proxy repo
Configured for https, port 8443
Proxy remote storage: https://registry-1.docker.io
Proxy docker index: "use docker hub", pre-filled as https://index.docker.io/
Allowing anonymous docker pull
Enabled Docker Bearer Token Realm
Enabled docker v1 API
Enabled foreign layer caching
My setup details on Ubuntu docker client
Trusted self-signed cert in /etc/docker/certs.d
Trusted self-signed cert in /usr/local/share/ca-certificates + update-ca-certificates
Enabled Docker daemon debugging in /etc/docker/daemon.json
Enabled Docker proxy via httpsProxy in /home/myuser/.docker/config.json
Enabled Docker proxy via httpsProxy in /etc/systemd/system/docker.service.d/https-proxy.conf, reloaded/restarted Docker daemon
My test from the client
docker pull hello-world:latest
returns error Error response from daemon: Get https://registry-1.docker.io/v2/: Bad Request
In debug logs:
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.190545462Z" level=debug msg="Calling HEAD /_ping"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.190878019Z" level=debug msg="Calling GET /v1.40/info"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.213218413Z" level=debug msg="Calling POST /v1.40/images/create?fromImage=hello-world&tag=latest"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.213290250Z" level=debug msg="Trying to pull hello-world from https://registry-1.docker.io v2"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234803592Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: Bad Request"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234865780Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: Bad Request"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234976364Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://registry-1.docker.io/v2/: Bad Request"
Next Steps
I'm watching the logs on the server while this is happening. It shows no errors. However the client side seems to indicate the request is partly working.
I tried increasing org.apache.http.wire to DEBUG as per this other SO question/answer, but that also showed nothing.
How do I continue debugging?
If you examine the docker output you’ll notice it isn’t going to nexus, it is making the request to https://registry-1.docker.io. To pull from Nexus you need to prepend the host and port to the pull request.
docker pull hostname:8443/hello-world:latest
There isn’t any way in docker to have it default to a private registry btw, so you’ll always need to prepend host:port.

Running docker image hangs and then times out in the prod server

I have a docker image that I can run properly in my local VM. Everything runs fine.
I save the image and load it in the prod server.
I can see the image by using docker images
next I try to run it with docker run -p 9191:9191 myservice
It hangs and eventually times out.
The log shows the following:
time="2018-08-15T16:14:35.058232400-07:00" level=debug msg="HCSShim::CreateContainer succeeded id=b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152 handle=
39044064"
time="2018-08-15T16:14:35.058232400-07:00" level=debug msg="libcontainerd: Create() id=b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152, Calling start()"
time="2018-08-15T16:14:35.058232400-07:00" level=debug msg="libcontainerd: starting container b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152"
time="2018-08-15T16:14:35.058232400-07:00" level=debug msg="HCSShim::Container::Start id=b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152"
time="2018-08-15T16:18:25.393050900-07:00" level=debug msg="Result: {\"Error\":-2147023436,\"ErrorMessage\":\"This operation returned because the timeout period expired.\
"}"
time="2018-08-15T16:18:25.394050800-07:00" level=error msg="libcontainerd: failed to start container: container b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db7
9e4152 encountered an error during Start: failure in a Windows system call: This operation returned because the timeout period expired. (0x5b4)"
time="2018-08-15T16:18:25.394050800-07:00" level=debug msg="HCSShim::Container::Terminate id=b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152"
time="2018-08-15T16:18:25.394050800-07:00" level=debug msg="libcontainerd: cleaned up after failed Start by calling Terminate"
time="2018-08-15T16:18:25.394050800-07:00" level=error msg="Create container failed with error: container b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152
encountered an error during Start: failure in a Windows system call: This operation returned because the timeout period expired. (0x5b4)"
time="2018-08-15T16:18:25.424053800-07:00" level=debug msg="attach: stdout: end"
time="2018-08-15T16:18:25.425055000-07:00" level=debug msg="attach: stderr: end"
time="2018-08-15T16:18:25.427054100-07:00" level=debug msg="Revoking external connectivity on endpoint boring_babbage (b20f403df0ed25ede9152f77eb0f8e049677f1279b68862a25b
b9e2ab94babfb)"
time="2018-08-15T16:18:25.459087300-07:00" level=debug msg="[DELETE]=>[/endpoints/31e66619-5b57-47f2-9256-bbba54510e3b] Request : "
time="2018-08-15T16:18:25.548068700-07:00" level=debug msg="Releasing addresses for endpoint boring_babbage's interface on network nat"
time="2018-08-15T16:18:25.548068700-07:00" level=debug msg="ReleaseAddress(172.25.224.0/20, 172.25.229.142)"
time="2018-08-15T16:18:25.561064000-07:00" level=debug msg="WindowsGraphDriver Put() id b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152"
time="2018-08-15T16:18:25.561064000-07:00" level=debug msg="hcsshim::UnprepareLayer flavour 1 layerId b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152"
time="2018-08-15T16:18:25.566074800-07:00" level=debug msg="hcsshim::UnprepareLayer succeeded flavour 1 layerId=b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db7
9e4152"
time="2018-08-15T16:18:25.566074800-07:00" level=debug msg="hcsshim::DeactivateLayer Flavour 1 ID b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e4152"
time="2018-08-15T16:18:25.668075600-07:00" level=debug msg="hcsshim::DeactivateLayer succeeded flavour=1 id=b928213e42b2103cd33b676ed08a15529be10fffcfd7e709af86df5db79e41
52"
I can see when it is trying to create the container and then it fails.
But why?
added more information
I finally found out how to check the server status for running container and I am getting this error message:
So it means the server doesn't have a network gateway?
How can I fix this problem?
Still keep looking
More information
I did delete all NAT and create a new one, so the online check passed now.
However, I still encounter other errors and can't run the image.
Something in the virtual network is wrong, I just can't find the right information to fix it.... :(

Why does "docker login" fail in Docker Quickstart Terminal but work from within the default machine?

I've installed Docker Toolbox in Windows 8.1 and have been following the installation tutorial. When getting to the step where you create and push your own image, I got this error when I attempted to run docker login ... .
### VIA Docker Quickstart Terminal
### docker login --username=myuser --password="mypass" --email=myemail#gmail.com
time="2015-11-17T03:20:58.160803558Z" level=debug msg="Calling POST /v1.21/auth"
time="2015-11-17T03:20:58.160838971Z" level=info msg="POST /v1.21/auth"
time="2015-11-17T03:20:58.169033324Z" level=debug msg="hostDir: /etc/docker/certs.d/https:/registry-win-tp3.docker.io/v1"
time="2015-11-17T03:20:58.169071565Z" level=debug msg="pinging registry endpoint https://registry-win-tp3.docker.io/v1/"
time="2015-11-17T03:20:58.169084660Z" level=debug msg="attempting v1 ping for registry endpoint https://registry-win-tp3.docker.io/v1/"
time="2015-11-17T03:20:58.898542338Z" level=debug msg="Error unmarshalling the _ping PingResult: invalid character '<' looking for beginning of value"
time="2015-11-17T03:20:58.898803841Z" level=debug msg="PingResult.Version: \"\""
time="2015-11-17T03:20:58.898818084Z" level=debug msg="Registry standalone header: ''"
time="2015-11-17T03:20:58.898836197Z" level=debug msg="PingResult.Standalone: true"
time="2015-11-17T03:20:58.898853685Z" level=debug msg="attempting v1 login to registry endpoint https://registry-win-tp3.docker.io/v1/"
time="2015-11-17T03:20:59.478756938Z" level=error msg="Handler for POST /v1.21/auth returned error: Unexpected status code [403] : <html><body><h1>403 Forbidden</h1>\nRequest forbidden by administrative rules.\n</body></html>\n\n"
time="2015-11-17T03:20:59.478815334Z" level=error msg="HTTP Error" err="Unexpected status code [403] : <html><body><h1>403 Forbidden</h1>\nRequest forbidden by administrative rules.\n</body></html>\n\n" statusCode=500
Trying to solve the issue, I tried running docker login ... from within the Docker default VM. And there it works!
### VIA default virtual machine (192.168.99.100)
### docker login --username=myuser --password="mypass" --email=myemail#gmail.com https://index.docker.io/v1/
time="2015-11-17T03:20:46.053333255Z" level=debug msg="Calling POST /v1.21/auth"
time="2015-11-17T03:20:46.053404176Z" level=info msg="POST /v1.21/auth"
time="2015-11-17T03:20:46.082796012Z" level=debug msg="hostDir: /etc/docker/certs.d/https:/index.docker.io/v1"
time="2015-11-17T03:20:46.082930763Z" level=debug msg="pinging registry endpoint https://index.docker.io/v1/"
time="2015-11-17T03:20:46.082946790Z" level=debug msg="attempting v1 ping for registry endpoint https://index.docker.io/v1/"
time="2015-11-17T03:20:46.082959103Z" level=debug msg="attempting v1 login to registry endpoint https://index.docker.io/v1/"
I notice that they're using two different URLs and that the first one encounters a parsing error. The credentials are obviously correct since they work from within the VM, unless the two domains don't share users. Are the URLs or the response being mangled by MINGW64?
Update February 2016
PR 19891 "Enable cross-platforms login to Registry" is supposed to fixed the issue
Use a daemon-defined Registry URL for docker login.
This allows a Windows client interacting with a Linux daemon to properly use the default Registry endpoint instead of the Windows specific one.
It is in commit 19eaa71 (maybe for docker 1.10?)
This is reported both in docker/docker issue 15612 and docker/docker issue 18019
After some analysis of the source code I’ve detected that we have different registry URLs for Windows and UNIX.
Windows: https://registry-win-tp3.docker.io/v1/
Unix: https://index.docker.io/v1/
The Windows url comes from a recent PR 15417 with the comment:
// Currently it is a TEMPORARY link that allows Microsoft to continue
// development of Docker Engine for Windows.
So it is possible this url won't work (unless you are on a very recent Windows Server 2016)
There seems to be a workaround in docker/hub-feedback issues 473, which involves:
specifying the default index registry of docker io,
docker login --username=myuser --password=mypassword --email=myemail https://index.docker.io/v1/
WARNING: login credentials saved in C:\Users\myuser\.docker\config.json
Login Succeeded
modifying the config.json file created by the previous step, in order to add the same credentials for index.docker.io for the registry-win:
config.json:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "myhash",
"email": "myemail"
},
"https://registry-win-tp3.docker.io/v1/": {
"auth": "myhash",
"email": "mydomain"
}
}
}
After that, a docker push index.docker.io/myuser/myrepo:latest does work.

Resources