I'm using Gitea (on Kubernetes, behind an Ingress) as a Docker image registry. On my network I have gitea.avril aliased to the IP where it's running. I recently found that my Kubernetes cluster was failing to pull images:
Failed to pull image "gitea.avril/scubbo/<image_name>:<tag>": rpc error: code = Unknown desc = failed to pull and unpack image "gitea.avril/scubbo/<image_name>:<tag>": failed to resolve reference "gitea.avril/scubbo/<image_name>:<tag>": failed to authorize: failed to fetch anonymous token: unexpected status: 530
While trying to debug this, I found that I am unable to login to the registry, even though curling with the same credentials succeeds:
$ curl -k -u "scubbo:$(cat /tmp/gitea-password)" https://gitea.avril/v2/_catalog
{"repositories":[...populated list...]}
# Tell docker login to treat `gitea.avril` as insecure, since certificate is provided by Kubernetes
$ cat /etc/docker/daemon.json
{
"insecure-registries": ["gitea.avril"]
}
$ docker login -u scubbo -p $(cat /tmp/gitea-password) https://gitea.avril
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://gitea.avril/v2/": received unexpected HTTP status: 530
The first request shows up as a 200 OK in the Gitea logs, the second as a 401 Unauthorized.
I get a similar error when I kubectl exec onto the Gitea container itself, install Docker, and try to docker login localhost:3000 - after an error indicating that server gave HTTP response to HTTPS client, it falls back to the http protocol and similarly reports a 530.
I've tried restart Gitea with GITEA__log__LEVEL=Debug, but that didn't result in any extra logging. I've also tried creating a fresh user (in case I have some weirdness cached somewhere) and using that - same behaviour.
EDIT: after increasing log level to Trace, I noticed that successful attempts to curl result in the following lines:
...rvices/auth/basic.go:67:Verify() [T] [638d16c4] Basic Authorization: Attempting login for: scubbo
...rvices/auth/basic.go:112:Verify() [T] [638d16c4] Basic Authorization: Attempting SignIn for scubbo
...rvices/auth/basic.go:125:Verify() [T] [638d16c4] Basic Authorization: Logged in user 1:scubbo
whereas attempts to docker login result in:
...es/container/auth.go:27:Verify() [T] [638d16d4] ParseAuthorizationToken: no token
This is the case even when doing docker login localhost:3000 from the Gitea container itself (that is - this is not due to some authentication getting dropped by the Kubernetes Ingress).
I'm not sure what could be causing this - I'll start up a fresh Gitea registry to compare.
EDIT: in this Github issue, the Gitea team pointed out that standard docker authentication includes creating a Bearer token which references the ROOT_URL, explaining this issue.
Text below preserved for posterity:
...Huh. I have a fix, and I think it indicates some incorrect (or, at least, unexpected) behaviour; but in fairness it only comes about because I'm doing some pretty unexpected things as well...
TL;DR attempting to docker login to Gitea from an alternative domain name can result in an error if the primary domain name is unavailable; apparently because, while doing so, Gitea itself makes a call to ROOT_URL rather than localhost
Background
Gitea has a configuration variable called ROOT_URL. This is, among other things, used to generate the copiable "HTTPS" links from repo pages. This is presumed to be the "main" URL on which users will access Gitea.
I use Cloudflared Tunnels to make some of my Kubernetes services (including Gitea) available externally (on <foo>.scubbo.org addresses) without opening ports to the outside world. Since Cloudflared tunnels do not automatically update DNS records when a new service is added, I have written a small tool[0] which can be run as an initContainer "before" restarting the Cloudflared tunnel, to refresh DNS[1].
Cold-start problem
However, now there is a cold-start problem:
(Unless I temporarily disable this initContainer) I can't start Cloudflared tunnels if Gitea is unavailable (because it's the source for the initContainer's image)
Gitea('s public address) will be unavailable until Cloudflared tunnels start up.
To get around this cold-start problem, in the Cloudflared initContainers definition, I reference the image by a Kubernetes Ingress name (which is DNS-aliased by my router) gitea.avril rather than by the public (Cloudflared tunnel) name gitea.scubbo.org. The cold-start startup sequence then becomes:
Cloudflared tries to start up, fails to find a registry at gitea.avril, continues to attempt
Gitea (Pod and Ingress) start up
Cloudflared detects that gitea.avril is now responding, pulls the Cloudflared initContainer image, and successfully deploys
gitea.scubbo.org is now available (via Cloudflared)
So far, so good. Except that testing now indicates[2] that, when trying to docker login (or docker pull, or presumably, many other docker commands) to a Gitea instance will result in a call to the ROOT_URL domain - which, if Cloudflared isn't up yet, will result in an error[3].
So what?
My particular usage of this is clearly an edge case, and I could easily get around this in a number of ways (including moving my "Cloudflared tunnel startup" to a separately-initialized, only-privately-available registry). However, what this reduces to is that "docker API calls to a Gitea instance will fail if the ROOT_URL for the instance is unavailable", which seems like unexpected behaviour to me - if the API call can get through to the Gitea service in the first place, it should be able to succeed in calling itself?
However, I totally recognize that the complexity of fixing this (going through and replacing $ROOT_URL with localhost:$PORT throughout Gitea) might not be worth the value. I'll open an issue on the Gitea team, but I'd be perfectly content with a WILLNOTFIX.
Footnotes
[0]: Note - depending on when you follow that link, you might see a red warning banner indicating "_Your ROOT_URL in app.ini is https://gitea.avril/ but you are visiting https://gitea.scubbo.org/scubbo/cloudflaredtunneldns_". That's because of this very issue!
[1]: Note from the linked issue that the Cloudflared team indicate that this is unexpected usage - "We don't expect the origins to be dynamically added or removed services behind cloudflared".
[2]: I think this is new behaviour, as I'm reasonably certain that I've done a successful "cold start" before. However, I wouldn't swear to it.
[3]: After I've , the error is instead error parsing HTTP 404 response body: unexpected end of JSON input: "" rather than the 530-related errors I got before. This is probably a quirk of Cloudflared's caching or DNS behaviour. I'm working on a minimal reproducing example that circumvents Cloudflared.
I was using docker in linux machine where I was pulling images from my local docker repo over http authentication. Now I need to use the same thing on windows setup. Issue is when I am trying to pull image using command
docker pull <IP>:port/abc/xyz
it gives me error Private registry push fail: server gave HTTP response to HTTPS client
I have modified the daemon.json file to
{"registry-mirrors":[],"insecure-registries":["<IP>:port"], "debug":true, "experimental": false}
even after this its not getting started. And showing me the same error.
I have faced the same issue. What you have to do is just give the insecure-registries, and remove all other configurations. Just copy paste the below json inside daemon.json file ( available in "C:\Users{user-name}.docker\daemon.json" or "/etc/docker/daemon.json")
{
"insecure-registries" :["<IP>:port"]
}
From the documentation found here: https://books.sonatype.com/nexus-book/3.0/reference/docker.html
I can conclude that I cannot create a private docker registry unless I expose it through https.
Docker relies on secure connections using SSL to connect to the repositories. You are therefore required to expose the repository manager to your client tools via HTTPS. This can be configured via an external proxy server or directly with the repository manager. Further details can be found in Section 5.9.4, “Inbound SSL - Configuring to Serve Content via HTTPS”.
I have done all these steps (using reverse-proxy on https://localhost:5001 forwarding to nexus proxy registry with http connector). However now that I want to start pulling from my local registry, I cannot find a way to access it through https.
The following command which is describe here: https://docs.docker.com/engine/reference/commandline/pull/#pull-from-a-different-registry
docker pull localhost:5001/hello-world
returns:
Error response from daemon: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "\n\n400 Bad Request\n\nBad Request\nYour browser sent a request that this server could not understand.\nReason: You're speaking plain HTTP to an SSL-enabled server port.\n Instead use the HTTPS scheme to access this URL, please.\n\n\n"
and when I try this:
docker pull https://localhost:5001/hello-world
I get:
invalid reference format
The solution to this is:
either having a valid SSL certificate for the proxy you re accessing the repository through
or
creating a self-signed certificate and manually inserting it in the Windows Trusted root authorities certificates
of the computer you want to access the registry from.
This should resolve any issues and relevant messages. Try accessing https://proxyUrl:5000/v2 and you should now be getting a different message than before as well as be able to pull and push to the registry.
I have the registry v2 container and docker_auth up and running. The registry uses self sign certificate which was created with my CA. I can pull and push images without any problem so the configuration is working properly.
I would like to have a UI for this registry to browse images. I have tried many of them:
https://github.com/kwk/docker-registry-frontend not support token based auth
https://shipyard-project.com only supports v1 registries
https://github.com/SUSE/Portus looks quite heavy weight for me
https://github.com/mkuchin/docker-registry-web i configure it with my keyfile and set the issuer. But i couldn't add my repo, it always returns 401. In the docker registry i found "token intended for another audience: \"mydockerrepo:5000\"". I tried with different names but none of them worked. I check the crt and key files and i use the correct crt-key pair.
Is there any other UI i should try?
Can somebody help me with the "audience" error message?
I have a CI-server which is building en pushing images to a secure registry. Every image gets a tag which is equal to the ID of the build.
I can see the images in my secure registry (self-signed). I can also use them so they're working fine.
But I'm unable to see them by URL in my browser:
In the browser of the server I'm performing:
https://localhost:5000/var/lib/registry/docker/registry/v2/repositories/conti/myapp/_manifests/tags/19
But I'm getting a 404-error instead of the image ID (after I've accepted the certificate).
What am I doing wrong? I get a 404 for every https://localhost:5000/...
I see nothing when just going to https://localhost:5000
[x#localhost ~]$ curl -k https://localhost:5000/var/lib/registry/docker/registry/v2/repositories/conti/myapp/_manifests/tags/19
404 page not found
Docker registry V2 doesn't support that particular endpoint, so it's not possible at the moment.