I'm using Gitea (on Kubernetes, behind an Ingress) as a Docker image registry. On my network I have gitea.avril aliased to the IP where it's running. I recently found that my Kubernetes cluster was failing to pull images:
Failed to pull image "gitea.avril/scubbo/<image_name>:<tag>": rpc error: code = Unknown desc = failed to pull and unpack image "gitea.avril/scubbo/<image_name>:<tag>": failed to resolve reference "gitea.avril/scubbo/<image_name>:<tag>": failed to authorize: failed to fetch anonymous token: unexpected status: 530
While trying to debug this, I found that I am unable to login to the registry, even though curling with the same credentials succeeds:
$ curl -k -u "scubbo:$(cat /tmp/gitea-password)" https://gitea.avril/v2/_catalog
{"repositories":[...populated list...]}
# Tell docker login to treat `gitea.avril` as insecure, since certificate is provided by Kubernetes
$ cat /etc/docker/daemon.json
{
"insecure-registries": ["gitea.avril"]
}
$ docker login -u scubbo -p $(cat /tmp/gitea-password) https://gitea.avril
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://gitea.avril/v2/": received unexpected HTTP status: 530
The first request shows up as a 200 OK in the Gitea logs, the second as a 401 Unauthorized.
I get a similar error when I kubectl exec onto the Gitea container itself, install Docker, and try to docker login localhost:3000 - after an error indicating that server gave HTTP response to HTTPS client, it falls back to the http protocol and similarly reports a 530.
I've tried restart Gitea with GITEA__log__LEVEL=Debug, but that didn't result in any extra logging. I've also tried creating a fresh user (in case I have some weirdness cached somewhere) and using that - same behaviour.
EDIT: after increasing log level to Trace, I noticed that successful attempts to curl result in the following lines:
...rvices/auth/basic.go:67:Verify() [T] [638d16c4] Basic Authorization: Attempting login for: scubbo
...rvices/auth/basic.go:112:Verify() [T] [638d16c4] Basic Authorization: Attempting SignIn for scubbo
...rvices/auth/basic.go:125:Verify() [T] [638d16c4] Basic Authorization: Logged in user 1:scubbo
whereas attempts to docker login result in:
...es/container/auth.go:27:Verify() [T] [638d16d4] ParseAuthorizationToken: no token
This is the case even when doing docker login localhost:3000 from the Gitea container itself (that is - this is not due to some authentication getting dropped by the Kubernetes Ingress).
I'm not sure what could be causing this - I'll start up a fresh Gitea registry to compare.
EDIT: in this Github issue, the Gitea team pointed out that standard docker authentication includes creating a Bearer token which references the ROOT_URL, explaining this issue.
Text below preserved for posterity:
...Huh. I have a fix, and I think it indicates some incorrect (or, at least, unexpected) behaviour; but in fairness it only comes about because I'm doing some pretty unexpected things as well...
TL;DR attempting to docker login to Gitea from an alternative domain name can result in an error if the primary domain name is unavailable; apparently because, while doing so, Gitea itself makes a call to ROOT_URL rather than localhost
Background
Gitea has a configuration variable called ROOT_URL. This is, among other things, used to generate the copiable "HTTPS" links from repo pages. This is presumed to be the "main" URL on which users will access Gitea.
I use Cloudflared Tunnels to make some of my Kubernetes services (including Gitea) available externally (on <foo>.scubbo.org addresses) without opening ports to the outside world. Since Cloudflared tunnels do not automatically update DNS records when a new service is added, I have written a small tool[0] which can be run as an initContainer "before" restarting the Cloudflared tunnel, to refresh DNS[1].
Cold-start problem
However, now there is a cold-start problem:
(Unless I temporarily disable this initContainer) I can't start Cloudflared tunnels if Gitea is unavailable (because it's the source for the initContainer's image)
Gitea('s public address) will be unavailable until Cloudflared tunnels start up.
To get around this cold-start problem, in the Cloudflared initContainers definition, I reference the image by a Kubernetes Ingress name (which is DNS-aliased by my router) gitea.avril rather than by the public (Cloudflared tunnel) name gitea.scubbo.org. The cold-start startup sequence then becomes:
Cloudflared tries to start up, fails to find a registry at gitea.avril, continues to attempt
Gitea (Pod and Ingress) start up
Cloudflared detects that gitea.avril is now responding, pulls the Cloudflared initContainer image, and successfully deploys
gitea.scubbo.org is now available (via Cloudflared)
So far, so good. Except that testing now indicates[2] that, when trying to docker login (or docker pull, or presumably, many other docker commands) to a Gitea instance will result in a call to the ROOT_URL domain - which, if Cloudflared isn't up yet, will result in an error[3].
So what?
My particular usage of this is clearly an edge case, and I could easily get around this in a number of ways (including moving my "Cloudflared tunnel startup" to a separately-initialized, only-privately-available registry). However, what this reduces to is that "docker API calls to a Gitea instance will fail if the ROOT_URL for the instance is unavailable", which seems like unexpected behaviour to me - if the API call can get through to the Gitea service in the first place, it should be able to succeed in calling itself?
However, I totally recognize that the complexity of fixing this (going through and replacing $ROOT_URL with localhost:$PORT throughout Gitea) might not be worth the value. I'll open an issue on the Gitea team, but I'd be perfectly content with a WILLNOTFIX.
Footnotes
[0]: Note - depending on when you follow that link, you might see a red warning banner indicating "_Your ROOT_URL in app.ini is https://gitea.avril/ but you are visiting https://gitea.scubbo.org/scubbo/cloudflaredtunneldns_". That's because of this very issue!
[1]: Note from the linked issue that the Cloudflared team indicate that this is unexpected usage - "We don't expect the origins to be dynamically added or removed services behind cloudflared".
[2]: I think this is new behaviour, as I'm reasonably certain that I've done a successful "cold start" before. However, I wouldn't swear to it.
[3]: After I've , the error is instead error parsing HTTP 404 response body: unexpected end of JSON input: "" rather than the 530-related errors I got before. This is probably a quirk of Cloudflared's caching or DNS behaviour. I'm working on a minimal reproducing example that circumvents Cloudflared.
Related
While following Heroku's docs for how to push a docker image to their registry, I keep running into this error:
> docker push registry.heroku.com/<MY-APP>/web cd
Using default tag: latest
The push refers to repository [registry.heroku.com/<MY-APP>/web]
e0d052f1dc62: Preparing
41ec0e96eb83: Preparing
d081ada49467: Waiting
73c3e7ef7bc6: Waiting
unauthorized: authentication required
I continue to get a Login Succeeded whenever I try to use docker login, so I'm not sure what the issue is.
I tried to debug using the Docker Daemon logs but those weren't helpful.
Turns out I was bitten by what I'd consider to be a bug with the Heroku registry that stems from a debate about how to deny the user properly when they're logged in but try to access a resource that either doesn't exist or isn't theirs so that sensitive info, like the existence of a resource, isn't exposed (check this summary if you're interested).
TL;DR - Heroku shuold be sending a 404 but send a 401 instead - Go make the app via the UI and then try again.
I've an on-prem instance of Gitlab-CE 13.0.5 running, I'm using the official docker image of Gitlab.
I've enabled the integrated container registry.
Testing the login and push at the registry using a personal access token works, both on the commandline and within a CI script.
Using the CI job token in a CI script, the docker login passes, the docker push fails.
Using a group access token (with the read and write registry privilege), both login and then of course also push fails. Testing the group access token manually on the commandline the login step also fails.
I've checked the logfile of the registry, I only see the access denied message, no further hint whats might be wrong.
I've considered to tag the image with the correct hiearchy of group and project name.
Has anyone an idea where I should continue to search?
Thanks and cheers
Wolfgang
Finally, I found it!
If there is a port number in the registry name in the login command, exactly the same name including the port number has to be used when tagging and pushing an image.
So, if in the gitlab configuration in the variable gitlab_rails['registry_port'] = "443" the default port number 443 is mentioned, it appears in the variable $CI_REGISTRY and you have to use it in the tag and the push command.
Setting the variable gitlab_rails['registry_port'] = "" to an empty string let the system still use the port 443 - since it is the default port. However, it will be removed from the name.
To be honest, I was a bit surprised.
I set up a docker containing a nodejs server and which receives arguments from a path. I can communicate well with my docker instance when I run a curl command (I have an error message, but this is normal)
I'm trying to communicate with my docker instance from the outside with my Nginx server, but I'm having some problems.
Indeed, when I enter the access url, I get a 404 error. My configuration file looks like this.
when I try I go to the access url, I do have a 404 error.
My concern is on the "location" part of my configuration, but I don't really see how to solve the problem. If you have any leads, I'll take them :)
Here is my whole scenario.
I have a RHEL 7.1 vmware image, with the corporate proxy properly configured, accessing stuff over http or https works properly.
Installed docker-engine, and added the HTTP_PROXY setting to /etc/systemd/system/docker.service.d/http-proxy.conf. I can verify the proxy setting is picked up by executing:
sudo systemctl show docker --property Environment
which will print:
Environment=HTTP_PROXY=http://proxy.mycompany.com:myport/ with real values of course.
Pulling and running docker images works correctly this way.
The goal is to work with the binary distribution of openshift-origin. I downloaded the binaries, and started setting up things as per the walkthrough page on github:
https://github.com/openshift/origin/blob/master/examples/sample-app/README.md
Starting openshift seems to work as I can:
* login via the openshift cli
* create a new project
* even access the web console
But when I try to create an app in the project (also via the cli):
oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git
It fails:
error: can't look up Docker image "centos/ruby-22-centos7": Internal error occurred: Get https://registry-1.docker.io/v2/: dial tcp 52.71.246.213:443: connection refused
I can access (without authentication though) this endpoint via the browser on the VM or via WGET.
Hence I believe DOCKER fails to pick up the proxy settings. After some searching I also fear if there are IPTABLES settings missing. Referring to:
https://docs.docker.com/v1.7/articles/networking/
But I don't know if I should fiddle with the IPTABLES settings, should not Docker figure that out itself?
Check your HTTPS_PROXY environment property.
I'm trying to create private registry for docker and I'm relying on instructions given on the docker site. I have a seperate linux box where this registry is installed, then I'm trying to push my images from local(osx box with docker toolbox). I keep on getting 405 from registry server. I'm quite new to docker. I was hoping the default basic configuration to work without much trouble.
configuration
Latest docker toolbox.
Latest registry installation.
I only changes the TLS configuration to post the request over http.
Error
The push refers to a repository [192.168.1.98:5000/complete] (len: 1)
Sending image list
Error: Status 405 trying to push repository complete: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>405 Method Not Allowed</title>\n</head><body>\n<h1>Method Not Allowed</h1>\n<p>The requested method PUT is not allowed for the URL /v1/repositories/complete/.</p>\n</body></html>\n"
The doc mentions
405 Method Not Allowed
Manifest put is not allowed because the registry is configured as a pull-through cache or for some other reason
(like a read-only mode)
UNSUPPORTED: The operation was unsupported due to a missing implementation or invalid set of parameters.
The same doc uses urls which include /v2, not like the one used in the question (/v1/repositories/complete)
The instructions include:
Getting the headers correct is very important. For all responses to any request under the “/v2/” url space, the Docker-Distribution-API-Version header should be set to the value “registry/2.0”, even for a 4xx response
Make sure you are running a v2 registry image (which is now docker distribution)
The OP Charith actually found in the comments:
mistake in port forwarding on the registry host: The 5000 port was servicing from another server.
I've switched to an available port and everything started working.