I'm running Sonatype Nexus as Proxy registry for Quay.io and docker.io.
I'm pulling the images with a custom domain proxy-hub.example.com and proxy-quay.example.com.
When Nexus is down obviously I can't download any images, so I thought I can use HAproxy to set the original URL.
backend registry_quay
balance roundrobin
server-template Nexus_nexus 1 Nexus_nexus:8085 check resolvers docker resolve-prefer ipv4 init-addr libc,none
server quay quay.io:443 check backup ssl verify none
This backend works fine as when nexus is down the backup takes over.
With the same settings docker.io fails with error 503 when I turn off Nexus.
backend registry_hub
balance roundrobin
server-template Nexus_nexus 1 Nexus_nexus:8083 check resolvers docker resolve-prefer ipv4 init-addr libc,none
server hub registry-1.docker.io:443 check backup ssl verify none
I'm quite sure that something needs to be rewritten in the requests but I don't know what.
Related
We have an internal Harbor registry setup on a CentOS server for storing the Docker images. It is accessed through a SSL URL.
The SSL configuration is done on the load balancer. The server where Harbor registry is setup just has the basic application configuration with the http port defined as 5044.
When we try to push any images from any remote machine using the ip:port combination, it works fine.
However, when using the SSL URL, it fails with the below error:
dial tcp <load balancer ip>:80: connect: connection refused
We haven't configured port 80 to be used explicitly. We checked the Harbor and nginx configuration files but doesn't seem like we have port 80 mentioned anywhere.
The load balancer is configured to use and port 5044.
Also, the pull works as expected with ip:port and SSL URL both.
Can someone help on what might we be missing in the configuration?
Thank you!
I have set up a private registry and secured it according to the recipe with an nginx reverse proxy. nginx listens on port 5000 using SSL.
docker pull myregistry:5000/foo:latest from a remote machine to that registry works fine.
However, that same command on myregistry itself results in docker trying to access the registry (through nginx) via HTTP, not HTTPS.
Since nginx listens using SSL, it returns an error ("The plain HTTP request was sent to HTTPS port").
According to the Docker documentation, local registries are automatically considered as insecure.
In my case, I want the local registry also to be considered as secure, so that docker pull myregistry:5000/foo:latest works on the same machine. How to achieve that?
There is only an option to mark remote registries as insecure, but not to mark a specific registry as secure.
Obviously, I cannot use a different port to listen for plain HTTP, since that would change the image name. I also did not find a way to make nginx accept HTTP traffic on the same port based on IP address.
I'm using haproxy as a bastion server / cluster gateway, as only some of the nodes in my network have direct access to the external network. My internal nodes are part of a kubernetes cluster, and need to be able to pull images from a private registry external to my cluster which requires certificate identities.
k8s cluster internal node -> haproxy on edge node -> docker registry
I'm trying to configure my back_end in my haproxy.cfg to route to the docker registry, and update the request with the certificate identity of the edge node. (Internal node does not have certs acceptable to the docker registry, and I've not been allowed to host the external node's certs on the internal node.) What I have right now looks like the below...
frontend ft_ssl
bind <boxIP>:443
mode http
default_backend bk_global_import_registry_certs
backend bk_global_import_registry_certs
mode http
balance roundrobin
server registry_alias-0 <registryIP>:443 ssl ca-file fullyqualified/ca.crt crt fullyqualified/file.pem check
server registry_alias-1 <registryIP2>:443 ssl ca-file fullyqualified/ca.crt crt fullyqualified/file.pem check
I currently have the HTTPS_PROXY setting in /etc/systemd/system/docker.service.d/http-proxy.conf and am getting a 400 Bad Request. Scrubbed log message below, with only changes as removal of IPs or typos.
InternalIP:randomPort [09/Jul/2019:13:28:08.659] ft_ssl bk_global_import_registry_certs 0/0/10/5/15 400 350 - - ---- 1/1/0/0/0 0/0 {} "CONNECT externalFQDN:443 HTTP/1.1"
For those looking at this via the kubernetes or docker tags, I also considered setting up a pull-through cache, but realized this only works with Docker's public registry - see open Docker GitHub issue 1431 for other folks trying to find ways to get past that, as well.
Posting the answer that resolved the situation for us, in case it helps others...
All traffic from the internal node now routes through to the HAProxy.
I'm no longer using the HTTPS_PROXY setting in /etc/systemd/system/docker.service.d/http-proxy.conf. Proxying through was not appropriate, since I could not use my internal node's certificate to authenticate against the docker registry.
From the internal node, I'm now treating the docker registry as an insecure registry by adding its address to /etc/docker/daemon.json in insecure-registries: [].
We can now access our internal private registry using the certificates passed along on the forward on the HAProxy backend. We do verify the certificate on the front-end, but because we have the registry listed in insecure-registries, Docker accepts the cert name mismatch.
One side effect noticed: pulling an image from "default" docker registry without specifying a prefix does not succeed using our solution. To pull from Docker, for example, you'd pull from registry-1.docker.io/imageName:imageVersion, rather than just imageName:imageVersion.
I might be mistaking but as far as I know Haproxy cannot be used as a direct HTTP proxy, it only acts as a reverse HTTP proxy. You should use Squid or something like that to be a direct HTTP proxy.
If you want Haproxy to terminate SSL for you like you do, then you will need to change hostname part of your Docker image to be the hostname of your Haproxy node. You should also make sure your Docker daemons trust the Haproxy certificate or add Haproxy to the list of INSECURE_REGISTRIES on all Kube nodes.
Sonatype Nexus documentation indicates that you can push to a docker registry without a HTTP connector as per this guide.
It states the following:
Nexus would be listening at a non-https connector such as the default
8081. Docker repositories are added into Nexus as normal but would NOT be configured with any connector port values.
And the example shows:
The reverse proxy would direct docker push commands received at
https://project.example.com:8086 to
http://locahost:8081/repository/docker-hosted-project
I'm failing to understand how http://locahost:8081/repository/docker-
hosted-project resolves to a registry. I can't test this directly on the Nexus server since the docker client does not support context paths.
How is this supposed to work?
Did the following on Tutum:
Registry
Started a registry:2.1.1 service
Published the port 5000 and configured the registry service with:
VIRTUAL_HOST=https://my-registry.my-host.net
TCP_PORTS=5000/ssl
SSL_CERT="..."
Now, pointed the my-registry.my-host.net DNS to the registry service endpoint and testing the registry with:
docker login my-registry.my-host.net:5000
Works just fine, including the SSL!
HAProxy
Started a tutum/haproxy:latest service, published the 443 port, added API access and linked to the registry service, everything else is default
Pointed my-registry.my-host.net DNS to the haproxy service endpoint and tested the registry login with:
docker login my-registry.my-host.net
This time, the request fails with:
503 Service Unavailable
No server is available to handle this request.
What am I missing?
Note: everything was done from Tutum's Dashboard web UI.
Additionally, here's the generated haproxy.cfg from the HAProxy service container, for those who have experience with HAProxy, but not necessarily with Tutum:
https://gist.github.com/lazabogdan/3bf52984faa092b1a50b (note: the registry service ID has been masked with XXXXXXXX and the real FQDN has been replaced with my-registry.my-host.net)
Solved it.
I had to do the following:
Update the environment variable for the registry service from TCP_PORTS=5000/ssl to TCP_PORTS=5000
On the haproxy service, expose port 5000 on the container AND publish it on the host to port 443.
Now, I can successfully do:
docker login my-registry.my-host.net