Unable to push to private docker registry via TLS - docker

I have QNAP NAS behind my router with public IP 1.2.3.4. I have certificate for xxxx.yyyy.cz. The certificate is valid, I am able to reach my NAS over HTTPS. I installed docker registry:2.7 on my NAS. This is container environment configuratin:
REGISTRY_HTTP_ADDR 0.0.0.0:5443
REGISTRY_HTTP_TLS_CERTIFICATE /certs/client.cert
REGISTRY_HTTP_TLS_KEY /certs/client.key
I set up port forwarding 5443 to 5443 TCP. In certs directory are 3 files:
/certs # ls -al
total 24
drwxrwxrwx 2 root root 4096 Oct 20 17:02 .
drwxr-xr-x 1 root root 4096 Oct 20 17:01 ..
-rwxrwxrwx 1 root root 1688 Oct 20 16:42 ca.crt
-rwxrwxrwx 1 root root 2060 Oct 20 16:42 client.cert
-rwxrwxrwx 1 root root 1704 Oct 20 16:42 client.key
I am able to get response from registry with curl or via browser:
$ curl --cacert Downloads/certs/ca.crt https://xxxx.yyyy.cz:5443/v2/_catalog ; echo $?
{"repositories":[]}
0
So I am sure certificate are right and registry is running correctly. When I see the container logs, I am still receiving this messages:
2019/10/20 17:51:10 http: TLS handshake error from 1.2.3.4:58164: tls: first record does not look like a TLS handshake
2019/10/20 17:51:30 http: TLS handshake error from 1.2.3.4:58334: tls: first record does not look like a TLS handshake
2019/10/20 17:51:50 http: TLS handshake error from 1.2.3.4:58498: tls: first record does not look like a TLS handshake
2019/10/20 17:52:11 http: TLS handshake error from 1.2.3.4:58654: tls: first record does not look like a TLS handshake
2019/10/20 17:52:31 http: TLS handshake error from 1.2.3.4:58810: tls: first record does not look like a TLS handshake
2019/10/20 17:52:51 http: TLS handshake error from 1.2.3.4:58982: tls: first record does not look like a TLS handshake
2019/10/20 17:53:12 http: TLS handshake error from 1.2.3.4:59136: tls: first record does not look like a TLS handshake
When I try to push something to my registry, I am receiving error:
$ docker push xxxx.yyyy.cz:5443/myimage:latest
The push refers to repository [xxxx.yyyy.cz:5443/myimage]
Get https://xxxx.yyyy.cz:5443/v2/: x509: certificate signed by unknown authority
and in docker logs I can see error message:
2019/10/20 18:43:28 http: TLS handshake error from 1.2.3.4:41632: remote error: tls: bad certificate
I used this and this instructions, but it did not helped. After I logged to the container, I checked my cert files sha256, they are okay.
How can I use TLS on my docker registry and why it does not accept
my certs?
Why it does not work via docker command?

I had problem with client.cert. It should contains also ca.crt as its mentioned here in section USE AN INTERMEDIATE CERTIFICATE:
A certificate issuer may supply you with an intermediate certificate. In this case, you must concatenate your certificate with the intermediate certificate to form a certificate bundle. You can do this using the cat command:
cat domain.crt intermediate-certificates.pem > certs/domain.crt
You can use the certificate bundle just as you use the domain.crt file in the previous example.

Related

Connecting via TLS to OPENLDAP: Certificate not found

I have an OpenLDAP Docker instance from Osixia and am trying to query it securely from the client using TLS. The query works without encryption using $ ldapwhoami -H ldap://localhost -x and does not work when using the -ZZ flag to start TLS operation $ ldapwhoami -H ldap://localhost -x -ZZ - it returns ldap_start_tls: Can't contact LDAP server (-1). How can i make this work? Below are all the steps i took:
Run LDAP server in docker:
$ docker run -p 389:389 -p 636:636 --name ldap-service --hostname ldap-service \
--env LDAP_ADMIN_PASSWORD="password" --env LDAP_BASE_DN="dc=example,dc=org" --detach osixia/openldap:1.4.0
Test Connectivity - shows success, it returns anonymous
$ ldapwhoami -H ldap://localhost -x
anonymous
Preparations for TLS connectivity - Configure client to trust SERVER Certificate Authority (CA)
SERVER DOCKER CONTAINER: TLS certs are autoconfigured upon runtime in the osixia/openldap image. Copy contents of CA in /container/service/slapd/assets/certs/ca.crt
CLIENT: Paste the copied SERVER ca.crt into CLIENT folder /usr/local/share/ca-certificates/ca.crt , then run sudo update-ca-certificates to add it. Confirm success of adding by checking that the CA is inside /etc/ssl/certs/ca-certificates.crt
CLIENT: In file/etc/ldap/ldap.conf I added the line TLS_CACERT /etc/ssl/certs/ca-certificates.crt
Test TLS connectivity from CLIENT via -ZZ flag to start TLS operation:
$ ldapwhoami -H ldap://localhost -x -ZZ
ldap_start_tls: Can't contact LDAP server (-1)
additional info: The TLS connection was non-properly terminated.
Further logs from inside LDAP docker:
5ff42195 conn=1079 fd=12 ACCEPT from IP=172.17.0.1:39420 (IP=0.0.0.0:636)
TLS: can't accept: No certificate was found..
5ff42195 conn=1079 fd=12 closed (TLS negotiation failure)
Test TLS connectivity from CLIENT via LDAP Secure URI scheme ldaps://
$ ldapwhoami -H ldaps://localhost -x
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
Things i tried out:
I read through https://www.openldap.org/doc/admin24/tls.html and subsequently installed the Server CA on the client.
I read through this post: ldapsearch over ssl/tls doesn't work, I changed the settings in /etc/ldap/ldap.conf to include the below items, but to no avail.
TLS_CACERT /etc/ssl/certs/ca-certificates.crt
TLS_REQCERT ALLOW
PORT 636
HOST localhost // i also tried 'ldap-service'
I found the solution:
Add --env LDAP_TLS_VERIFY_CLIENT=try to the docker run command. Source
For Googlers,
Presto does not supply client certificates (client certificate verification, two-way verification) when connecting to LDAP service, so you will need --env LDAP_TLS_VERIFY_CLIENT=tryor never if you use osixia/openldap, or, edit ldap.conf and set TLS_REQCERT never and restart the LDAP service.

Tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage

Error Message:
dockerd: http: TLS handshake error from 10.129.37.6:55989: tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage
Can anyobe please help me why i am receiving the above error message while trying to connect to the docker daemon from the same server as client using:
curl https://hostname:2376/images/json --cert /data/scripts/mms/server.crt --key /data/scripts/mms/server.key -v
Does the server you are connecting expect client certificate?
If Yes, then --cert /data/scripts/mms/server.crt should be client certificate and --key /data/scripts/mms/server.key should be client key along with --cacert <server root CA certificate> (this can be suppressed by passing -k option). Corresponding Client certificate chain or Root cert should be present on the server trust store.
If No, then run as below:
curl https://hostname:2376/images/json --cacert /data/scripts/mms/server.crt -v. Assuming server.crt is server CA cert.

Not possible to establish TLS with MockServer: curl: (60) Peer's Certificate issuer is not recognized

I am trying to set up a docker network consisting of two containers:
MockServer running on 443
Client (fedora) issuing requests to MockServer
I've installed The MockServer CA X.509 taken from https://github.com/mock-server/mockserver/blob/master/mockserver-core/src/main/resources/org/mockserver/socket/CertificateAuthorityCertificate.pem
into /etc/pki/ca-trust/source/anchors/key.pem followed by update-ca-trust command.
Still, when I am trying to reach MockServer with curl, I am receiving this:
bash-4.2# curl https://www.hostname.net/simpleFirst --verbose
* About to connect() to www.hostname.net port 443 (#0)
* Trying 172.20.128.2...
* Connected to www.hostname.net (172.20.128.2) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* Server certificate:
* subject: C=UK,ST=England,L=London,O=MockServer,CN=localhost
* start date: Jul 24 14:52:38 2020 GMT
* expire date: Jul 29 14:52:38 2021 GMT
* common name: localhost
* issuer: C=UK,ST=England,L=London,O=MockServer,CN=www.mockserver.com
* NSS error -8179 (SEC_ERROR_UNKNOWN_ISSUER)
* Peer's Certificate issuer is not recognized.
* Closing connection 0
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Any advice or help is very much appreciated. Thanks in advance!

'npm ERR! self signed certificate' when running npm install as part of docker build

I have a Dockerfile inside which I run npm install
...
RUN npm install
....
This errors out with the following:
npm ERR! Linux 4.4.0-92-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v7.10.1
npm ERR! npm v4.2.0
npm ERR! code DEPTH_ZERO_SELF_SIGNED_CERT
npm ERR! self signed certificate
I don't hit this issue when running npm install directly on the same machine. (i.e when I am running it not part of docker build). So I don't believe the issue is related to any http proxies.
#Tarun,
Here is the output of curl -v https://docker.com
----* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
server certificate verification OK
server certificate status verification SKIPPED
common name: *.docker.com (matched)
server certificate expiration date OK
server certificate activation date OK
certificate public key: RSA
certificate version: #3
subject: CN=*.docker.com
start date: Fri, 11 Aug 2017 00:00:00 GMT
expire date: Tue, 11 Sep 2018 12:00:00 GMT
issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
compression: NULL
ALPN, server did not agree to a protocol
GET / HTTP/1.1
Host: docker.com
User-Agent: curl/7.47.0
And here is the curl command from inside a docker container on the box.
root#2145cd2e9997:/app/c3po# curl -v https://docker.com
Rebuilt URL to: https://docker.com/
Hostname was NOT found in DNS cache
Trying 52.55.168.191...
Connected to docker.com (52.55.168.191) port 443 (#0)
successfully set certificate verify locations:
CAfile: none
CApath: /etc/ssl/certs
SSLv3, TLS handshake, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS alert, Server hello (2):
SSL certificate problem: self signed certificate
Closing connection 0
SSLv3, TLS alert, Client hello (1):
curl: (60) SSL certificate problem: self signed certificate
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
If you are running a few npm commands from Dockerfile, then you can add this before you run the npm commands:
RUN npm config set strict-ssl false
RUN npm install
Note: it is recommended to copy your company's root certificates (if you are behind a company firewall) to the container rather than disabling SSL altogether.
I figured this out. I was running jenkins on the docker host, and I was using iptables to forward port 443 to 8080. This was conflicting with docker's own iptable rules which was causing all my issues with self signed certs.
This stackoverflow post helped me identify the issue: SSL certificate verification fails inside docker container on specific server

Logging in to private docker registry v2 behind haproxy

I am trying to set up a new Docker Registry (v2) with HAProxy. For the Docker Registry I am using the image from the docker hub and running it with docker run -d -p 5000:5000 -v /path/to/registry:/tmp/registry registry:2.0.1. And this is a subset of my HAProxy configuration:
global
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048
userlist auth_list
group docker_registry users root
user root password ***PASSWORD***
backend docker-registry
server 127.0.0.1:5000_localhost 127.0.0.1:5000 cookie 127.0.0.1:5000_localhost
frontend shared-frontend
mode http
bind *:80
bind *:443 ssl crt *** CERT FILES ***
option accept-invalid-http-request
acl domain_d.mydomain.com hdr(host) -i d.mydomain.com
acl auth_docker_registry_root http_auth(auth_list) root
redirect scheme https if !{ ssl_fc } domain_d.mydomain.com
http-request auth realm Registry if !auth_docker_registry_root { ssl_fc } domain_d.mydomain.com
use_backend docker-registry if domain_d.mydomain.com
The important things to note are that I am using HAProxy to do SSL termination and HTTP auth rather than the registry.
My issue occurs when I try to login to the new registry. If I run docker login https://d.mydomain.com/v2/ then enter the user root and password I get the following error messages:
Docker Client:
FATA[0009] Error response from daemon: invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt
Docker Daemon:
ERRO[0057] Handler for POST /auth returned error: invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt
ERRO[0057] HTTP Error: statusCode=500 invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt
So I try adding --insecure-registry d.mydomain.com to:
/etc/default/docker with DOCKER_OPTS= -H unix:///var/run/docker.sock --insecure-registry d.mydomain.com
the arguments of starting docker manually with docker -d --insecure-registry d.mydomain.com
neither of these, or any other I have found online, work. Each time, after restarting docker and attempting to log in again gives me the same error message.
A few other things I have tried:
In a browser going to d.mydomain.com results in a 404
In a browser going to d.mydomain.com/v2/ results in: {}
Replacing https://d.mydomain.com/v2/ in the login command with all of these with no success
http://d.mydomain.com/v2/
d.mydomain.com/v2/
http://d.mydomain.com/
d.mydomain.com/
This setup with HAProxy doing the SSL termination and HTTP auth has worked in the past using the first version of the registry and older versions of docker. So has anything in Docker registry v2 changed? Does this still work? If it hasn't changed, why won't the --insecure-registry flag do anything anymore?
Also, I have been working on getting this working for a while so I may have forgotten all the things I have tried. If there is something that may work, let me know and I will give it a try.
Thanks,
JamesStewy
Edit
This edit has been moved to the answer below
I have got it working. So here is my new config:
haproxy.cfg
global
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048
userlist auth_list
group docker_registry users root
user root password ***PASSWORD***
backend docker-registry
server 127.0.0.1:5000_localhost 127.0.0.1:5000 cookie 127.0.0.1:5000_localhost
backend docker-registry-auth
errorfile 503 /path/to/registry_auth.http
frontend shared-frontend
mode http
bind *:80
bind *:443 ssl crt *** CERT FILES ***
option accept-invalid-http-request
acl domain_d.mydomain.com hdr(host) -i d.mydomain.com
redirect scheme https if !{ ssl_fc } domain_d.mydomain.com
acl auth_docker_registry_root http_auth(auth_list) root
use_backend docker-registry-auth if !auth_docker_registry_root { ssl_fc } domain_d.mydomain.com
rsprep ^Location:\ http://(.*) Location:\ https://\1
use_backend docker-registry if domain_d.mydomain.com
registry_auth.http
HTTP/1.0 401 Unauthorized
Cache-Control: no-cache
Connection: close
Content-Type: text/html
Docker-Distribution-Api-Version: registry/2.0
WWW-Authenticate: Basic realm="Registry"
<html><body><h1>401 Unauthorized</h1>
You need a valid user and password to access this content.
</body></html>
The differences being the http-request auth line has been replaced with use_backend docker-registry-auth. The backend docker-registry-auth has no servers to it will always give a 503 error. But the 503 error file has been changed to registry_auth.http. In registry_auth.http the error code is overridden to 401, the header WWW-Authenticate is set to Basic realm="Registry", the basic HAProxy 401 error page is supplied and, most importantly, the header Docker-Distribution-Api-Version is set to registry/2.0.
As a result this hacky work-around setup works exactly the same as the old http-request auth line except the custom header Docker-Distribution-Api-Version is now set. This allows this set up to pass the test which starts on line 236 of https://github.com/docker/docker/blob/v1.7.0/registry/endpoint.go.
So now when I run docker login d.mydomain.com, login is successful and my credentials are added to .docker/config.json.
The second issue was that I couldn't push to the new repository even through it logged in. This was fixed by adding the rsprep line in the frontend. What this line does is modify the Location header (if it exists) to turn all http:// to https://.
I also found this bit of documentation for future reference.
As a small clarification to the previous answer: I had to change this line:
WWW-Authenticate: Basic realm="Registry"
To this:
WWW-Authenticate: Basic realm="Registry realm"
and then everything worked...
BTW, hashing the pass can be done using mkpasswd (part of whois deb package)

Resources