Docker: private registry access - docker

I'm trying to push an image to my docker private repository:
docker pull busybox
docker tag busybox living-registry.com:5000/busybox
docker push living-registry.com:5000/busybox
Docker tells me:
The push refers to a repository [living-registry.com:5000/busybox]
Get https://living-registry.com:5000/v1/_ping: read tcp 195.83.122.16:39714->195.83.122.16:5000: read: connection reset by peer
These commands are being performed on a CoreOS.
In another machine, I've started my registry using this command:
docker run -d -p 5000:5000 --restart=always --name registry \
-v /root/docker-registry/auth:/auth \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v /root/docker-registry/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/registry.key \
-v /root/docker-registry/data:/var/lib/registry \
registry:2
Everything seems to be right:
# netstat -tupln | grep 5000
tcp6 0 0 :::5000 :::* LISTEN 3160/docker-proxy
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
27e79f6a504c registry:2 "/bin/registry serve " About an hour ago Restarting (2) 36 minutes ago 0.0.0.0:5000->5000/tcp registry
So, when I'm trying to log in:
[root#jenkins certs]# docker login living-registry.com:5000
Username: xxxx
Password: xxxx
Error response from daemon: Get https://living-registry.com:5000/v1/users/: read tcp 195.83.122.16:39756->195.83.122.16:5000: read: connection reset by peer
Any ideas?
EDIT
I've already added the certificate (ca.crt) in /etc/ssl/certs and in /etc/docker/certs.d/x.x.x.x:5000/.
From this CoreOS instance, I'm trying to perform that:
$ docker login https://x.x.x.x:5000
Username: xxx
Password:
Email: xxx#mail.com
And it tells me:
Error response from daemon: invalid registry endpoint https://x.x.x.x:5000/v0/: unable to ping registry endpoint https://x.x.x.x:5000/v0/
v2 ping attempt failed with error: Get https://x.x.x.x:5000/v2/: EOF
v1 ping attempt failed with error: Get https://x.x.x.x:5000/v1/_ping: EOF. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add --insecure-registry x.x.x.x:5000 to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/x.x.x.x:5000/ca.crt
I've also tried to get the connection directly with openssl:
openssl s_client -connect x.x.x.x:5000
The output is:
CONNECTED(00000003)
140180300502672:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 308 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1467812448
Timeout : 300 (sec)
Verify return code: 0 (ok)
---

For self-signed certificates, the crt must be copied into
/etc/docker/cert.d/hostname:port/ca.crt
cf : https://docs.docker.com/engine/security/certificates/
I create certificates :
openssl req -x509 -nodes -days 3650d -newkey rsa:2048 -keyout /root/docker-registry/certs/registry.key -out /root/docker-registry/certs/registry.crt -days 3650d
cp /root/docker-registry/certs/registry.crt /etc/docker/cert.d/x.x.x.x:5000/ca.crt

Related

Docker Container Refuses to NOT use Proxy for Docker Network

I'm having issues trying to get networking to work correctly in my container inside a corp domain/behind a proxy.
I've correctly configured (I think) Docker to get around the proxy for downloading images, but now my container is having trouble talking to another container inside the same docker-compose network.
So far, the only resolution is to manually append the docker-compose network to the no_proxy variable in the docker config, but this seems wrong and would need to be configured for each docker-compose network and requires a restart of docker.
Here is how I configured the docker proxy settings on host:
cat << "EOF" >docker_proxy_setup.sh
#!/bin/bash
#Proxy
#ActiveProxyVar=127.0.0.1:80
#Domain
corpdom=domain.org
httpproxyvar=http://$ActiveProxyVar/
httpsproxyvar=http://$ActiveProxyVar/
mkdir ~/.docker
cat << EOL >~/.docker/config.json
{
"proxies":
{
"default":
{
"httpProxy": "$httpproxyvar",
"httpsProxy": "$httpsproxyvar",
"noProxy": ".$corpdom,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
}
}
}
EOL
mkdir -p /etc/systemd/system/docker.service.d
cat << EOL >/etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=$httpproxyvar"
Environment="HTTPS_PROXY=$httpsproxyvar"
Environment="NO_PROXY=.$corpdom,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
EOL
systemctl daemon-reload
systemctl restart docker
#systemctl show --property Environment docker
docker run hello-world
EOF
chmod +x docker_proxy_setup.sh
docker_proxy_setup.sh
and basically if I change to this:
#Domain
corpdom=domain.org,icinga_icinga-net
I am able to use curl to test network and it works correctly, but ONLY when using container_name.icinga_icinga-net
Eg:
This fails curl -k -u root:c54854140704eafc https://icinga2-api:5665/v1/objects/hosts
While this succeeds curl -k -u root:c54854140704eafc https://icinga2-api.icinga_icinga-net:5665/v1/objects/hosts
Note that using curl --noproxy seems to have no effect.
Here is some output from container for reference, any ideas what I can do to have containers NOT use proxy for Docker Networks (private IPv4)?
root#icinga2-web:/# ping icinga2-api
PING icinga2-api (172.30.0.5) 56(84) bytes of data.
64 bytes from icinga2-api.icinga_icinga-net (172.30.0.5): icmp_seq=1 ttl=64 time=0.138 ms
64 bytes from icinga2-api.icinga_icinga-net (172.30.0.5): icmp_seq=2 ttl=64 time=0.077 ms
^C
--- icinga2-api ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1025ms
rtt min/avg/max/mdev = 0.077/0.107/0.138/0.030 ms
root#icinga2-web:/# curl --noproxy -k -u root:c54854140704eafc https://172.30.0.5:5665/v1/objects/hosts
curl: (56) Received HTTP code 503 from proxy after CONNECT
root#icinga2-web:/# curl -k -u root:c54854140704eafc https://172.30.0.5:5665/v1/objects/hosts
curl: (56) Received HTTP code 503 from proxy after CONNECT
root#icinga2-web:/# curl -k -u root:c54854140704eafc https://icinga2-api:5665/v1/objects/hosts
curl: (56) Received HTTP code 503 from proxy after CONNECT
root#icinga2-web:/# curl -k -u root:c54854140704eafc https://icinga2-api.icinga_icinga-net:5665/v1/objects/hosts
{"results":[{"attrs":{"__name":"icinga2-api","acknowledgement":0,"acknowledgement_expiry":0,"acknowledgement_last_change":0,"action_url":"","active":true,"address":"127.0.0.1","address6":"::1","check_attempt":1,"check_command":"hostalive","check_interval":60,"check_period":"","check_timeout":null,"command_endpoint":"","display_name":"icinga2-api","downtime_depth":0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":false,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":true,"event_command":"","executions":null,"flapping":false,"flapping_current":0,"flapping_ignore_states":null,"flapping_last_change":0,"flapping_threshold":0,"flapping_threshold_high":30,"flapping_threshold_low":25,"force_next_check":false,"force_next_notification":false,"groups":["linux-servers"],"ha_mode":0,"handled":false,"icon_image":"","icon_image_alt":"","last_check":1663091644.161905,"last_check_result":{"active":true,"check_source":"icinga2-api","command":["/usr/lib/nagios/plugins/check_ping","-H","127.0.0.1","-c","5000,100%","-w","3000,80%"],"execution_end":1663091644.161787,"execution_start":1663091640.088944,"exit_status":0,"output":"PING OK - Packet loss = 0%, RTA = 0.05 ms","performance_data":["rta=0.055000ms;3000.000000;5000.000000;0.000000","pl=0%;80;100;0"],"previous_hard_state":99,"schedule_end":1663091644.161905,"schedule_start":1663091640.087908,"scheduling_source":"icinga2-api","state":0,"ttl":0,"type":"CheckResult","vars_after":{"attempt":1,"reachable":true,"state":0,"state_type":1},"vars_before":{"attempt":1,"reachable":true,"state":0,"state_type":1}},"last_hard_state":0,"last_hard_state_change":1663028345.921676,"last_reachable":true,"last_state":0,"last_state_change":1663028345.921676,"last_state_down":0,"last_state_type":1,"last_state_unreachable":0,"last_state_up":1663091644.161787,"max_check_attempts":3,"name":"icinga2-api","next_check":1663091703.191943,"next_update":1663091771.339701,"notes":"","notes_url":"","original_attributes":null,"package":"_etc","paused":false,"previous_state_change":1663028345.921676,"problem":false,"retry_interval":30,"severity":0,"source_location":{"first_column":1,"first_line":18,"last_column":20,"last_line":18,"path":"/etc/icinga2/conf.d/hosts.conf"},"state":0,"state_type":1,"templates":["icinga2-api","generic-host"],"type":"Host","vars":{"disks":{"disk":{},"disk /":{"disk_partitions":"/"}},"http_vhosts":{"http":{"http_uri":"/"}},"notification":{"mail":{"groups":["icingaadmins"]}},"os":"Linux"},"version":0,"volatile":false,"zone":""},"joins":{},"meta":{},"name":"icinga2-api","type":"Host"}]}
root#icinga2-web:/#
PS: I'm fairly certain this is not a specific issue to icinga as I've had some random proxy issues w/ other containers. But, I can say I've tested this icinga compose setup outside corp domain and it worked fine 100%.
Partial Resolution!
I would still prefer to use CIDR to have no_proxy work via container name without having to adjust docker-compose/.env but I got it to work.
A few things I did:
Added lowercase to docker service -->:
cat << EOL >/etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=$httpproxyvar"
Environment="HTTPS_PROXY=$httpsproxyvar"
Environment="NO_PROXY=.$corpdom,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
Environment="http_proxy=$httpproxyvar"
Environment="https_proy=$httpsproxyvar"
Environment="no_proxy=.$corpdom,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
EOL
Added no_proxy in caps and lower to docker-compose containers and set in .env
Note: lower and CAPS should be used
environment:
- 'NO_PROXY=${NO_PROXY}'
- 'no_proxy=${NO_PROXY}'
NO_PROXY=.domain.org,127.0.0.0/8,172.16.0.0/12,icinga_icinga-net
I would prefer to append to the existing variable at least, but I tried the following and it made the variable no_proxy = ,icinga_icinga-net
NO_PROXY=$NO_PROXY,icinga_icinga-net
NO_PROXY=${NO_PROXY},icinga_icinga-net
Note: NO_PROXY was set on host via export
I still don't understand why it fails when using:
curl --noproxy -k -u root:c54854140704eafc https://172.30.0.4:5665/v1/objects/hosts
when I have no_proxy 172.16.0.0/12 which should equal 172.16.0.0 – 172.31.255.255 but doesn't work.
Update:
I tried setting no_proxy to the IP explicitly (no CIDR) and that worked, but it still failed w/ just container as host (no .icinga-net).
This is all related to this great post -->
https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/
This is the best I can come up with, happy to reward better answers!
Docker Setup (Global):
#!/bin/bash
#Proxy
ActiveProxyVar=127.0.0.7
#Domain
corpdom=domain.org
#NoProxy
NOT_PROXY=127.0.0.0/8,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8,.$corpdom
httpproxyvar=http://$ActiveProxyVar/
httpsproxyvar=http://$ActiveProxyVar/
mkdir ~/.docker
cat << EOL >~/.docker/config.json
{
"proxies":
{
"default":
{
"httpProxy": "$httpproxyvar",
"httpsProxy": "$httpsproxyvar",
"noProxy": "$NOT_PROXY"
}
}
}
EOL
mkdir -p /etc/systemd/system/docker.service.d
cat << EOL >/etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=$httpproxyvar"
Environment="HTTPS_PROXY=$httpsproxyvar"
Environment="NO_PROXY=$NOT_PROXY"
Environment="http_proxy=$httpproxyvar"
Environment="https_proy=$httpsproxyvar"
Environment="no_proxy=$NOT_PROXY"
EOL
systemctl daemon-reload
systemctl restart docker
#systemctl show --property Environment docker
#docker run hello-world
docker-compose.yaml:
environment:
- 'NO_PROXY=${NO_PROXY}'
- 'no_proxy=${NO_PROXY}'
.env:
--Basically, add docker-compose network then each container name...
NO_PROXY=127.0.0.0/8,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8,.icinga_icinga-net,icinga2-api,icinga2-web,icinga2-db,icinga2-webdb,icinga2-redis,icinga2-directordb,icinga2-icingadb,icinga2-web_director

Docker Private Registry only works with either HTTPS , or HTTP not both at the same time

I created a Self-signed certificate (with IP address - SAN) for Docker Private Registry and successfully access https://[IPADDRESS]/v2/_catalog but can't access it over the HTTP connection. It says ERR_INVALID_HTTP_RESPONSE and docker logs show :
http: TLS handshake error from 192.168.1.7:58316: tls: first record does not look like a TLS handshake
I follow this instructions for self-signed IP certificate: https://nodeployfriday.com/posts/self-signed-cert/ :
[req]
default_bits = 4096
default_md = sha256
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = VA
L = SomeCity
O = MyCompany
OU = MyDivision
CN = 192.168.13.10
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = #alt_names
[alt_names]
IP.1 = 192.168.1.7
I saved a conf file and run generate command:
openssl req -new -nodes -x509 -days 365 -keyout domain.key -out domain.crt -config <path/to/req/file/from/above>
then created domain.crt and domain.key, then started container:
docker run -d -p 5000:5000 --restart=always --name registry \
-v /home/dataserver/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry
How can I access it over HTTP? The registry only work with HTTPS :/ Thanks now

pika rabbitmq docker with tls-gen connection reset, no logs

I've configured a docker w rabbitmq, copied the certs from tls-gen there, stop/started and I can only get the connection established and a server reset right after sending a TLS Client Hello. No useful message from ssl. Logs all empty. Troubleshoot TLS guide didn't help. Thanks for any help.
docker run -d -it --hostname=[REDACTED] -e RABBITMQ_LOGS=/var/log/rabbitmq/ -e RABBITMQ_SASL_LOGS=/var/log/rabbitmq/ -e RABBITMQ_DEFAULT_USER=[REDACTED] -e RABBITMQ_DEFAULT_PASS=[REDACTED] --name rabbitmq1 -p [REDACTED]:5672 -p [REDACTED]:15672 -v /scratch/databases/rabbitmq/var/lib/rabbitmq:/var/lib/rabbitmq rabbitmq:3-management
docker cp tls-gen/basic/result/ca_certificate.pem rabbitmq1:/etc/rabbitmq/
docker cp tls-gen/basic/result/server_key.pem rabbitmq1:/etc/rabbitmq/
docker cp tls-gen/basic/result/server_certificate.pem rabbitmq1:/etc/rabbitmq/
docker exec -it rabbitmq1 /bin/bash -c "cd /etc/rabbitmq;chown rabbitmq:rabbitmq *.pem"
docker cp rabbitmq.conf rabbitmq1:/etc/rabbitmq/
docker exec -it rabbitmq1 /bin/bash -c "cd /etc/rabbitmq;chown rabbitmq:rabbitmq rabbitmq.conf"
I was sure to restart the container so the changes were applied. Also double checked cert permissions.
docker:/etc/rabbitmq/
-rw-rw-r--. 1 rabbitmq rabbitmq 1.2K Nov 22 02:29 ca_certificate.pem
drwxrwxrwx. 1 rabbitmq rabbitmq 61 Nov 6 01:51 conf.d
-rw-r--r--. 1 root root 71 Nov 22 07:34 enabled_plugins
-rw-r--r--. 1 rabbitmq rabbitmq 578 Nov 22 07:34 rabbitmq.conf
-rw-rw-r--. 1 rabbitmq rabbitmq 1.3K Nov 22 02:29 server_certificate.pem
-rw-------. 1 rabbitmq rabbitmq 1.9K Nov 22 02:29 server_key.pem
Redirected logs here knowing that the container start env variables should overide. Either way, logs seemed empty.
/etc/rabbitmq/rabbitmq.conf
loopback_users.guest = false
# listeners.tcp.default = 5672
management.tcp.port = 15672
listeners.tcp = none
listeners.ssl.default = 5671
ssl_options.cacertfile = /etc/rabbitmq/ca_certificate.pem
ssl_options.certfile = /etc/rabbitmq/server_certificate.pem
ssl_options.keyfile = /etc/rabbitmq/server_key.pem
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.password = [REDACTED]
log.dir = /tmp/
log.file = r.log
log.file.level = debug
Tried multiple configurations for context.options to no avail.
receive.py (python)
#!/usr/bin/env python
import ssl, pika, sys, os
import traceback
from pika.credentials import ExternalCredentials
def main():
connection = None
credentials = pika.PlainCredentials('[REDACTED]', '[REDACTED]')
context = ssl.create_default_context(cafile="tls-gen/basic/result/ca_certificate.pem")
context.load_cert_chain("tls-gen/basic/result/client_certificate.pem", "tls-gen/basic/result/client_key.pem")
#context.options = dict(ssl_version=ssl.PROTOCOL_TLSv1, cert_reqs=ssl.CERT_REQUIRED)
ssl_options = pika.SSLOptions(context, "[REDACTED]")
#ssl_options = pika.SSLOptions(ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT), "local")
#ssl_options = pika.SSLOptions(ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER), "localhost")
#ssl_options = ssl_options | pika.SSLOptions(ssl.SSLContext(ssl.PROTOCOL_TLSv1_2), "[REDACTED]")
try:
connection = pika.BlockingConnection(pika.ConnectionParameters(host='[REDACTED]',
port=[REDACTED],
virtual_host='/',
ssl_options = ssl_options)),
credentials=credentials))
#credentials=ExternalCredentials()))
except Exception as e:
exc_type, _, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print("receive: ({0}) {1} {2} {3}".format(str(e), exc_type, fname, exc_tb.tb_lineno))
traceback.print_exc()
sys.exit(1)
channel = connection.channel()
channel.queue_declare(queue='hello')
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
print('Interrupted')
try:
sys.exit(0)
except SystemExit:
os._exit(0)
A wrong PEM password here generates a very different message, I'm sure I'm using the right one.
python3 receive.py
Enter PEM pass phrase:
receive: ([Errno 104] Connection reset by peer) <class 'ConnectionResetError'> receive.py 18
Traceback (most recent call last):
File "receive.py", line 18, in main
connection = pika.BlockingConnection(pika.ConnectionParameters(host='[REDACTED]',
File "[REDACTED]/rabbitmq/pika/venv/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 359, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "[REDACTED]/rabbitmq/pika/venv/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 450, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "[REDACTED]/rabbitmq/pika/venv/lib/python3.8/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
It looks like openssl likes the connection.
openssl s_client -connect localhost:5671
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 CN = TLSGenSelfSignedtRootCA, L = $$$$
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=0 CN = [REDACTED], O = server
verify return:1
---
Certificate chain
0 s:CN = [REDACTED], O = server
i:CN = TLSGenSelfSignedtRootCA, L = $$$$
1 s:CN = TLSGenSelfSignedtRootCA, L = $$$$
i:CN = TLSGenSelfSignedtRootCA, L = $$$$
---
Server certificate
-----BEGIN CERTIFICATE-----
[REDACTED]
-----END CERTIFICATE-----
subject=CN = [REDACTED], O = server
issuer=CN = TLSGenSelfSignedtRootCA, L = $$$$
---
Acceptable client certificate CA names
CN = TLSGenSelfSignedtRootCA, L = $$$$
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2338 bytes and written 431 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: [REDACTED]
Session-ID-ctx:
Master-Key: [REDACTED]
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1606031943
Timeout : 7200 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
Extended master secret: no
---
closed
here the steps I took to fix this:
Get rid of the docker and install a local rabbitmq on mac for debugging this configuration. For some reason, the rabbitmq docker wasn't logging anything;
I've added a real valid letsencrypt certificate to the server;
first error I saw in the client after that was the following:
ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1076)
This is because verify_peer is enabled, in that case we needed to bundle the ca_certificate.pem from tls-gen/basic/result/ca_certificate.pem to the chain.pem
cat ../tls-gen/basic/result/ca_certificate.pem chain.pem > chain_bundle.pem
rabbitmq.conf
...
ssl_options.cacertfile = /usr/local/etc/rabbitmq/chain_bundle.pem # <- bundled w self signed cert
ssl_options.certfile = /usr/local/etc/rabbitmq/cert.pem
ssl_options.keyfile = /usr/local/etc/rabbitmq/privkey.pem
...
after that, a different error on pika showed:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)
That's because pika didn't like tls-gen/basic/result/ca_certificate.pem, instead, I used venv//lib/python3.7/site-packages/certifi/cacert.pem after installing pip install certifi.
Later on, I bundled tls-gen/basic/result/ca_certificate.pem + venv//lib/python3.7/site-packages/certifi/cacert.pem, and added a password to context.load_cert_chain
Wireshark shows messages fully encrypted, now I just need to deploy.
EDIT:
I was able to run it via docker after re-trying the configuration that worked:
#/bin/bash
docker run -d -it --hostname=\
-e RABBITMQ_LOGS=/tmp/log\
-e RABBITMQ_DEFAULT_USER=\
-e RABBITMQ_DEFAULT_PASS=\
-e RABBITMQ_SSL_CACERTFILE=/cert/chain_bundle2.pem \
-e RABBITMQ_SSL_CERTFILE=/cert/cert.pem\
-e RABBITMQ_SSL_KEYFILE=/cert/privkey.pem\
-e RABBITMQ_SSL_FAIL_IF_NO_PEER_CERT=true\
-e RABBITMQ_SSL_VERIFY=verify_peer\
--name rabbitmq1-verifypeer -p ... :5671 -p ... :15672\
-v ... /rabbitmq/var/log:/tmp/log\
-v ... /rabbitmq/var/lib/rabbitmq:/var/lib/rabbitmq\
-v ... /cert/:/cert/ rabbitmq:3.8.9-management

Docker private registry using selfsigned certificates

I want to run a private docker registry which is widely available.
So I will be able to push and pull images from other servers.
I'm following this tutorials: doc1 & doc2
I performed 3 steps:
First I've created my certificate and key (as CNAME I filled in my ec2-hostname)
mkdir -p certs && openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
-x509 -days 365 -out certs/domain.crt
Than I've created my docker registry, using this key.
docker run -d -p 5000:5000 --restart=always --name registry \
-v `pwd`/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
registry:2
Than I copied the content of domain.crt to /etc/docker/certs.d/ec2-xx-xx-xx-xx.compute.amazonaws.com:5000/ca.crt
I restarted my docker: sudo service docker restart
When I try to push an image I get the following error:
unable to ping registry endpoint https://ec2-xx-xx-xx-xx.compute.amazonaws.com:5000/v0/
v2 ping attempt failed with error: Get https://ec2-xx-xx-xx-xx.compute.amazonaws.com:5000/v2/: net/http: TLS handshake timeout
v1 ping attempt failed with error: Get https://ec2-xx-xx-xx-xx.compute.amazonaws.com:5000/v1/_ping: net/http: TLS handshake timeout
I really don't know what I'm missing or doing wrong. Can someone please help me. Thanks
I'm not sure if you copy/pasted your pwd directly... but the file path should be /etc/docker/certs.d
You currently have etc/docker/cert.d/registry.ip:5000/domain.crt
The error message says "TLS handshake timeout". This indicates that either no process is listening on port 5000 (check using netstat) or the port is closed from the location where you are trying to push the image (open port in the AWS security group).
From what I've seen docker login is way more sensitive to properly crafted self-signed certs than browsers are + there's an interesting gotcha I'll point out at the very bottom, so read the whole thing.
According to this site:
https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html
Bash# openssl x509 -noout -text -in ca.crt
X509v3 Basic Constraints: critical
CA:TRUE
^You should see something like this is you provisioned your certs right.
While following random how-to guides on the net I was able to generate ca.crt and website.crt
When I ran the above command I didn't see that output, but I noticed:
If I imported the cert as trusted in Mac or Win my browser would be happy and say yeap valid cert, but docker login on RHEL7 would complain with messages like)
x509: certificate signed by unknown authority
I tried following directions related to using: /etc/docker/certs.d/mydockerrepo.lan:5000/ca.crt
on https://docs.docker.com/engine/security/certificates/
It got me a better error message (which caused me to find the above site in the first place)
x509: certificate signed by unknown authority (possibly because of
"x509: invalid signature: parent certificate cannot sign this kind of
certificate" while trying to verify candidate authority certificate
After 2 days of messing around I figured it out:
When I was taught programming I was taught the concept of a short self-contained example, so going to try doing that here for ansible, leveraging the openssl built-in modules, I'm running latest ansible 2.9, but this should work for ansible 2.5++ in theory:
Short Self Contained Example:
#Name this file generatecertificates.playbook.yml
#Run using Bash# ansible-playbook generatecertificates.playbook.yml
#
#What to Expect:
#Run Self Contained Stand Alone Ansible Playbook --Get-->
# currentworkingdir/certs/
# ca.crt
# ca.key
# mydockerrepo.private.crt
# mydockerrepo.private.key
#
#PreReq Ansible 2.5++
#PreReq Bash# pip3 install cryptograph >= 1.6 or PyOpenSSL > 0.15 (if using selfsigned provider)
---
- hosts: localhost
connection: local
gather_facts: no
vars:
- caencryptionpassword: "myrootcaencryptionpassword"
- dockerepodns: "mydockerrepo.private"
- rootcaname: "My Root CA"
tasks:
- name: get current working directory
shell: pwd
register: pathvar
- debug: var=pathvar.stdout
- name: Make sub directory
file:
path: "{{pathvar.stdout}}/certs"
state: directory
register: certsoutputdir
- debug: var=certsoutputdir.path
- name: "Generate Root CA's Encrypted Private Key"
openssl_privatekey:
size: 4096
path: "{{certsoutputdir.path}}/ca.key"
cipher: auto
passphrase: "{{caencryptionpassword}}"
- name: "Generate Root CA's Self Signed Certificate Signing Request"
openssl_csr:
path: "{{certsoutputdir.path}}/ca.csr"
privatekey_path: "{{certsoutputdir.path}}/ca.key"
privatekey_passphrase: "{{caencryptionpassword}}"
common_name: "{{rootcaname}}"
basic_constraints_critical: yes
basic_constraints: ['CA:TRUE']
- name: "Generate Root CA's Self Signed Certificate"
openssl_certificate:
path: "{{certsoutputdir.path}}/ca.crt"
csr_path: "{{certsoutputdir.path}}/ca.csr"
provider: selfsigned
selfsigned_not_after: "+3650d" #Note: Mac won't trust by default due to https://support.apple.com/en-us/HT210176, but you can explitly trust to make it work.
privatekey_path: "{{certsoutputdir.path}}/ca.key"
privatekey_passphrase: "{{caencryptionpassword}}"
register: cert
- debug: var=cert
- name: "Generate Docker Repo's Private Key"
openssl_privatekey:
size: 4096
path: "{{certsoutputdir.path}}/{{dockerepodns}}.key"
- name: "Generate Docker Repo's Certificate Signing Request"
openssl_csr:
path: "{{certsoutputdir.path}}/{{dockerepodns}}.csr"
privatekey_path: "{{certsoutputdir.path}}/{{dockerepodns}}.key"
common_name: "{{dockerepodns}}"
subject_alt_name: 'DNS:{{dockerepodns}},DNS:localhost,IP:127.0.0.1'
- name: "Generate Docker Repo's Cert, signed by Root CA"
openssl_certificate:
path: "{{certsoutputdir.path}}/{{dockerepodns}}.crt"
csr_path: "{{certsoutputdir.path}}/{{dockerepodns}}.csr"
provider: ownca
ownca_not_after: "+365d" #Cert valid 1 year
ownca_path: "{{certsoutputdir.path}}/ca.crt"
ownca_privatekey_path: "{{certsoutputdir.path}}/ca.key"
ownca_privatekey_passphrase: "{{caencryptionpassword}}"
register: cert
- debug: var=cert
Interesting Gotcha/Final Step:
RHEL7Bash# sudo cp ca.crt /etc/pki/ca-trust/source/anchors/ca.crt
RHEL7Bash# sudo update-ca-trust
RHEL7Bash# sudo systemctl restart docker
The gotcha is that you have to restart docker, for docker login to recognize updates to CA's newly added to the system.

How to create docker registry mirror on CentOS

I try and try to create mirror in docker-registry. I have read the tutorial in this. And use the way in this. I'm sure I have add variable to docker daemon. And I succeed on mac OS X. But it didn't work at all on my centOS.
I run my docker daemon using this command:
docker -g /opt/apps/docker/lib --insecure-registry http://10.11.150.76:5555 --registry-mirror=http://10.11.150.76:5555 -d
and I use this command to check docker daemon:
ps -ef | grep "docker"
It really has added to docker:
root 1232 30203 0 20:47 ? 00:00:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 10022 -container-ip 172.17.0.22 -container-port 22
root 1322 735 0 20:57 pts/7 00:00:00 grep --color=auto docker
root 30202 1 0 20:15 ? 00:00:00 sudo http_proxy=http://10.16.10.129:9526/ docker -g /opt/apps/docker/lib --insecure-registry http://10.11.150.76:5555 --registry-mirror=http://10.11.150.76:5555 -d
Then, I use this command to create mirror:
docker run -d -p 5555:5000 -e STORAGE_PATH=/mirror -e STANDALONE=false -e MIRROR_SOURCE=https://registry-1.docker.io -e MIRROR_SOURCE_INDEX=https://index.docker.io -v /Users/v11/Documents/docker-mirror:/mirror --restart=always --name mirror registry
Next,I pull the new image, for example:
docker pull ubuntu
It did't work at all, because I can't find image in my path "/Users/v11/Documents/docker-mirror". I print the mirror log information and it will show me error information:
ConnectionError: HTTPSConnectionPool(host='index.docker.io',
port=443): Max retries exceeded with url:
/v1/repositories/library/hello-world/images (Caused by : [Errno 110] Connection timed out)
and docker log :
INFO[0023] POST /v1.19/images/create?fromImage=hello-world%3Alatest
ERRO[0027] Unable to create endpoint for http://10.11.150.76:5555/:
invalid registry endpoint https://10.11.150.76:5555/v0/: unable to
ping registry endpoint https://10.11.150.76:5555/v0/ v2 ping attempt
failed with error: Get https://10.11.150.76:5555/v2/: EOF v1 ping
attempt failed with error: Get https://10.11.150.76:5555/v1/_ping:
EOF. If this private registry supports only HTTP or HTTPS with an
unknown CA certificate, please add --insecure-registry
10.11.150.76:5555 to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the
flag; simply place the CA certificate at
/etc/docker/certs.d/10.11.150.76:5555/ca.crt
I really search many relevant questions to find solving. But I still have no idea about it. How to do it? Thanks.
The '--registry-mirror' flag to docker has been designed to only work for mirroring the official docker.io repository. In 2015, they said the registry V2 would be made to work with the '--registry-mirror' flag, but this capability is still unavailable.

Resources