I have a docker-compose containing two services:
a Python uWSGI app on port 5000 and
an Nginx reverse proxy on port 433 (with a self-signed certificate).
The latter has been set up to proxy requests into the service running in the first container, which is doing it fine.
The Python service has an endpoint that receives a POST request with a JSON payload (an object) containing a $schema attribute that allows the service to validate it:
"$schema": "https://localhost/schemas/schema.json"
The interesting part is that this schema has been set up on the service itself, which means that the attribute passed in - as the URL indicates - is being served by it. The problem happens when the service tries to validate a payload against a schema that it cannot GET (requesting the URL above).
My conclusions
I can understand why fetching from localhost fails, as there isn't any service running on localhost, 443 in the respective container:
jsonschema.exceptions.RefResolutionError: HTTPSConnectionPool(host='localhost', port=443):
Max retries exceeded with url: /schemas/schema.json
(Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7c26d9f5b0>:
Failed to establish a new connection: [Errno 111] Connection refused'))
As an alternative, I tried to use the IP for the Nginx container (https://nginx/schemas/schema.json), which actually finds the container, but goes through the SSL verification on the self-signed certificate.
jsonschema.exceptions.RefResolutionError: HTTPSConnectionPool(host='nginx', port=443):
Max retries exceeded with url: /schemas/schema.json
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1108)')))
In a third attempt, I used 0.0.0.0 which correctly routes the request and validates the object correctly.
However, I couldn't understand why it worked. As well as why the service is able to make the request through the proxy but I cannot CURL myself inside the Python service container simply using curl --insecure https://0.0.0.0/...
Any help in figuring out the real cause or a better understanding of how networking should work here would be very appreciated.
Related
So doing a POC, with Nginx load balancer and freeradius docker. When the request packet is sent to the freeradius container from the proxy, the reply is received by the proxy and all is well.
So the containers are working as expected 100%.
However when the proxy sends the request to the freeradius container through the Nginx load balancer, the freeradius gets the access request packet, but then the access accept packet replies back with the containers IP.
So the proxy sends another request and the freeradius container replies back with the containers IP.
Proxy sends the request to radius:
Sent Accounting-Request Id 90 from xxx.xxx.xxx.xxx:55039 to xxx.xxx.xxx.xxx:1813 length 467
Radius replies
Sent Accounting-Response Id 1 from 172.18.0.5:1813 to xxx.xxx.xxx.xxx:37741 length 25
Proxy never gets the response:
(29) No proxy response, giving up on request and marking it done
(29) ERROR: Failing proxied request for user "vlan1106/lab-1-1070-1106", due to lack of any response from home server 154.119.32.156 port 1813
The Nginx load balancers are configured with transparent, as I need to know the source IP from where the request is made. When not in transparent then it works as the reply packet has the correct IP in the header.
Is there anything that I'm missing here? Been head butting this problem since Saturday.
So I have GitLab EE server (Omnibus) installed and set up on Ubuntu 20.04.
Next, following official documentation found on GitLab PlantUML integration, I started PlantUML in a docker container which I did with the following command:
docker run -d --name plantuml -p 8084:8080 plantuml/plantuml-server:tomcat
Next, I also configured /etc/gitlab/gitlab.rb file and added next line for redirection as my GitLab server is using SSL:
nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n"
In the GitLab server GUI in admin panel, in Settings -> General, when I expand PlantUML, I set the value of PlantUML URL to (two ways):
1st approach:
https://HOSTNAME:8084/-/plantuml
Then, when trying to reach it through the browser through this address(https://HOSTNAME:8084/-/plantuml), I get
This site can’t provide a secure connection.
HOSTNAME sent an invalid response.
ERR_SSL_PROTOCOL_ERROR
2nd approach:
Also I tried to put before that I tried different value in in Settings -> General -> PlantUML -> PlantUML URL:
https://HOSTNAME/-/plantuml
Then, when trying to reach it through the browser through this address (https://HOSTNAME/-/plantuml), I get
502
Whoops, GitLab is taking too much time to respond
In both cases when I trace logs with gitlab-ctl tail I get the same errors:
[crit] *901 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: CLIENT_IP, server: 0.0.0.0:443
[error] 1123593#0: *4 connect() failed (113: No route to host) while connecting to upstream
My question is which of the above two ways is correct to access PlantUML with the above configuration and is there any configuration I am missing?
I believe the issue is that you are running the plantuml in a docker container and then trying to reach it via gitlab (on localhost) with name.
In order to check if that is the issue please change
proxy_pass http://plantuml:8080/
to
proxy_pass http://localhost:8080/
and trying again with the first approach.
Your second approach seems to be missing the container port in the url.
Hello i am having trouble with vagrant setup.
So i am trying to ping serverless API which runs on http://localhost:3000/ (and it's outside vagrant project).
Right now my vagrant project runs on https://localhost:4443/.
Overall trying to CURL request from vagrant project to another serverless project.
Tried to use http://localhost:3000/ in CURL request but getting Failed to connect to localhost port 3000: Connection refused
Tried to use VM ip 10.0.2.15 address same
Tried to do port forwarding in vagrantfile config.vm.network :forwarded_port, guest: 3000, host: 3000 and use machine IP address 192.168.0.16, getting empty response from server, when i try to do telnet 192.168.0.16 3000 getting
Trying 192.168.0.16...
Connected to 192.168.0.16.
Escape character is '^]'.
Connection closed by foreign host.
Any idea what to try?
I had to use VM IP address something like
curl -X GET http://10.0.2.2:3000
These errors may be caused due to follow reasons, ensure the following steps are followed. To connect the local host with the local virtual machine(host). Here, I'am connecting http://localhost:3001/ to the http://abc.test Steps to be followed:
1.We have to allow CORS, placing Access-Control-Allow-Origin: in header of request may not work. Install a google extension which enables a CORS request.*
2.Make sure the credentials you provide in the request are valid.
3.Make sure the vagrant has been provisioned. Try vagrant up --provision this make the localhost connect to db of the homestead.
Try changing the content type of the header. header:{ 'Content-Type' : 'application/x-www-form-urlencoded; charset=UTF-8;application/json' }
this point is very important.
I am not able to login to local docker registry,getting error as:
"**Error response from daemon: Get http://127.0.0.1:5000/v1/users/: read tcp 127.0.0.1:46779->127.0.0.1:5000: read: connection reset by peer"**
Your registry is most probably not behind a SSL proxy or SSL configured, so you need to configure your client-docker to allow unsecure registrys, see https://docs.docker.com/registry/insecure/
I am trying to set up a new Docker Registry (v2) with HAProxy. For the Docker Registry I am using the image from the docker hub and running it with docker run -d -p 5000:5000 -v /path/to/registry:/tmp/registry registry:2.0.1. And this is a subset of my HAProxy configuration:
global
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048
userlist auth_list
group docker_registry users root
user root password ***PASSWORD***
backend docker-registry
server 127.0.0.1:5000_localhost 127.0.0.1:5000 cookie 127.0.0.1:5000_localhost
frontend shared-frontend
mode http
bind *:80
bind *:443 ssl crt *** CERT FILES ***
option accept-invalid-http-request
acl domain_d.mydomain.com hdr(host) -i d.mydomain.com
acl auth_docker_registry_root http_auth(auth_list) root
redirect scheme https if !{ ssl_fc } domain_d.mydomain.com
http-request auth realm Registry if !auth_docker_registry_root { ssl_fc } domain_d.mydomain.com
use_backend docker-registry if domain_d.mydomain.com
The important things to note are that I am using HAProxy to do SSL termination and HTTP auth rather than the registry.
My issue occurs when I try to login to the new registry. If I run docker login https://d.mydomain.com/v2/ then enter the user root and password I get the following error messages:
Docker Client:
FATA[0009] Error response from daemon: invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt
Docker Daemon:
ERRO[0057] Handler for POST /auth returned error: invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt
ERRO[0057] HTTP Error: statusCode=500 invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt
So I try adding --insecure-registry d.mydomain.com to:
/etc/default/docker with DOCKER_OPTS= -H unix:///var/run/docker.sock --insecure-registry d.mydomain.com
the arguments of starting docker manually with docker -d --insecure-registry d.mydomain.com
neither of these, or any other I have found online, work. Each time, after restarting docker and attempting to log in again gives me the same error message.
A few other things I have tried:
In a browser going to d.mydomain.com results in a 404
In a browser going to d.mydomain.com/v2/ results in: {}
Replacing https://d.mydomain.com/v2/ in the login command with all of these with no success
http://d.mydomain.com/v2/
d.mydomain.com/v2/
http://d.mydomain.com/
d.mydomain.com/
This setup with HAProxy doing the SSL termination and HTTP auth has worked in the past using the first version of the registry and older versions of docker. So has anything in Docker registry v2 changed? Does this still work? If it hasn't changed, why won't the --insecure-registry flag do anything anymore?
Also, I have been working on getting this working for a while so I may have forgotten all the things I have tried. If there is something that may work, let me know and I will give it a try.
Thanks,
JamesStewy
Edit
This edit has been moved to the answer below
I have got it working. So here is my new config:
haproxy.cfg
global
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048
userlist auth_list
group docker_registry users root
user root password ***PASSWORD***
backend docker-registry
server 127.0.0.1:5000_localhost 127.0.0.1:5000 cookie 127.0.0.1:5000_localhost
backend docker-registry-auth
errorfile 503 /path/to/registry_auth.http
frontend shared-frontend
mode http
bind *:80
bind *:443 ssl crt *** CERT FILES ***
option accept-invalid-http-request
acl domain_d.mydomain.com hdr(host) -i d.mydomain.com
redirect scheme https if !{ ssl_fc } domain_d.mydomain.com
acl auth_docker_registry_root http_auth(auth_list) root
use_backend docker-registry-auth if !auth_docker_registry_root { ssl_fc } domain_d.mydomain.com
rsprep ^Location:\ http://(.*) Location:\ https://\1
use_backend docker-registry if domain_d.mydomain.com
registry_auth.http
HTTP/1.0 401 Unauthorized
Cache-Control: no-cache
Connection: close
Content-Type: text/html
Docker-Distribution-Api-Version: registry/2.0
WWW-Authenticate: Basic realm="Registry"
<html><body><h1>401 Unauthorized</h1>
You need a valid user and password to access this content.
</body></html>
The differences being the http-request auth line has been replaced with use_backend docker-registry-auth. The backend docker-registry-auth has no servers to it will always give a 503 error. But the 503 error file has been changed to registry_auth.http. In registry_auth.http the error code is overridden to 401, the header WWW-Authenticate is set to Basic realm="Registry", the basic HAProxy 401 error page is supplied and, most importantly, the header Docker-Distribution-Api-Version is set to registry/2.0.
As a result this hacky work-around setup works exactly the same as the old http-request auth line except the custom header Docker-Distribution-Api-Version is now set. This allows this set up to pass the test which starts on line 236 of https://github.com/docker/docker/blob/v1.7.0/registry/endpoint.go.
So now when I run docker login d.mydomain.com, login is successful and my credentials are added to .docker/config.json.
The second issue was that I couldn't push to the new repository even through it logged in. This was fixed by adding the rsprep line in the frontend. What this line does is modify the Location header (if it exists) to turn all http:// to https://.
I also found this bit of documentation for future reference.
As a small clarification to the previous answer: I had to change this line:
WWW-Authenticate: Basic realm="Registry"
To this:
WWW-Authenticate: Basic realm="Registry realm"
and then everything worked...
BTW, hashing the pass can be done using mkpasswd (part of whois deb package)