SSL proxy pass with NGINX leading to bad gateway through Docker - docker

Background
My setup is based on the following tutorial:
Dockerizing Django with Postgres, Gunicorn, and NGINX
TL;DR: (italics: not covered by tutorial; personal adventure)
3 Docker services for: nginx -> django -> postgres (arrow indicating dependency)
Nginx proxy passes requests to exposed port in Django service.
HTTP (non-SSL) requests working
require SSL connections by redirecting http -> https
Details
I've generated a self-signed certificate to test out ssl redirects with NGINX locally before trying to get it to work on a VPS in production. I'm quite new to working with NGINX and so I'm not entirely sure what's going wrong or how to diagnose problems.
Here's what I want to happen with the NGINX file I've provided below... (spoilers: it doesn't):
Go to http://localhost
Get redirected to https://localhost
Warning from browser about a self-signed cert; accept warning and proceed
Site rendered fine, SSL redirect working!
But this isn't the case. I get a 502 Bad Gateway, and NGINX outputs the following logs:
prod_1 | 192.168.144.1 - - [03/Jun/2019:00:01:44 +0000] "GET / HTTP/1.1" 502 158 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0" "-"
prod_1 | 2019/06/03 00:01:44 [error] 8#8: *1 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 192.168.144.1, server: , request: "GET / HTTP/1.1", upstream: "https://192.168.144.3:8000/", host: "localhost"
Can anyone tell me what's going on or how to fix it? I feel like there's probably a whole bunch wrong with my conf file even outside of the SSL redirect, but I don't really know how to identify any problems. The conf file is below...
upstream mysite {
server web:8000;
}
# redirect http traffic to https
server {
listen 80;
listen [::]:80;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
location / {
proxy_pass https://mysite;
proxy_ssl_server_name on;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
}
location /assets/ {
alias /usr/src/site/assets/;
}
location /media/ {
alias /usr/src/site/media/;
}
ssl_certificate /etc/ssl/certificates/site.crt;
ssl_certificate_key /etc/ssl/certificates/site.key;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
}

Based on your error
prod_1 | 2019/06/03 00:01:44 [error] 8#8: *1 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 192.168.144.1, server: , request: "GET / HTTP/1.1", upstream: "https://192.168.144.3:8000/", host: "localhost"
I would say it's likely that you have some kind of protocol mismatch between Nginx and Django. It's likely that Django is expecting non-secure communication (http). Your nginx configuration indicates that you have configured it to communicate with Django via https:
proxy_pass https://mysite;
My suggestion is to make sure that the communication between Nginx and Django is using the same protocol, either http or https.
Whether you want to use http or https is up to you. There are two differing schools of thought on whether http is secure here.
The first school of thought is that http IS secure in this scenario, since the communication is within the same machine.
The second school of thought is to secure all communications, and to use https. However if you agree with this line of thinking, you will have to make sure that the communication between your web server and database is also using a secure protocol. After all, you are only as secure as your weakest link.
I tend to lean towards the first school of thought. Though this is not necessarily what is appropriate for you.

Related

Why doesn't Nginx reverse proxy on WSL2 see Rails app in Docker Container using hostname?

I've got several Rails websites running in Docker dev containers. Docker is running in WSL (Ubuntu 20.04) on Windows 11. Nginx is running in Ubuntu as a reverse proxy, IIS is turned off in Windows. The Ubuntu /etc/hosts file is automatically populated from the hosts file in Windows. It is set up like this because others on the team are running Linux on Macs but I switch between Rails and .Net development.
An example website is mysite1.localhost which is exposed on port 8081 on Docker and there is an entry of '127.0.0.1 mysite1.localhost' in both hosts files.
The problem I have is browsing (Chrome on Windows) localhost:8081 returns 200 from the website, great, but using the hostname mysite1.localhost returns 502 Bad Gateway.
I am assuming Nginx doesn't know about Docker or something like that?
Here is the mysite1.conf for Nginx:
server {
listen 80;
listen [::]:80;
server_name mysite1.localhost;
resolver 127.0.0.1;
location ~* "^/shared-nav" {
proxy_set_header Accept-Encoding "";
proxy_pass http://localhost:3000/stuff$is_args$args;
}
location / {
ssi on;
ssi_silent_errors off;
log_subrequest on;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8081;
add_header Cache-Control "no-cache";
if ($request_filename ~* ^.*?/([^/]*?)$) {
set $filename $1;
}
if ($filename ~* ^.*?\.(eot)|(ttf)|(woff)|(woff2)$) {
add_header Access-Control-Allow-Origin *;
}
}
}
I can see two problems in nginx/error.log:
2023/02/02 08:50:10 [warn] 2841#2841: conflicting server name "mysite1.localhost" on 0.0.0.0:80, ignored
2023/02/02 08:50:12 [error] 2845#2845: *52 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: mysite1.localhost, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8081/", host: "mysite1.localhost"
It doesn't seem to matter whether or not docker is running.
For the conflicting server name waring, I've tried looking for temporary files that need deleting but cannot find anything.
Most of the other questions I've looked at involve solving problems with containerized Nginx where as this is sitting in WSL.
Please let me know if I can better explain the problem, thanks for any help.
One thing I didn't take into consideration was using VS Code to dev in the containers means that the ports were forwarded to Windows - although the containers are running in WSL, VS Code is running in Windows.
So, I could either turn IIS back on and use it as a reverse proxy or try Nginx for Windows. I've opted for the latter as it means I can share the same config files as the Linux guys and will see how it works out, for now I can browse the websites by hostname.
If anyone else needs to work with this set up, I'm happy to pass on my experiences.

How to call GRPC Server in NGINX config

How to call GRPC Server which is located in docker container on Swarm cluster from NGINX reverse proxy?
GRPC Server in container/service called webui with kestrel development certificate installed
NGINX Proxy which is located outside the stack and routes access to Swarm stacks
GRPC Client is located on a separate virtual machine on another network, the browser page at https://demo.myorg.com is available
part nginx.conf
server {
listen 443 ssl;
server_name demo.myorg.com;
...
location / {
proxy_pass https://namestack_webui;
}
GRPC Client appsetting.json
{
"ConnectionStrings": {
"Database": "Data Source=Server_name;Initial Catalog=DB;User Id=user;Password=pass;MultipleActiveResultSets=True;"
}
...
"GRPCServerUri": "https://demo.myorg.com/",
...
}
}
Problem when connecting GRPC Client to Server, i get error
END] GetOpcDaServerSettingsQuery. Time spent: 7,7166ms
fail: Grpc.Net.Client.Internal.GrpcCall[6]
Error starting gRPC call.
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> System.ComponentModel.Win32Exception (0x80090367): No common application protocol exists between the client and the server. Application protocol negotiation failed..
--- End of inner exception stack trace ---
Tried to write and specify a kestrel development certificate (for GRPC Client) that is loaded into the Swarm stack (namestack) through which the other containers in the stack are authenticated, the error is the same.
I understand that it is necessary to specify in appsetting.json the GRPC Server container address (https://namestack_webui), but it is behind NGINX, and I can only specify the GRPC host address (https://demo.myorg.com), tell me what is wrong?
The perfect solution for such a case was not found online.
I finally figured out and found a solution to my question, and I publish it for discussion.
If there are no comments against, then mark it as correct, at least it works for me and will work for YOU.
to proxy grpc connections through NGINX in the configuration, the location section must specify something similar to the url /PackageName.ServiceName/MethodName (This is indicated here by https://learn.microsoft.com/en-aspnetus/aspnet/core/grpc/troubleshoot?view=aspnetcor7.0#unable-to-start-aspnet-core-grpc-app-on-macos )
This URL can be checked with the developer or in the logs when grpc client connects
Should be used to proxy directive grpc_pass grpcs://namecontainer;
Should use http2 protocol.
So the correct configuration file for nginx in my case should look like this
server {
listen 443 ssl **http2**;
server_name demo.myorg.com;
ssl_certificate ...;
ssl_certificate_key ...;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers RC4:HIGH:!aNULL:!MD5:!kEDH;
add_header Strict-Transport-Security 'max-age=604800';
underscores_in_headers on;
large_client_header_buffers 4 16k;
location / {
proxy_pass https://name_container;
# Configuration for WebSockets
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_cache off;
# WebSockets were implemented after http/1.0
proxy_http_version 1.1;
# Configuration for ServerSentEvents
proxy_buffering off;
# Configuration for LongPolling or if your KeepAliveInterval is longer than 60 seconds
proxy_read_timeout 100s;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-URL-SCHEME https;
}
location /App.Name.Api.Contract.ApiService/UpdateOpcDaTags {
grpc_pass grpcs://name_container;
}
}

Docker port is not working over https after setting up an SSL over ubuntu nginx

I set up the Letsencrypt certificate directly to an AWS EC2 Ubuntu instance running Nginx and a docker server using port 9998. The domain is set up on Route 53. Http is redirected to https.
So https://example.com is working fine but https://example.com:9998 gets ERR_SSL_PROTOCOL_ERROR. If I use the IP address like http://10.10.10.10:9997 is working and checked the server using port 9998 okay.
The snapshot of the server on docker is:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
999111000 img-server "/bin/sh -c 'java -j…" 21 hours ago Up 21 hours 0.0.0.0:9998->9998/tcp hellowworld
It seems something is missing between Nginx and the server using port 9998. How can I fix it?
Where have you configured the ssl certificate ? Only Nginx?
The reason why you cannot visit https://example.com:9998 using ssl protocal is that that port provides http service rather than https.
I suggest not to publish 9998 of hellowworld and proxy all the traffic with nginx (if Nginx is also started with docker and in the same network).
Configure https in Nginx and the origin sever provides http.
This is a sample configuration https://github.com/newnius/scripts/blob/master/nginx/config/conf.d/https.conf
server {
listen 80;
server_name example.com;
return 301 https://example.com$request_uri;
}
server {
listen 443;
server_name example.com;
access_log logs/example.com/access.log main;
error_log /var/log/nginx/debug.log debug;
ssl on;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
location / {
proxy_pass http://apache:80;
proxy_set_header Host $host;
proxy_set_header CLIENT-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ~ /.well-known {
allow all;
proxy_pass http://apache:80;
}
# Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store(Mac).
location ~ /\. {
deny all;
}
}

Error when accessing the Wildfly Management Console - Authentication required

I'm receiving the error Authentication required after I login in the Wildfly 13 Management Console.
If I type a user or password wrong, it asks again, but if I type correctly it shows the page with the error message (so I assume the user and password are correct, but something else after that gives the error).
I'm using docker to run a nginx container and a wildfly container.
The nginx listens externally on port 9991 and proxy pass the request to the wildfly container, but it shows the error described before.
It just happens with the Wildfly Console, every other request proxied, even request proxied to a websocket or to Wildfly on port 8080, are done successfully.
The Wildfly container listens externally on port 9990 and I can access the console successfully in this port. If on docker I map the port "9992:9990" I still can access the console successfully through port 9992.
So, it seems that this is not related to docker, but to the Wildfly Console itself. Probably some kind of authentication that is not happening successfully when using a reverse proxy in the middle.
I have a demo docker project on https://github.com/lucasbasquerotto/pod/tree/0.0.6, and you can download the tag 0.0.6 that has everything setup to work with Wildfly 13 and nginx, and to simulate this error.
git clone -b 0.0.6 --single-branch --depth 1 https://github.com/lucasbasquerotto/pod.git
cd pod
docker-compose up -d
Then, if you access the container directly in http://localhost:9990 with user monitor and password Monitor#70365 everything works.
But if you access http://localhost:9991 with the same credentials, through the nginx reverse proxy, you receive the error.
My nginx.conf file:
upstream docker-wildfly {
server wildfly:9990;
}
location / {
proxy_pass http://docker-wildfly;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
I've also tried with:
proxy_set_header X-Forwarded-Proto $scheme;
And also with the Authorization header (just the 2nd line and also with both):
proxy_set_header Authorization $http_authorization;
proxy_pass_header Authorization;
And also defining the host header with the port (instead of just $host):
proxy_set_header Host $server_addr:$server_port;
I've tried the above configurations isolated and combined together. All to no avail.
Any sugestions?
Has anyone successfully accessed the Wildfly Console through a reverse proxy?
Update (2018-09-22)
It seems Wildfly uses a digest authentication (instead of basic).
I see the header in the console like the following:
Authorization: Digest username="monitor", realm="ManagementRealm", nonce="AAAAAQAAAStPzpEGR3LxjJcd+HqIX2eJ+W8JuzRHejXPcGH++43AGWSVYTA=", uri="/console/index.html", algorithm=MD5, response="8d5b2b26adce452555d13598e77c0f63", opaque="00000000000000000000000000000000", qop=auth, nc=00000005, cnonce="fe0e31dd57f83948"
I don't see much documentation about using nginx to proxy pass requests with digest headers (but I think it should be transparent).
One question I saw here in SO is https://serverfault.com/questions/750213/http-digest-authentication-on-proxied-server, but there is no answer so far.
I saw that there is the nginx non-official module https://www.nginx.com/resources/wiki/modules/auth_digest/, but in the github repository (https://github.com/atomx/nginx-http-auth-digest) it says:
The ngx_http_auth_digest module supplements Nginx's built-in Basic
Authentication module by providing support for RFC 2617 Digest
Authentication. The module is currently functional but has only been
tested and reviewed by its author. And given that this is security
code, one set of eyes is almost certainly insufficient to guarantee
that it's 100% correct. Until a few bug reports come in and some of
the ‘unknown unknowns’ in the code are flushed out, consider this
module an ‘alpha’ and treat it with the appropriate amount of
skepticism.
Also it doesn't seem to me allright to hardcode the user and pass in a file to be used by nginx (the authentication should be transparent to the reverse proxy in this case).
In any case, I tried it and it correctly asks me to authenticate, even if the final destination does not have a digest authentication, like when trying to connect to the wildfly site (not console), it asks when trying to connect to nginx (before proxying the request), then it forwards successfully to the destination, except in the case of wildfly console, it keeps asking me to authenticate forever.
So I think this is not the solution. The problem seems to be in what the nginx is passing to the Wildfly Console.
I had the same problem with the HAL management console v3.3 and 3.2
I could not get ngnix HTTPS working due to authentication errors, even though the page prompted http basic auth user and pass
This was tested in standalone mode on the same server
My setup was :
outside (https) -> nginx -> http://halServer:9990/
This resulted in working https but with HAL authentication errors (seen in the browsers console) the webpage was blank.
At first access the webpage would ask http basic auth credentials normally, but then almost all https requests would return an authentication error
I managed to make it work correctly by first enabling the HAL console https with a self signed certificate and then configuring nginx to proxy pass to the HAL HTTPS listener
Working setup is :
outside (https) -> nginx (https) -> https://halServer:9993/
Here is the ngnix configuration
server {
listen 80;
listen [::]:80;
listen 443 ssl;
listen [::]:443 ssl;
server_name halconsole.mywebsite.com;
# SSL
ssl_certificate /keys/hal_fullchain.pem;
ssl_certificate_key /keys/hal_privkey.pem;
ssl_trusted_certificate /keys/hal_chain.pem;
# security
include nginxconfig.io/security.conf;
# logging
access_log /var/log/nginx/halconsole.mywebsite.com.access.log;
error_log /var/log/nginx/halconsole.mywebsite.com.error.log warn;
# reverse proxy
location / {
# or use static ip, or nginx upstream
proxy_pass https://halServer:9993;
include nginxconfig.io/proxy.conf;
}
# additional config
include nginxconfig.io/general.conf;
include nginxconfig.io/letsencrypt.conf;
}
# subdomains redirect
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name *.halconsole.mywebsite.com;
# SSL
ssl_certificate /keys/hal_fullchain.pem;
ssl_certificate_key /keys/hal_privkey.pem;
ssl_trusted_certificate /keys/hal_chain.pem;
return 301 https://halconsole.mywebsite.com$request_uri;
}
proxy.conf
proxy_http_version 1.1;
proxy_cache_bypass $http_upgrade;
# Proxy headers
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Forwarded $proxy_add_forwarded;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-By $server_addr;
# Proxy timeouts
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
The easiest way to enable https console is by using the console itself
generate a java JKS keystore using either the command line keytool or a GUI program
I like GUIs, so I used Key Store Explorer https://github.com/kaikramer/keystore-explorer
copy keystore file on the halServer server where it has read access (no need to keep it secret) i placed mine inside wildfly data dir in a "keystore" directory.
# your file paths might differ, don't copy paste
cp /home/someUser/sftp_uploads/managementKS /opt/wildfly/standalone/data/keystore/managementKS
set permissions
# your file paths might differ, don't copy paste
chown --recursive -H wildfly:wildfly /opt/wildfly/standalone/data/keystore
(use vpn) login to cleartext console http://halServer:9990/
add keystore : navigate :
configuration -> subsystems -> security (elytron) -> other settings (click view button)
stores -> keystore -> add
...
Name = managementKS
Type = JKS
Path = keystore/managementKS
Relative to = jboss.server.data.dir
Credential Reference Clear Text = keystore-password click Add
result in standalone.xml
<key-store name="managementKS">
<credential-reference clear-text="keystore-password"/>
<implementation type="JKS"/>
<file path="keystore/managementKS" relative-to="jboss.server.data.dir"/>
</key-store>
add key manager : navigate :
ssl -> key manager -> add
...
Name = managementKM
Credential Reference Clear Text = keystore-password
Key Store = managementKS
result in standalone.xml
<key-manager name="managementKM" key-store="managementKS">
<credential-reference clear-text="keystore-password"/>
</key-manager>
add ssl context : navigate :
ssl -> server ssl context -> add
...
Name = managementSSC
Key Manager = managementKM
...
Edit added : Protocols = TLSv1.2
save
result in standalone.xml
<server-ssl-contexts>
<server-ssl-context name="managementSSC" protocols="TLSv1.2" key-manager="managementKM"/>
</server-ssl-contexts>
go back
runtime -> server (click view button)
http management interface (edit)
set secure socket binding = management-https
set ssl context = managementSSC
save
restart wildfly
systemctl restart wildfly

Nginx reverse proxy to Unicorn - nginx configuration is incorrect?

I followed this tutorial to configure my rails app to run Unicorn, and be reversed proxy by Nginx on my AWS ubuntu instance.
I am able to access the nginx, "Welcome to nginx!" default page, running on the site from the outside(security is configured properly). But it is not serving the rails app.
My unicorn is running under /home/ubuntu/appname/shared/sockets/unicorn.sock=
I ran sudo service unicorn restart just in case.
when i run sudo service --status-all it returns:
[ ? ] unicorn_gpei-tk
As I mentioned nginx is definitely running, here is the config located in /etc/nginx/sites-available/default:
upstream app {
# Path to Unicorn SOCK file, as defined previously
server unix:/home/ubuntu/appname/shared/sockets/unicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name localhost;
root /home/ubuntu/appname/public;
try_files $uri/index.html $uri #app;
location #app {
proxy_pass http://app;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
}
error_page 500 502 503 504 /500.html;
client_max_body_size 4G;
keepalive_timeout 10;
}
symlink in /etc/nginx/sites-enabled is default -> /etc/nginx/sites-available/default -- I guess second question would be: is the sites-enabled necessary? The tutorial does not mention/require it, I came across this after the 1000 other resources I am attempting to decipher this issue.
Other info: if I run Unicorn as a user I am able to connect to it on port 8080, so Unicorn does run on it's own as well and serve the site.
And I've also restarted nginx service a few times as well, still does not update the config, probably because I have it wrong somewhere.
Really am not seeing what I am missing, any clues/ideas? thanks.
Edit:
I went into /var/log/nginx/error.log (there's an error.log.1 but it doesnt seem to be updating) and this is showing up:
2016/07/28 16:21:19 [error] 11763#11763: *2 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: XX.XXX.XXX.XX, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "my_public_address.com", referrer: "http://my_public_address.com/"
and /var/log/nginx/access.log is:
XX.XXX.XXX.XX - - [28/Jul/2016:16:24:58 +0000] "GET /favicon.ico HTTP/1.1" 404 571 "http://my_public_address.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
I found the issue and fixed it.
You need to include the configuration for what you want. in /etc/nginx/nginx.conf comment out the default config because it is pointing to localhost.
add the configuration you want to the here. in my case it was this:
/etc/nginx/sites-available/* i only have one config file in there but just felt like putting *.
restart nginx service.
many newbie hours of headache all gone now.
Check the permission of /home/ubuntu/appname/shared/sockets/unicorn.sock, see if it can be accessed by nginx running user.

Resources