Websocket on Docker with nginx - docker

I am trying to build a service with two parts: a backend and a frontend. Both are on a different docker container and are communicating through the docker-compose configuration and a nginx container.
For https access, everything's good, but when I am trying to work with websocket, I have an upgrade error, even if the Nginx configuration got this information
Error message : websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header
I am using a fasthttp and fasthttp/websocket for my Golang backend. The code is working on localhost (no nginx configuration), but the combinaison of Docker + nginx seems to break something.
The front-end works with react and is a simple let socket = new WebSocket(wss.mydomain.com/ws/uploadPicture/);
EDIT :
when using ctx.Request.Header.ConnectionUpgrade() just before upgrader.Upgrade, the result is true, but ctx.Response.Header.ConnectionUpgrade() is false
Thank you !
Golang Backend
var upgrader = websocket.FastHTTPUpgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: func(ctx *fasthttp.RequestCtx) bool {
return true
},
}
func InitRouter() func(*fasthttp.RequestCtx) {
router := fasthttprouter.New()
router.GET("/ws/uploadPicture/", doWS)
return router.Handler
}
func doWS(ctx *fasthttp.RequestCtx) {
err := upgrader.Upgrade(ctx, func(conn *websocket.Conn) {
//SHOULD DO STUFF
})
if (err != nil) {
logs.Error(err.Error()) //HIT THIS ERROR
return
}
}
...
fasthttp.ListenAndServe(`:8000`, InitRouter())
Nginx.conf
#############################################################################
## NGINX CONFIGURATION FOR THE WEBAPP
#############################################################################
upstream webapp {
server webapp:3000;
keepalive 4;
}
server {
listen [::]:443 ssl;
listen 443 ssl;
server_name mydomain.com;
ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
proxy_pass http://webapp;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
}
#############################################################################
## NGINX CONFIGURATION FOR THE PROXY
#############################################################################
upstream proxy {
server proxy:8000;
keepalive 4;
}
server {
listen [::]:443 ssl;
listen 443 ssl;
server_name api.mydomain.com;
ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
add_header 'Access-Control-Allow-Origin' "$http_origin";
add_header 'Access-Control-Allow-Credentials' true;
proxy_pass http://proxy;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
}
#############################################################################
## NGINX CONFIGURATION FOR THE PROXY
#############################################################################
upstream proxyws {
server proxy:8000;
keepalive 4;
}
server {
listen [::]:443 ssl;
listen 443 ssl;
server_name wss.mydomain.com;
ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_pass http://proxyws;
}
}
Docker-compose
#############################################################################
## IMAGE FOR THE PROXY
#############################################################################
proxy:
container_name: proxy
build: ./src/Proxy
restart: always
env_file: .env
ports:
- "8000:8000"
#############################################################################
## IMAGE FOR THE WEBAPP
#############################################################################
webapp:
container_name: webapp
build: ./src/Webapp
restart: always
volumes:
- ./src/Webapp:/home/app
- /home/app/.next
- /home/app/node_modules
ports:
- 3000:3000
#############################################################################
## IMAGE THE NGINX & CERTBOT FOR REVERSE PROXY
#############################################################################
nginx:
image: nginx:latest
container_name: nginx
volumes:
- ./data/nginx:/etc/nginx/conf.d
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
ports:
- 80:80
- 443:443

I'm the maintainer of websocket package for fasthttp. The issue is fixed on master branch.
Downdload it with:
go get github.com/fasthttp/websocket#master
And try again, please.
If the issue continue, please open an issue in https://github.com/fasthttp/websocket/issues
Soon, I will release a new version.

Turns out it's not an issue with Nginx/Docker, but with the http package in golang that is supposed to upgrade the connection.
I was using fasthttp with fasthttp/websocket wich is supposed to work, but which does not in a docker container.
I tried to switch to httprouter with the official (not fork as for the fasthttp ones) gorilla/websocket and the connection successfully upgrade.
Gonna check where the issue come from !

I believe upgrade should be a string:
proxy_set_header Connection "upgrade";

Related

NginX in docker 2nd domain return 502 bad gateway

I have 2 domains pointing to one virtual private server ubuntu 21. The first domain(running on port 3000) works as expected, the second domain(running on port 4000 on container and 5000 on host) does not and return nginx 502 bad gateway. I have added port 4000 point to 80 on nginx container:
I have configured like below:
docker-compose.yml:
version: '3'
services:
nginx:
image: nginx:stable-alpine
ports:
- "3000:80" # nginx listen on 80
- "4000:80"
volumes:
- ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
pwm-node:
build: .
image: my_acc/pwm-node
environment:
- PORT=3000
depends_on:
- mongo
mongo:
image: mongo
volumes:
- mongo-db:/data/db
redis:
image: redis
volumes:
mongo-db:
nginx conf:
server {
listen 80;
server_name first_domain.com www.first_domain.com;
# Redirect http to https
location / {
return 301 https://first_domain.com$request_uri;
}
}
server {
listen 80;
server_name second_domain.com www.second_domain.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:4000;
proxy_redirect off;
}
}
server {
listen 443 ssl http2;
server_name first_domain.com www.first_domain.com;
ssl on;
server_tokens off;
ssl_certificate /etc/nginx/ssl/live/first_domain.com/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/live/first_domain.com/privkey.pem;
ssl_dhparam /etc/nginx/dhparam/dhparam-2048.pem;
ssl_buffer_size 8k;
ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://pwm-node:3000;
proxy_redirect off;
}
}
looks like nginx does not accept http://localhost:4000;. I may have to add node-app-4000 to docker-compose.yml as a service and replace localhost with node-app-4000

Nginx & Docker - how to forward to an internal address?

I'm searching for a long time for a solution that can solve my problem. I guess the answer is already given but I'm not searching for the right terms.
I'm using NGINX to forward all requests for port 80 and this works well. Because these ones are forwarded to my own public domain. Now I got a service that I do not want to publish on the internet and just have a different port in my network for it so e.g. 192.168.123.1:10000.
That is what my nginx.conf looks like for exemplary service. I got more server blocks for different services. The important part is the proxy_pass which is set here to be forwarded to the Docker container nextcloudpi. But how can I internally proxy_pass something without a real domain?
server {
listen 80 default_server;
server_name _;
server_name_in_redirect off;
location / {
return 404;
}
}
server {
listen 80;
listen [::]:80;
server_name my-domain.de cloud.my-domain.de www.my-domain.de;
return 301 https://$host$request_uri;
}
# Cloud
server {
server_name cloud.my-domain.de;
#access_log /var/log/nginx/cloud-access.log;
error_log /var/log/nginx/cloud-error.log;
listen 443 ssl http2;
listen [::]:443 ssl http2;
client_max_body_size 100G;
location / {
proxy_send_timeout 1d;
proxy_read_timeout 1d;
proxy_buffering off;
proxy_hide_header Upgrade;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#add_header Front-End-Https on;
proxy_pass https://nextcloudpi;
}
}
I want to use it for invoice ninja for example. How do I set it in Docker then? I normally use expose to let NGINX do everything to do with port 80. But if I want a different internal IP how do I do this? I know how to do it normally in Docker like I tried but that won't work without NGINX:
invoiceninja:
container_name: invoiceninja
image: invoiceninja/invoiceninja:latest
ports:
- 10000:80
restart: always
volumes:
- /storage/appdata/invoiceninja/public:/var/app/public
- /storage/appdata/invoiceninja/storage:/var/app/storage
networks:
- invoiceninja
env_file:
- .secrets/invoiceninja.env
depends_on:
- invoiceninja-db
Basically, how do I forward port 80 of the invoice ninja Docker container to a different port to access it internally like 192.168.123.1:10000.

Docker-compose and nginx lead to connect() failed (111: Connection refused) while connecting to upstream error

I run and application and nginx in docker containers and struggle to get it running:
This is my nginx.conf
server {
server_name data-mastery.com; # managed by Certbot
location / {
proxy_pass http://shinyproxy:4000;
}
listen 443 ssl; # managed by Certbot
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate /etc/nginx/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/nginx/privkey.pem; # managed by Certbot
include /etc/nginx/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/nginx/ssl-dhparams.pem; # managed by Certbot
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 600s;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
server {
if ($host = data-mastery.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80 ;
listen [::]:80 ;
server_name data-mastery.com;
return 404; # managed by Certbot
}
This is my docker-compose file:
version: "3.7"
services:
shinyproxy:
build: ./shinyproxy
container_name: shinyproxy
expose:
- 4000
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "./shinyproxy-logs/server:/log"
- "./shinyproxy-logs/container:/container-logs"
- "./shinyproxy:/opt/shinyproxy"
nginx:
build: ./nginx
container_name: nginx
restart: always
depends_on:
- shinyproxy
ports:
- "80:80"
- "443:443"
Building the images worked correctly, however when I run docker-compose up, nginx spits out the following error nginx | 2021/06/27 07:16:37 [error] 22#22: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 95.157.1.112
I thought this might happen, because nginx should have to wait for the shinyproxy app, so I added a depends-on, but this did not solve my issue.
Does anyknow know where my Problem is?
Ok, as so often, when you ask a question, you find the answer 5 minutes later:
The shinyproxy application has got a yaml file where you also have to set a port which I had to overwrite:
proxy:
port: 4000

How to setup nginx as reverse proxy with LetsEncrypt SSL encryption using Docker

I am trying to setup SSL for my homepage (www.myhomepage.com) using LetsEncrypt on a nginx reverse-proxy. I have an additional host without SSL running for testing proxying to multiple hosts (www.myotherhomepagewithoutssl.com).
The reverse-proxy and two hosts are running in three separate docker containers.
I got both hosts to work without SSL, but the encrypted one does not work, when trying to use SSL. The LetsEncrypt certificates appear to be setup/obtained correctly and are persisted in a docker volume.
I am trying to follow and adapt this tutorial to setup the LetsEncrypt SSL encryption:
http://tom.busby.ninja/letsecnrypt-nginx-reverse-proxy-no-downtime/
When trying to connect to the SSL encrypted host under www.myhomepage.com using Firefox I get this error:
Unable to connect
The other non-encrypted host under www.myotherhomepagewithoutssl.com works. And as I stated above, when I have www.myhomepage.com setup without SSL (in the same way as www.myotherhomepagewithoutssl.com), it is also reachable.
My complete setup is listed below and consists of:
* reverse_proxy_testing.sh: Bash script to clean-up, build and start the containers.
* compose_reverse_proxy.yaml: Docker-Compose file.
* reverse_proxy.docker: Dockerfile for setting up the reverse-proxy with nginx.
* nginx.conf: nginx config-file for the reverse-proxy.
I suspect that my error is located somewhere inside nginx.conf, but I cannot find it.
Any help is much appreciated!
nginx.conf:
worker_processes 1;
events { worker_connections 1024; }
http {
sendfile on;
server {
deny all;
}
upstream myhomepage {
server myhomepage_blog:80;
}
upstream docker-apache {
server apache:80;
}
server {
listen 80;
listen [::]:80;
server_name www.myhomepage.com myhomepage.com;
return 302 https://$server_name$request_uri;
}
server {
listen 443 ssl;
listen [::]:443;
server_name www.myhomepage.com myhomepage.com;
ssl_certificate /etc/letsencrypt/live/myhomepage.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/myhomepage.com/privkey.pem;
location /.well-known {
root /var/www/ssl-proof/myhomepage.com/;
}
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://myhomepage;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 900s;
}
}
server {
listen 80;
server_name www.myotherhomepagewithoutssl.com myotherhomepagewithoutssl.com;
location / {
proxy_pass http://docker-apache;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
}
}
reverse_proxy.docker:
FROM nginx:alpine
COPY nginx.conf /etc/nginx/nginx.conf
RUN mkdir -p /var/www/ssl-proof/myhomepage.com/.well-known
RUN apk update && apk add certbot
compose_reverse_proxy.yaml:
version: '3.3'
services:
reverseproxy:
image: reverseproxy
ports:
- 80:80
restart: always
volumes:
- proxy_letsencrypt_ssl_proof:/var/www/ssl-proof
- proxy_letsencrypte_certificates:/etc/letsencrypt
apache:
depends_on:
- reverseproxy
image: httpd:alpine
restart: always
myhomepage_blog:
image: wordpress
links:
- myhomepage_db:mysql
environment:
- WORDPRESS_DB_PASSWORD=somepassword
- VIRTUAL_HOST=myhomepage.com
volumes:
- myhomepage_code:/code
- myhomepage_html:/var/www/html
restart: always
myhomepage_db:
image: mariadb
environment:
- MYSQL_ROOT_PASSWORD=somepassword
- MYSQL_DATABASE=wordpress
volumes:
- myhomepage_dbdata:/var/lib/mysql
restart: always
volumes:
myhomepage_dbdata:
myhomepage_code:
myhomepage_html:
proxy_letsencrypt_ssl_proof:
proxy_letsencrypte_certificates:
reverse_proxy_testing.sh:
#!/bin/bash
docker rm testreverseproxy_apache_1 testreverseproxy_myhomepage_blog_1 testreverseproxy_myhomepage_db_1 testreverseproxy_reverseproxy_1
docker build -t reverseproxy -f reverse_proxy.docker .
docker-compose -f reverse_proxy_compose.yml up

Nginx as reverse proxy server for Nexus - can't connect in docker environment

I have environment builded upon docker containers (in boot2docker). I have following docker-compose.yml file to quickly setup nginx and nexus servers :
version: '3.2'
services:
nexus:
image: stefanprodan/nexus
container_name: nexus
ports:
- 8081:8081
- 5000:5000
nginx:
image: nginx:latest
container_name: nginx
ports:
- 5043:443
volumes:
- /opt/dm/nginx2/nginx.conf:/etc/nginx/nginx.conf:ro
Nginx has following configuration (nginx.conf)
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
proxy_send_timeout 120;
proxy_read_timeout 300;
proxy_buffering off;
keepalive_timeout 5 5;
tcp_nodelay on;
server {
listen 80;
server_name demo.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
server_name demo.com;
# allow large uploads of files - refer to nginx documentation
client_max_body_size 1024m;
# optimize downloading files larger than 1G - refer to nginx doc before adjusting
#proxy_max_temp_file_size 2048m
#ssl on;
#ssl_certificate /etc/nginx/ssl.crt;
#ssl_certificate_key /etc/nginx/ssl.key;
location / {
proxy_pass http://nexus:8081/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
}
}
}
Nexus seems to work very well. I call sucessfully curl http://localhost:8081 on docker host machine. This return me html of nexus login site. Now I want to try nginx server. It is configured to listen on 443 port, but SSL is right now disabled (I wanted to test it before diving into SSL configuration). As you can notice, my ngix container maps port 443 to port 5043. Thus, I try to use following curl command : curl -v http://localhost:5043/. Now I expect that my http request is going to be send to nginx and proxied to proxy_pass http://nexus:8081/; nexus. Nexus hostname is visible within docker container network and is accesible from nginx container. Unfortunately in reponse I receive :
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5043 (#0)
> GET / HTTP/1.1
> Host: localhost:5043
> User-Agent: curl/7.49.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
I was checking nginx logs, error, access but these logs are empty. Can somebody help me solving this problem ? It should be just a simple example of proxying requests, but maybe I misunderstand some concept ?
Do you have an upstream directive in your nginx conf (placed within the http directive)?
upstream nexus {
server <Nexus_IP>:<Nexus_Port>;
}
Only then nginx can correctly resolve it. The docker-compose service name nexus is not injected to the nginx container on runtime.
You can try links in docker-compose:
https://docs.docker.com/compose/compose-file/#links
This gives you an alias for the linked container in your /etc/hosts. But you still need an upstream directive. Update: If resolvable, you can as well use the names directly in nginx directives like location.
https://serverfault.com/questions/577370/how-can-i-use-environment-variables-in-nginx-conf
As #arnold's answer you are missing the upstream configuration in your nginx. I saw you are using the stefanprodan nexus image, see his blog for the full configuration. Below you can find mine (remember to open ports 8081 and 5000 of nexus even the entrance point is the 443). Besides you need to include the certificate because docker client requires ssl working:
worker_processes 2;
events {
worker_connections 1024;
}
http {
error_log /var/log/nginx/error.log warn;
access_log /dev/null;
proxy_intercept_errors off;
proxy_send_timeout 120;
proxy_read_timeout 300;
upstream nexus {
server nexus:8081;
}
upstream registry {
server nexus:5000;
}
server {
listen 80;
listen 443 ssl default_server;
server_name <yourdomain>;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
ssl_certificate /etc/letsencrypt/live/<yourdomain>/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/<yourdomain>/privkey.pem;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
keepalive_timeout 5 5;
proxy_buffering off;
# allow large uploads
client_max_body_size 1G;
location / {
# redirect to docker registry
if ($http_user_agent ~ docker ) {
proxy_pass http://registry;
}
proxy_pass http://nexus;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
}
}
}
The certificates are generated using letsencrypt or certbot. The rest of the configuration is to have an A+ in ssllabs analysis as it is explained here
Your docker-compose of 5000 port is an dynamic port(Because it hadn't been exposed ) , so you cannot connect the 5000 port because the
ports:
- 8081:8081
- 5000:5000
are not efficient .
you can use like this:
Build a new Dockerfile and expose 5000 port (Mine name is )
FROM sonatype/nexus3:3.16.2
EXPOSE 5000```
Use new image to start the container and publish the port .
version: "3.7"
services:
nexus:
image: 'feibor/nexus:3.16.2-1'
deploy:
placement:
constraints:
- node.hostname == node1
restart_policy:
condition: on-failure
ports:
- 8081:8081/tcp
- 5000:5000/tcp
volumes:
- /mnt/home/opt/nexus/nexus-data:/nexus-data:z

Resources