NGINX - Gunicorn - HTTP 502 upstream prematurely closed - docker

I face a very strange behavior into a docker container.
This running was running a while until it crashed, I then mounted the docker container into another infrastructure and since it crashes.
The logs :
[INFO] Booting worker with pid: 315
"GET /oidc/authenticate/ HTTP/1.1" 302 0 "https://build.contoso.com/" "<user agent>"
"GET /xxx/yyy/ HTTP/1.1" 302 0 "-" "<user agent>"
"GET /oidc/callback/?state=state&session_state=session_state HTTP/1.1" 302 0 "-" "<user agent>"
[CRITICAL] WORKER TIMEOUT (pid:45)
[error] 306#306: *54 upstream prematurely closed connection while reading response header from upstream, client: 192.168.3.110, server: app, request: "GET /xxx/yyy/ HTTP/1.1", upstream: "http://unix:/run/gunicorn.sock:/xxx/yyy/", host: "build.contoso.com"
"GET /xxx/yyy/ HTTP/1.1" 502 559 "-" "<user agent>"
[INFO] Worker exiting (pid: 45)
My gunicorn is launched using :
$VENV/bin/gunicorn app.wsgi:application --daemon --bind unix:/run/gunicorn.sock --workers $workers --capture-output --enable-stdio-inheritance
and is up and running :
p# ps aux | grep gunicorn
root 34 0.0 0.0 30016 20832 ? S 16:41 0:01 /app/venv/bin/python /app/venv/bin/gunicorn app.wsgi:application --daemon --bind unix:/run/gunicorn.sock --workers 17 --capture-output --enable-stdio-inheritance
root 37 0.0 0.0 66068 52104 ? S 16:41 0:00 /app/venv/bin/python /app/venv/bin/gunicorn app.wsgi:application --daemon --bind unix:/run/gunicorn.sock --workers 17 --capture-output --enable-stdio-inheritance
[...]
root 413 0.0 0.0 5196 528 pts/0 S+ 21:25 0:00 grep gunicorn
My nginx configuration:
# cat /etc/nginx/sites-enabled/gunicorn
upstream gunicorn {
server unix:/run/gunicorn.sock;
}
server {
listen 80;
server_name app;
location /static/ {
alias /app/static/;
}
location / {
include proxy_params;
proxy_redirect off;
proxy_pass http://gunicorn;
}
}
Still it crashes, any help welcomed.

Related

Multiple Flask app api's via Nginx gateway on the same port 80 gives 502 error

Tying to set up multiple flask API's via Nginx gateway (uwsgi) seems unable to add multiple locations / in the Nginx conf file. The authentication service seems working fine on port 80 but the streaming service always returns 502.
Structure (with docker files)
-authentication - It works fine
-streaming - Gives 502 http://127.0.0.1/devapp/api/v1/home/list
-ngix
Nginx config :
server {
listen 80;
location / {
include uwsgi_params;
uwsgi_pass authentication:8080;
}
location /devapp/ {
include uwsgi_params;
uwsgi_pass streaming:8081;
} }
uwsgi.ini: (for the streaming module)
#Contains uwsgi configurations
[uwsgi]
wsgi-file= run.py
callable = app
socket = :8081
processes = 4
threads = 2
master = true
chomod-socket = 660
vaccum = true
die-on-term = true
module = run
Nginx Log:
docker logs --tail=10 -f nginx
2021/09/13 02:19:51 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/09/13 02:19:51 [notice] 1#1: start worker processes
2021/09/13 02:19:51 [notice] 1#1: start worker process 23
2021/09/13 02:19:51 [notice] 1#1: start worker process 24
2021/09/13 02:19:51 [notice] 1#1: start worker process 25
2021/09/13 02:19:51 [notice] 1#1: start worker process 26
2021/09/13 02:19:51 [notice] 1#1: start worker process 27
2021/09/13 02:19:51 [notice] 1#1: start worker process 28
2021/09/13 02:21:23 [error] 23#23: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.1, server: , request: "GET /devapp/api/v1/home/list HTTP/1.1", upstream: "uwsgi://172.21.0.3:8081", host: "127.0.0.1"
172.21.0.1 - - [13/Sep/2021:02:21:23 +0000] "GET /devapp/api/v1/home/list HTTP/1.1" 502 157 "-" "PostmanRuntime/7.26.8" "-"

nginx via docker - error:1408F10B:SSL routines:ssl3_get_record:wrong version number

I have a collabora server set up via docker. I'd like to put it behind a proxy so that it's accessible only via https:
I've followed documentation here and here.
On the second link I'm using set up option 2 of 3. I was unsure if I should be using option 1.
After running the container, I wanted to check that collabora server is running per the bottom of the first documentation link above. Obtaining the 'OK' message signifies that all is set up correctly:
curl -k https://localhost:9980
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
If I remove the s:
curl -k http://localhost:9980
OK
I have 3 services in the docker-compose, here's the last couple of lines of the logs:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee6bdb68f8e2 collabora/code "/bin/sh -c 'bash st…" 2 hours ago Up 2 hours 0.0.0.0:9980->9980/tcp collabora
1488465463c6 nginx:1.15-alpine "/bin/sh -c 'while :…" 2 hours ago Up 2 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp collabora_nginx_1
4361ed3f6df7 certbot/certbot "/bin/sh -c 'trap ex…" 2 hours ago Up 2 hours 80/tcp, 443/tcp collabora_certbot_1
The collabora server container/service:
docker logs collabora
wsd-00006-00042 2021-03-02 16:39:07.374169 [ websrv_poll ] WRN convert-to: Requesting address is denied: ::ffff:168.119.3.73| wsd/LOOLWSD.cpp:2315
wsd-00006-00042 2021-03-02 16:40:10.725982 [ websrv_poll ] ERR Looks like SSL/TLS traffic on plain http port| wsd/LOOLWSD.cpp:2383
The nginx container/service:
docker logs collabora_nginx_1
2021/03/02 16:49:47 [error] 10#10: *54 open() "/etc/nginx/html/robots.txt" failed (2: No such file or directory), client: 198.20.87.98, server: collabora.mydomain.de, request: "GET /robots.txt HTTP/1.1", host: "135.181.152.187"
198.20.87.98 - - [02/Mar/2021:16:49:47 +0000] "GET /robots.txt HTTP/1.1" 404 154 "-" "-" "-"
2021/03/02 16:49:48 [error] 10#10: *55 open() "/etc/nginx/html/sitemap.xml" failed (2: No such file or directory), client: 198.20.87.98, server: collabora.mydomain.de, request: "GET /sitemap.xml HTTP/1.1", host: "135.181.152.187"
198.20.87.98 - - [02/Mar/2021:16:49:48 +0000] "GET /sitemap.xml HTTP/1.1" 404 154 "-" "-" "-"
2021/03/02 16:49:50 [error] 10#10: *56 open() "/etc/nginx/html/.well-known/security.txt" failed (2: No such file or directory), client: 198.20.87.98, server: collabora.mydomain.de, request: "GET /.well-known/security.txt HTTP/1.1", host: "135.181.152.187"
198.20.87.98 - - [02/Mar/2021:16:49:50 +0000] "GET /.well-known/security.txt HTTP/1.1" 404 154 "-" "-" "-"
2021/03/02 16:49:51 [error] 10#10: *57 open() "/etc/nginx/html/favicon.ico" failed (2: No such file or directory), client: 198.20.87.98, server: collabora.mydomain.de, request: "GET /favicon.ico HTTP/1.1", host: "135.181.152.187"
198.20.87.98 - - [02/Mar/2021:16:49:51 +0000] "GET /favicon.ico HTTP/1.1" 404 154 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:80.0) Gecko/20100101 Firefox/80.0" "-"
2021/03/02 16:49:56 [error] 10#10: *62 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 104.155.101.3, server: collabora.mydomain.de, request: "GET / HTTP/1.1", host: "135.181.152.187"
104.155.101.3 - - [02/Mar/2021:16:49:56 +0000] "GET / HTTP/1.1" 404 154 "-" "python-requests/2.25.1" "-"
The certbot container:
docker logs collabora_certbot_1
logs collabora_certbot_1
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/collabora.mydomain.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not yet due for renewal
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certificates are not due for renewal yet:
/etc/letsencrypt/live/collabora.mydomain.de/fullchain.pem expires on 2021-05-31 (skipped)
No renewals were attempted.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
My docker-compose.yaml:
version: "3.5"
services:
collabora:
image: collabora/code
container_name: collabora
restart: always
depends_on:
- nginx
cap_add:
- MKNOD
environment:
- "extra_params=--o:ssl.enable=false --o:ssl.termination=true"
- domain=nx12345\.blah\.de
- dictionaries=en_US
ports:
- "9980:9980"
volumes:
- ./appdata/collabora:/config
nginx:
image: nginx:1.15-alpine
restart: unless-stopped
volumes:
- ./data/nginx:/etc/nginx/conf.d
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
ports:
- "80:80"
- "443:443"
command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
certbot:
image: certbot/certbot
restart: unless-stopped
volumes:
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
My nginx config app.conf:
server {
listen 80;
server_name mydomain.de;
server_tokens off;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
server_name collabora.mydomain.de;
ssl_certificate /etc/letsencrypt/live/collabora.mydomain.de/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/collabora.mydomain.de/privkey.pem;
# static files
location ^~ /loleaflet {
proxy_pass http://localhost:9980;
proxy_set_header Host $http_host;
}
# WOPI discovery URL
location ^~ /hosting/discovery {
proxy_pass http://localhost:9980;
proxy_set_header Host $http_host;
}
# Capabilities
location ^~ /hosting/capabilities {
proxy_pass http://localhost:9980;
proxy_set_header Host $http_host;
}
# main websocket
location ~ ^/lool/(.*)/ws$ {
proxy_pass http://localhost:9980;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $http_host;
proxy_read_timeout 36000s;
}
# download, presentation and image upload
location ~ ^/lool {
proxy_pass http://localhost:9980;
proxy_set_header Host $http_host;
}
# Admin Console websocket
location ^~ /lool/adminws {
proxy_pass http://localhost:9980;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $http_host;
proxy_read_timeout 36000s;
}
}
How can I correctly place the collabora server behind a proxy using nginx per the above set up using docker?
proxy_pass http://localhost:9980
In your nginx.conf, localhost is referring to the nginx container itself. If you are trying to route to the collabora container, change "localhost" to the name of the collabora container, collabora.
Refer to this post about localhost in docker-compose
Similar docker-compose and collabora config
The issue was networks. A concept I do not yet fully understand but adding this to my docker-compose did the trick:
networks:
collabora:
Then in each service I referenced this same network.

How do I run Webpack DevServer and Flask with Docker?

I can't seem to figure out how to run Flask Backend with Webpack Dev Server. The flask routes have a login decorator. Webpack serves my assets but I can't access any backend routes. Client side I prefix the routes with port :8080/someFlaskRoute but that get's redirected to /login which isn't on port 8080?
route decorator:
def login_required(f):
#wraps(f)
def decorated_function(*args, **kwargs):
expiration = session.get('expires', 0)
now = int(time.time())
if expiration == 0 or expiration < now or 'user_id' not in session:
return redirect(url_for('saml_login', _external=True, _scheme='https', next=request.url))
return f(*args, **kwargs)
return decorated_function
docker-compose.yml:
version: "3"
services:
server-dev:
build:
context: ../..
dockerfile: Dockerfile-server-base
network_mode: host
ports:
- 10005:10005
tty: true
stdin_open: true
command: uwsgi --http-socket 0.0.0.0:10005 --http-websockets --module myapp:app --master --processes 4 --enable-threads --honour-stdin --py-autoreload=3 --buffer-size=65535
client-dev:
image: node:12.13.1-slim
network_mode: host
ports:
- 10001:10001
- 3000:3000
command: yarn dev
Webpack Dev Server:
devServer: {
host: '0.0.0.0',
public: '0.0.0.0:0',
port: 10001,
sockPort: 80,
hotOnly: true,
publicPath: '/',
headers: { 'Access-Control-Allow-Origin': '*' }
}
I see the request come in in the logs:
server-dev_1 | [pid: 119|app: 0|req: 1/1] 127.0.0.1 () {52 vars in 1051 bytes} [Wed Jun 10 10:00:03 2020] GET /dashboard/all_data?is_alignment=false&group_id=&hkrgy8asx787tzwnvf2wll => generated 561 bytes in 5 msecs (HTTP/1.1 302) 3 headers in 272 bytes (1 switches on core 0)
But the response header gives me this:
Connection: keep-alive
Content-Length: 561
Content-Type: text/html; charset=utf-8
Date: Wed, 10 Jun 2020 10:00:03 GMT
Location: /login/?next=http%3A%2F%2Fmysite.com%2Fdashboard%2Fall_data
Generated-By: dev-machine
Server: nginx
nginx config:
upstream flask_upstream {
server localhost:10005;
}
server {
listen 8080;
server_name ~^(?!api).*.mysite.com;
location / {
proxy_pass http://flask_upstream;
proxy_set_header X-Forwarded-Protocol ssl;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
include /etc/nginx/conf.d/proxy.conf;
set $upstream_name flask_upstream;
}
}

Docker: nginx reverse proxy and how to configure multiple services (i.e on 8080 &8081)

I am a bit stuck with configuring multiple services where nginx is the proxy server.
running :
docker -v
Docker version 19.03.8, build afacb8b7f0
docker-compose -v
docker-compose version 1.23.2, build 1110ad01
I want to start with this test, everything in the same docker-compose.yml-file :
link to jwilder/nginx
proxy : nginx-server (jwilder/nginx-proxy:0.7.0 , which is nginx 1.17.6)
container1 : httpd:2.4
container2 : httpd:2.4
updating my /etc/hosts before I start
127.0.0.1 container1.com
127.0.0.1 container2.com
Here is my docker-compose.yml-file (obs -> version 3.7)
version: '3.7'
services:
proxy:
image: jwilder/nginx-proxy:0.7.0
container_name: proxy-test
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./nginx-proxy.conf:/etc/nginx/conf.d/nginx-proxy.conf:ro
container1:
image: httpd:2.4
container_name: container-1
environment:
- VIRTUAL_HOST:container1.com
ports:
- 8080:80
container2:
image: httpd:2.4
container_name: container-2
environment:
- VIRTUAL_HOST:container2.com
ports:
- 8081:80
here is my nginx-proxy.conf:
server {
listen 80;
server_name container1.com;
location / {
proxy_pass http://localhost:8080;
}
}
server {
listen 80;
server_name container2.com;
location / {
proxy_pass http://localhost:8081;
}
}
After this I run the
docker exec container-1 sed -i 's/It works!/Container 1/' /usr/local/apache2/htdocs/index.html AND docker exec container-2 sed -i 's/It works!/Container 2/' /usr/local/apache2/htdocs/index.html
Test 1 : with curl to the port 8080 and port 8081
curl localhost:8080
response -> Container 1
curl localhost:8081
response -> Container 2
Test 2 : with curl to container1.com AND container2.com
curl container1.com
status 502
curl container2.com
status 502
Are the settings in my conf wrong ?
Troubleshooting 1:
docker exec -it proxy-test bash
I can see that the nginx-proxy.conf is in the directory (/etc/nginx/conf.d)
/etc/nginx/conf.d/default.conf is there as well
Troubleshooting 2: The proxy-log (Connection refused - while connecting to upstream)
proxy-test | nginx.1 | 2020/04/03 10:52:08 [error] 61#61: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 172.29.0.1, server: container1.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "container1.com"
proxy-test | nginx.1 | 2020/04/03 10:52:08 [error] 61#61: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 172.29.0.1, server: container1.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "container1.com"
Found 2 solutions to this.
(1)
the first is to update the nginx-proxy.conf with the name of the containers instead of pointing to http://localhost:8080; and http://localhost:8081; :
new config-file
server {
listen 80;
server_name container1.com;
location / {
proxy_pass http://container-1;
}
}
server {
listen 80;
server_name container2.com;
location / {
proxy_pass http://container-2;
}
}
(2)
Leaving out the nginx-proxy.conf-file , docker-compose.yml will map things correctly.

Can't connect UWSGI container with NGINX container [Docker]

I can'f find my mistake myself, could anyone help me please.
So, I want to run Nginx with https and uWSGI+Flask in different containers for many reasons. I did it, but uwsgi container doesn't get request from nginx container.
My confing:
N.B. - IP address of my server has a 11.11.11.1 for only example propose.
Nginx Dockerfile:
FROM nginx:alpine
RUN apk add --no-cache openssl
RUN mkdir -p /etc/nginx/ssl/ \
&& cd /etc/nginx/ssl/ \
&& openssl req -newkey rsa:2048 -sha256 -nodes \
-keyout cert.key \
-x509 \
-days 9999 \
-out cert.pem \
-subj "/C=US/ST=New York/L=Brooklyn/O=Me/CN=11.11.11.1"
ADD nginx.conf /etc/nginx/nginx.conf
ADD nginx.custom.conf /etc/nginx/conf.d/nginx.custom.conf
EXPOSE 443
EXPOSE 80
uwsgi Dockerfile
FROM alpine:3.7
ADD requirements.txt requirements.txt
# Install uWSGI
RUN apk add --no-cache uwsgi-python3 python3 \
&& export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.6/site-packages:/usr/lib/python3.6/site-packages \
&& pip3 install --no-cache-dir -r requirements.txt
EXPOSE 4000
ADD ./app /app
WORKDIR /app
CMD [ "uwsgi", "--thunder-lock", "--ini", "/app/uwsgi.ini"]
uwsgi.ini:
[uwsgi]
app_base = /app
chmod-socket = 777
socket = 0.0.0.0:4000
chdir = %(app_base)
wsgi-file = uwsgi.py
callable = app
master = true
buffer-size = 32768
processes = 4
max-requests = 1000
harakiri = 20
vauum = true
reload-on-as = 512
die-on-term = true
plugins = python3
uwsgi.py:
from bot.controllers import app
if __name__ == '__main__':
app.run(host='0.0.0.0', port=4000, debug=True, use_reloader=False)
nginx.conf:
upstream flaskapp {
server 0.0.0.0:4000;
}
server {
listen 80;
listen 443 ssl;
server_name 11.11.11.1;
ssl on;
ssl_protocols SSLv3 TLSv1;
ssl_certificate /etc/nginx/ssl/cert.pem;
ssl_certificate_key /etc/nginx/ssl/cert.key;
location / {
include /etc/nginx/uwsgi_params;
uwsgi_pass flaskapp;
}
}
docker-compose.yml:
version: '0.1'
services:
app:
build: .
ports:
- "4000:4000"
links:
- nginx
nginx:
image: nginx_ssl:5.0
ports:
- "443:443"
log:
app_1 | uwsgi socket 0 bound to TCP address 0.0.0.0:4000 fd 3
app_1 | uWSGI running as root, you can use --uid/--gid/--chroot options
app_1 | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
app_1 | Python version: 3.6.3 (default, Nov 21 2017, 14:55:19) [GCC 6.4.0]
app_1 | *** Python threads support is disabled. You can enable it with --enable-threads ***
app_1 | Python main interpreter initialized at 0x55e6e3cc5f40
app_1 | uWSGI running as root, you can use --uid/--gid/--chroot options
app_1 | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
app_1 | your server socket listen backlog is limited to 100 connections
app_1 | your mercy for graceful operations on workers is 60 seconds
app_1 | mapped 507960 bytes (496 KB) for 4 cores
app_1 | *** Operational MODE: preforking ***
app_1 | Set the MIATA_SERVER_IP environment variable
app_1 | Set the MIATA_PUBLIC_CERT environment variable
app_1 | WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x55e6e3cc5f40 pid: 1 (default app)
app_1 | uWSGI running as root, you can use --uid/--gid/--chroot options
app_1 | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
app_1 | *** uWSGI is running in multiple interpreter mode ***
app_1 | spawned uWSGI master process (pid: 1)
app_1 | spawned uWSGI worker 1 (pid: 8, cores: 1)
app_1 | spawned uWSGI worker 2 (pid: 9, cores: 1)
app_1 | spawned uWSGI worker 3 (pid: 10, cores: 1)
app_1 | spawned uWSGI worker 4 (pid: 11, cores: 1)
nginx_1 | 2018/04/12 14:15:33 [error] 7#7: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.17.0.1, server: 11.11.11.1, request: "GET / HTTP/1.1", upstream: "uwsgi://0.0.0.0:4000", host: "77.37.214.6"
nginx_1 | 172.17.0.1 - - [12/Apr/2018:14:15:33 +0000] "GET / HTTP/1.1" 502 174 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.1 Safari/603.1.30" "-"
My question is why NGINX can't connect to the UWSGI? Where I did mistake or what I didn't do?
Thank you in advance!
The nginx container should link to the app container. In the nginx upstream, the uwsgi container should be referenced. This will allow nginx to reference the uwsgi container to proxy the requests.
nginx.conf
upstream flaskapp {
server app:4000;
}
server {
listen 80;
listen 443 ssl;
server_name 11.11.11.1;
ssl on;
ssl_protocols SSLv3 TLSv1;
ssl_certificate /etc/nginx/ssl/cert.pem;
ssl_certificate_key /etc/nginx/ssl/cert.key;
location / {
include /etc/nginx/uwsgi_params;
uwsgi_pass flaskapp;
}
}
docker-compose.yml
version: '2'
services:
app:
build: .
ports:
- "4000:4000"
nginx:
image: nginx_ssl:5.0
ports:
- "443:443"
links:
- app

Resources