Docker compose of nginx, express, letsencrypt SSL get 502 Bad gateway - docker

I am trying to find a way to publish nginx, express, and letsencrypt's ssl all together using docker-compose. There are many documents about this, so I referenced these and tried to make my own configuration, I succeed to configure nginx + ssl from this https://medium.com/#pentacent/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71
So now I want to put sample nodejs express app into nginx + ssl docker-compose. But I don't know why, I get 502 Bad Gateway from nginx rather than express's initial page.
I am testing this app with my left domain, and on aws ec2 ubuntu16. I think there is no problem about domain dns and security rules settings. All of 80, 443, 3000 ports opened already. and When I tested it without express app it shows well nginx default page.
nginx conf in /etc/nginx/conf.d
server {
listen 80;
server_name example.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name example.com;
server_tokens off;
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
ssl_certificate /etc/letsencrypt/live/sendpi.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/sendpi.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}
docker-compose.yml
version: '3'
services:
app:
container_name: express
restart: always
build: .
ports:
- '3000:3000'
nginx:
container_name: nginx
image: nginx:1.15-alpine
restart: unless-stopped
volumes:
- ./data/nginx:/etc/nginx/conf.d
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
ports:
- "80:80"
- "443:443"
command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
certbot:
image: certbot/certbot
restart: unless-stopped
volumes:
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
Dockerfile of express
FROM node:12.2-slim
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["npm", "start"]
I think SSL works fine, but there are some problems between express app and nginx. How can I fix this?

proxy_pass http://localhost:3000
is proxying the request to the 3000 port on the container that is running nginx. What you instead want is to connect to the 3000 port of the container running express. For that, we need to do two things.
First, we make the express container visible to nginx container at a predefined hostname. We can use links in docker-compose.
nginx:
links:
- "app:expressapp"
Alternatively, since links are now considered a legacy feature, a better way is to use a user defined network. Define a network of your own with
docker network create my-network
and then connect your containers to that network in compose file by adding the following at the top level:
networks:
default:
external:
name: my-network
All the services connected to a user defined network can access each other via name without explicitly setting up links.
Then in the nginx.conf, we proxy to the express container using that hostname:
location / {
proxy_pass http://app:3000
}

Warning: The --link flag is a legacy feature of Docker. It may eventually be removed. Unless you absolutely need to continue using it, we recommend that you use user-defined networks to facilitate communication between two containers instead of using --link.
Define networks in your docker-compose.yml and configure your services with the appropriate network:
version: '3'
services:
app:
restart: always
build: .
networks:
- backend
expose:
- "3000"
nginx:
image: nginx:1.15-alpine
restart: unless-stopped
depends_on:
- app
volumes:
- ./data/nginx:/etc/nginx/conf.d
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
networks:
- frontend
- backend
ports:
- "80:80"
- "443:443"
command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
certbot:
image: certbot/certbot
restart: unless-stopped
volumes:
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
networks:
frontend:
backend:
Note: the app service no longer publish's it's ports to the host it only exposes port 3000 (ref. exposing and publishing ports), it is only available to services connected to the backend network. The nginx service has a foot in both the backend and frontend network to accept incoming traffic from the frontend and proxy the connections to the app in the backend (ref. multi-host networking).
With user-defined networks you can resolve the service name:
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
upstream app {
server app:3000 max_fails=3;
}
server {
listen 80;
server_name example.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name example.com;
server_tokens off;
location / {
proxy_pass http://app;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
ssl_certificate /etc/letsencrypt/live/sendpi.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/sendpi.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}
}
Removing the container_name from your services makes it possible to scale the services: docker-compose up -d --scale nginx=1 app=3 - nginx will load balance the traffic in round-robin to the 3 app containers.

I think maybe a source of confusion here is the way the "localhost" designation behaves among running services in docker-compose. The way docker-compose orchestrates your containers, each of the containers understands itself to be "localhost", so "localhost" does not refer to the host machine (and if I'm not mistaken, there is no way for a container running on the host to access a service exposed on a host port, apart from maybe some security exploits). To demonstrate:
services:
app:
container_name: express
restart: always
build: .
ports:
- '2999:3000' # expose app's port on host's 2999
Rebuild
docker-compose build
docker-compose up
Tell container running the express app to curl against its own running service on port 3000:
$ docker-compose exec app /bin/bash -c "curl http://localhost:3000"
<!DOCTYPE html>
<html>
<head>
<title>Express</title>
<link rel='stylesheet' href='/stylesheets/style.css' />
</head>
<body>
<h1>Express</h1>
<p>Welcome to Express</p>
</body>
</html>
Tell app to try to that same service which we exposed on port 2999 on the host machine:
$ docker-compose exec app /bin/bash -c "curl http://localhost:2999"
curl: (7) Failed to connect to localhost port 2999: Connection refused
We will of course see this same behavior between running containers as well, so in your setup nginx was trying to proxy it's own service running on localhost:3000 (but there wasn't one, as you know).

Tasks
build NodeJS app
add SSL functionality from the box (that can work automatically)
Solution
https://github.com/evertramos/docker-compose-letsencrypt-nginx-proxy-companion
/ {path_to_the_project} /Docker-compose.yml
version: '3.7'
services:
nginx-proxy:
image: jwilder/nginx-proxy:alpine
restart: always
container_name: nginx-proxy
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./certs:/etc/nginx/certs:ro
- ./vhost.d:/etc/nginx/vhost.d
- ./html:/usr/share/nginx/html
- ./conf.d:/etc/nginx/conf.d
ports:
- "443:443"
- "80:80"
labels:
- "com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true"
letsencrypt:
image: jrcs/letsencrypt-nginx-proxy-companion
container_name: letsencrypt
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./certs:/etc/nginx/certs:rw
- ./vhost.d:/etc/nginx/vhost.d:rw
- ./html:/usr/share/nginx/html:rw
environment:
- NGINX_PROXY_CONTAINER=nginx-proxy
api:
container_name: ${APP_NAME}
build:
context: .
dockerfile: Dockerfile
command: npm start --port ${APP_PORT}
expose:
- ${APP_PORT}
# ports:
# - ${APP_PORT}:${APP_PORT}
restart: always
environment:
VIRTUAL_PORT: ${APP_PORT}
VIRTUAL_HOST: ${DOMAIN}
LETSENCRYPT_HOST: ${DOMAIN}
LETSENCRYPT_EMAIL: ${LETSENCRYPT_EMAIL}
NODE_ENV: production
PORT: ${APP_PORT}
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./certs:/etc/nginx/certs:ro
/ {path_to_the_project} /.env
APP_NAME=best_api
APP_PORT=3000
DOMAIN=api.site.com
LETSENCRYPT_EMAIL=myemail#gmail.com
Do not forget to connect DOMAIN to you server before you will run container there.
How it works?
just run docker-compose up --build -d

Related

Deploying a Docker container with Nginx and FastAPI on Google Cloud Platform from SSH-term

I have a some issue when trying to deploy a simple FastAPI application with Nginx on Google Cloud Platform. In my case I should use SSH-terminal to run Docker container with Nginx and FastAPI. My nginx.conf configuration looks like:
access_log /var/log/nginx/app.log;
error_log /var/log/nginx/app.log;
server {
server_name example.com;
listen 80;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name example.com;
ssl_certificate /root/ssl/cert.pem;
ssl_certificate_key /root/ssl/key.pem;
location / {
proxy_pass "http://example.com:8004/";
}
}
And my docker-compose.yml looks like:
version: '3.8'
services:
nginx-proxy:
image: nginx
container_name: nginx-proxy
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./nginx:/etc/nginx/conf.d
- ./ssl/cert1.pem:/root/ssl/cert.pem
- ./ssl/privkey1.pem:/root/ssl/key.pem
- ./ssl/dhparam.pem:/etc/nginx/dhparam/dhparam.pem
web:
environment: [.env]
build: ./project
ports:
- 8004:8000
command: gunicorn main:app -k uvicorn.workers.UvicornWorker -w 2 -b 0.0.0.0:8000
volumes:
- ./project:/usr/src/app
networks:
default:
external:
name: nginx-proxy
Also, I have a Google Cloud VM instance with Firewall HTTP, HTTPS traffic On option, and additionally configured Firewall with rules allowed TCP connections over 443 and 80 ports (Domain name is provided by Google Cloud also, and redirects to VM's external IP address when I put it in my browser address field).
I run my docker-image from SSH-terminal with docker-compose up --build, then I get 502 Bad Gateway Nginx error in my browser (after going to example.com). I would like to know whether it is possible to run the docker image this way from inside SSH-terminal, as well as which steps did I miss to do it the right way?

Docker Nginx on a non-docker host application

I have many docker containers which pass through docker nginx combo (docker-compose.yml outlined below) and they work very well. I want docker nginx to do the same to a non-docker app thats running on localhost:8080, that is I want docker nginx container to run connections to example.com to 127.0.0.1:8080 where 127.0.0.1:8080 is ran by a non-docker app (code-server do be specific but that shouldn't matter)
version: '3'
services:
nginx-proxy:
image: jwilder/nginx-proxy
container_name: nginx-proxy
ports:
- "80:80"
- "443:443"
volumes:
- /apps/proxy/conf:/etc/nginx/conf.d
- /apps/proxy/vhost:/etc/nginx/vhost.d
- /apps/proxy/html:/usr/share/nginx/html
- /apps/proxy/dhparam:/etc/nginx/dhparam
- /apps/proxy/certs:/etc/nginx/certs:ro
- /var/run/docker.sock:/tmp/docker.sock:ro
restart: always
letsencrypt:
image: jrcs/letsencrypt-nginx-proxy-companion
container_name: nginx-proxy-le
depends_on:
- nginx-proxy
volumes:
- /apps/proxy/vhost:/etc/nginx/vhost.d
- /apps/proxy/html:/usr/share/nginx/html
- /apps/proxy/dhparam:/etc/nginx/dhparam:ro
- /apps/proxy/certs:/etc/nginx/certs
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
- NGINX_PROXY_CONTAINER=nginx-proxy
restart: always
networks:
default:
external:
name: nginx-proxy
and its running well on docker containers, the moment i include "nginx-proxy" so that it can detect them, fantastic tool. I cant simply paste something like this into default.conf (conf of nginx )
server {
listen 80;
listen [::]:80;
server_name example.com;
location / {
proxy_pass http://localhost:8080/;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
proxy_set_header Accept-Encoding gzip;
}
}
My guess is that what is "localhost" for me is not "localhost" for nginx given that it is inside a docker (im new to this docker stuff, so I might be talking shit). I saw the issue where they mentioned many approaches, none of them worked for me. In particular I tried running a dummy docker as CWempe suggested
docker run -d \
-e VIRTUAL_HOST=foo.bar.com \
-e VIRTUAL_PORT=8080 \
-e UPSTREAM_NAME=webserver.local \
--rm \
cwempe/docker-dummy:latest
This didnt't work, nginx didnt even detect it which made me think its probably because its not on the nginx-network so i added that (and turned it into docker-compose for convenience)
version: '3.3'
services:
docker-dummy:
environment:
- VIRTUAL_HOST=example.com
- VIRTUAL_PORT=8080
- UPSTREAM_NAME=127.0.0.1
image: 'cwempe/docker-dummy:latest'
networks:
default:
external:
name: nginx-proxy
Then looking at default.conf i get
# mydomain.com
upstream mydomain.com {
## Can be connected with "nginx-proxy" network
# code-server_docker-dummy_1
server 172.25.0.7 down;
}
server {
server_name mydomain.com;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
include /etc/nginx/vhost.d/default;
location / {
proxy_pass http://example.com;
}
}
server {
server_name example.com;
listen 443 ssl http2 ;
access_log /var/log/nginx/access.log vhost;
return 500;
ssl_certificate /etc/nginx/certs/default.crt;
ssl_certificate_key /etc/nginx/certs/default.key;
}
So sure it has seen it but it also believes it is down and doesn't include the VIRTUAL_PORT at all and obviously the 127.25.0.7 IP doesn't make sense to me either. Changing 127.25.0.7 -> 127.0.0.1:8080 does nothing. Any idea how I can remedy this ? Thank you for your input in advance.

Docker-Compose: Service is not reachable after update

version: '3.7'
services:
shinyproxy:
build: /home/administrator/shinyproxy
deploy:
replicas: 1
placement:
constraints:
- node.hostname==testnode
user: root:root
hostname: shinyproxy
image: shinyproxy-example
restart: always
networks:
- sp-example-net
volumes:
- type: bind
source: /var/run/docker.sock
target: /var/run/docker.sock
- type: bind
source: /home/administrator/shinyproxy/application.yml
target: /opt/shinyproxy/application.yml
ports:
- 4000:4000
mariadb:
image: mariadb
networks:
- sp-example-net
volumes:
- type: bind
source: /home/administrator/mariadbdata
target: /var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: root_password
MYSQL_DATABASE: keycloak
MYSQL_USER: keycloak
MYSQL_PASSWORD: xyz
deploy:
placement:
constraints:
- node.hostname==testnode
keycloak:
image: jboss/keycloak
networks:
- sp-example-net
volumes:
- type: bind
source: /home/administrator/compose/nginx/fullchain.pem
target: /etc/x509/https/tls.crt
- type: bind
source: /home/administrator/compose/nginx/privkey.pem
target: /etc/x509/https/tls.key
- ./theme/:/opt/jboss/keycloak/themes/custom/
environment:
- PROXY_ADDRESS_FORWARDING=true
- KEYCLOAK_USER=xyzasd
- KEYCLOAK_PASSWORD=xyz
ports:
- 8443:8443
deploy:
placement:
constraints:
- node.hostname==testnode
restart: "always"
nginx_service:
image: nginx_custom
ports:
- '80:80'
- '443:443'
build: ./nginx/
networks:
- sp-example-net
networks:
sp-example-net:
driver: overlay
external: true
attachable: true
This is my compose file. The keycloak service authenticates shinyproxy users. I use docker-compose up --build -d to get everything running and it workes. Sometimes I have to change small parts of my shinyproxy service and update everything with the same command: Changes get detected and the output looks like this:
compose_keycloak_1 is up-to-date
Recreating compose_shinyproxy_1 ... done
I am running the services in combination with nginx and get the following error:
nginx_service_1 | 2020/06/05 10:02:11 [error] 7#7: *54 connect() failed (113: No route to host) while connecting to upstream, client: 185.130.32.1, server: myserver.com, request: "GET / HTTP/1.1", upstream: "http://10.0.3.181:4000/", host: "myserver.com"
Running docker-compose down and then docker-compose up --build again works, but I do not want to take down all of my services just to update one.
Can anyone tell me why this might happen and how to solve it?
Edit: I am more and more sure this is an nginx issue, not a docker issue. Might that be the case?
My nginx.conf looks like this:
server {
listen 443;
server_name server.com;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_certificate /etc/certs/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/certs/privkey.pem; # managed by Certbot
location / {
proxy_pass http://shinyproxy:4000; ### Übernahme der servicenamen aus Docker-compose
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 600s;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol $scheme;
}
....
edit2: Issue might be related to this:
https://github.com/docker/compose/issues/2003
Instead of doing docker-compose down and then docker-compose up --build for the whole project, you can actually, start that particular service by running docker-compose up -d serviceName. Have a look at the example.
-d stands for daemon/detached mode.
version: '3'
services:
test:
container_name: test
image: 'busybox'
command: 'sleep 5d'
test1:
container_name: test1
image: 'busybox'
command: 'sleep 4d'
$ docker-compose up -d
Creating network "proj_default" with the default driver
Creating test ... done
Creating test1 ... done
$ docker-compose ps
Name Command State Ports
--------------------------------
test sleep 5d Up
test1 sleep 5d Up
$ docker-compose up -d test1
Recreating test1 ... done
The solution was to restart nginx. It seems that like after every restart a container get will a new IP and nginx uses the old IP and will not be able to find it.
restart nginx container when upstream servers is updated
docker exec <nginx_container_id> nginx -s reload

Nginx deployed in docker container doesn't expose nuxtjs deployed in another docker container (502 Bad Gateway)

I'm trying to run nuxtjs application using nginx as proxy server in docker containers. So, I have 2 containers: nginx and nuxt
here is how I'm building nuxt application
FROM node:11.15
ENV APP_ROOT /src
RUN mkdir ${APP_ROOT}
WORKDIR ${APP_ROOT}
ADD . ${APP_ROOT}
RUN npm install
RUN npm run build
ENV host 0.0.0.0
The result seems to be fine
Next is nginx config
server {
listen 80;
server_name dev.iceik.com.ua;
location / {
proxy_pass http://nuxt:3000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
Also I've tried this nginx config
upstream nuxt {
server nuxt:3000;
}
server {
listen 80;
server_name dev.iceik.com.ua;
location / {
proxy_pass http://nuxt;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
And finally my docker-compose file
version: "3"
services:
nuxt:
build: ./app/
container_name: nuxt
restart: always
ports:
- "3000:3000"
command:
"npm run start"
nginx:
image: nginx:1.17
container_name: nginx
ports:
- "80:80"
volumes:
- ./nginx:/etc/nginx/conf.d
depends_on:
- nuxt
I can ping nuxt container from nginx container
Also here are opened ports
So, the expected result is that I can access my nuxt application.
However I'm getting 502 Bad Gateway
Do you have any ideas why nginx doesn't expose my nuxt application?
Thank you for any suggestions!
Nodejs is exposed localhost:3000 instead of 0.0.0.0:3000
Please correct it. It will work
Always good that your containers put into a network if they need to talk each other, other way is to use Host network(only works in linux). Try below docker-compose.yml they should be able to talk each other from the container names.
version: "3"
services:
nuxt:
build: ./app/
container_name: nuxt
restart: always
ports:
- "3000:3000"
command:
"npm run start"
networks:
- my_net
nginx:
image: nginx:1.17
container_name: nginx
ports:
- "80:80"
volumes:
- ./nginx:/etc/nginx/conf.d
depends_on:
- nuxt
networks:
- my_net
networks:
my_net:
driver: "bridge"

Zero Down Time Deployment Node.js/NGINX Docker

I have a React/Node.js application running on a single server using docker-compose. I'm trying to achieve a 0 downtime deployment for my react app. The process right now, does a webpack build (replaces the files in my dist folder) and then docker down and docker up. This whole process takes about 2-3 minutes.
I realized that with docker-compose I can scale my container up/down but I'm not sure how to only push my code to 1 of them, rebuild the webpack and npm restart it then kill other containers. I really don't want to use Kubernetes/Swarm or Openshift since it's a bit of overkill. I'm wondering if anyone else has achieved something similar to this.
My docker-compose looks like this:
node:
build:
context: ./env/docker/node
args:
- PROJECT_ROOT=/var/www/app
image: react_app:rapp_node
command: "npm run prod"
expose:
- "3333"
networks:
- react-net
volumes_from:
- volumes_source
tty: false
nginx:
env_file:
- ".env"
build:
context: ./env/docker/nginx
volumes_from:
- volumes_source
volumes:
- ./env/data/logs/nginx/:/var/log/nginx
- ./env/docker/nginx/sites/node.template:/etc/nginx/node.template
networks:
- react-net
- nginx-proxy
environment:
NGINX_HOST: ${NGINX_HOST}
VIRTUAL_HOST: ${NGINX_VIRTUAL_HOST}
LETSENCRYPT_HOST: ${NGINX_VIRTUAL_HOST}
ESC: $$
links:
- node:node
command: /bin/sh -c "envsubst < /etc/nginx/node.template > /etc/nginx/sites-available/node.conf && nginx -g 'daemon off;'"
volumes_source:
image: tianon/true
volumes:
- ./app:/var/www/app
And my nginx is something like this:
server {
server_name www.${NGINX_HOST};
return 301 ${ESC}scheme://${NGINX_HOST}${ESC}request_uri;
}
server {
listen 80;
server_name ${NGINX_HOST};
root /var/www/app;
location / {
proxy_pass http://node:3333;
proxy_http_version 1.1;
proxy_set_header Upgrade ${ESC}http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host ${ESC}host;
proxy_cache_bypass ${ESC}http_upgrade;
}
}
faster way how to refresh container is
docker-compose restart node
because if you do docker-compose down it will bring all services down, remove configured networks.
If you have scaled service you might try to restart it with
docker restart foldername_node_2
and you could have 0 downtime, but the nginx configuration is not sufficient cause it will randomly select node machine, so you will have to implement some backup server

Resources