Docker-Compose: Service is not reachable after update - docker

Question

version: '3.7'
services:
shinyproxy:
build: /home/administrator/shinyproxy
deploy:
replicas: 1
placement:
constraints:
- node.hostname==testnode
user: root:root
hostname: shinyproxy
image: shinyproxy-example
restart: always
networks:
- sp-example-net
volumes:
- type: bind
source: /var/run/docker.sock
target: /var/run/docker.sock
- type: bind
source: /home/administrator/shinyproxy/application.yml
target: /opt/shinyproxy/application.yml
ports:
- 4000:4000
mariadb:
image: mariadb
networks:
- sp-example-net
volumes:
- type: bind
source: /home/administrator/mariadbdata
target: /var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: root_password
MYSQL_DATABASE: keycloak
MYSQL_USER: keycloak
MYSQL_PASSWORD: xyz
deploy:
placement:
constraints:
- node.hostname==testnode
keycloak:
image: jboss/keycloak
networks:
- sp-example-net
volumes:
- type: bind
source: /home/administrator/compose/nginx/fullchain.pem
target: /etc/x509/https/tls.crt
- type: bind
source: /home/administrator/compose/nginx/privkey.pem
target: /etc/x509/https/tls.key
- ./theme/:/opt/jboss/keycloak/themes/custom/
environment:
- PROXY_ADDRESS_FORWARDING=true
- KEYCLOAK_USER=xyzasd
- KEYCLOAK_PASSWORD=xyz
ports:
- 8443:8443
deploy:
placement:
constraints:
- node.hostname==testnode
restart: "always"
nginx_service:
image: nginx_custom
ports:
- '80:80'
- '443:443'
build: ./nginx/
networks:
- sp-example-net
networks:
sp-example-net:
driver: overlay
external: true
attachable: true
This is my compose file. The keycloak service authenticates shinyproxy users. I use docker-compose up --build -d to get everything running and it workes. Sometimes I have to change small parts of my shinyproxy service and update everything with the same command: Changes get detected and the output looks like this:
compose_keycloak_1 is up-to-date
Recreating compose_shinyproxy_1 ... done
I am running the services in combination with nginx and get the following error:
nginx_service_1 | 2020/06/05 10:02:11 [error] 7#7: *54 connect() failed (113: No route to host) while connecting to upstream, client: 185.130.32.1, server: myserver.com, request: "GET / HTTP/1.1", upstream: "http://10.0.3.181:4000/", host: "myserver.com"
Running docker-compose down and then docker-compose up --build again works, but I do not want to take down all of my services just to update one.
Can anyone tell me why this might happen and how to solve it?
Edit: I am more and more sure this is an nginx issue, not a docker issue. Might that be the case?
My nginx.conf looks like this:
server {
listen 443;
server_name server.com;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_certificate /etc/certs/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/certs/privkey.pem; # managed by Certbot
location / {
proxy_pass http://shinyproxy:4000; ### Übernahme der servicenamen aus Docker-compose
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 600s;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol $scheme;
}
....
edit2: Issue might be related to this:
https://github.com/docker/compose/issues/2003

Answer 1

Instead of doing docker-compose down and then docker-compose up --build for the whole project, you can actually, start that particular service by running docker-compose up -d serviceName. Have a look at the example.
-d stands for daemon/detached mode.
version: '3'
services:
test:
container_name: test
image: 'busybox'
command: 'sleep 5d'
test1:
container_name: test1
image: 'busybox'
command: 'sleep 4d'
$ docker-compose up -d
Creating network "proj_default" with the default driver
Creating test ... done
Creating test1 ... done
$ docker-compose ps
Name Command State Ports
--------------------------------
test sleep 5d Up
test1 sleep 5d Up
$ docker-compose up -d test1
Recreating test1 ... done

Answer 2

The solution was to restart nginx. It seems that like after every restart a container get will a new IP and nginx uses the old IP and will not be able to find it.
restart nginx container when upstream servers is updated
docker exec <nginx_container_id> nginx -s reload