We use docker swarm with service discovery for Backend REST application. The services in swarm are configured with endpoint_mode: vip and are running in global mode. Nginx is proxy passed with service discovery aliases. When we update Backend services sometimes nginx throws 502 as service discovery may point to the updating service.
In such case, We wanted to retry the same endpoint again. How can we achieve this?
According to this we added upstream with the host's private IP and used proxy_next_upstream error timeout http_502; but still the problem persists.
nginx.conf
upstream servers {
server 192.168.1.2:443; #private ip of host machine
server 192.168.1.2:443 backup;
}
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
proxy_next_upstream http_502;
location /endpoint1 {
proxy_pass http://docker.service1:8080/endpoint1;
}
location /endpoint2 {
proxy_pass http://docker.service2:8080/endpoint2;
}
location /endpoint3 {
proxy_pass http://docker.service3:8080/endpoint3;
}
}
Here if http://docker.service1:8080/endpoint1 throws 502 we want to hit http://docker.service1:8080/endpoint1 again.
Additional queries:
Is there any way in docker swarm to make it stop pointing to updating service in service discovery till that service is fully up?
Is upstream necessary here since we directly use docker service discovery?
I suggest you add a health check directly at container level (here)
By doing so, docker pings periodically an endpoint you specified, if it's found unhealthy it will 1) stop routing traffic to it 2) kill the container and restart a new one. Therefore you upstream will be resolved to one of the healthy containers. No need to retry.
As for your additional questions, the first one, docker won't start routing til it's healthy. The second, nginx is still useful to distribute traffic according to endpoint url. But personally nginx + swarm vip mode is not a great choice because swarm load balancer is poorly documented, it doesn't support sticky session and you can't have proxy level health check, I would use traefik instead, it has its own load balancer.
Related
recently I'm trying to set up a litte Home Server with a buildin DNS.
The DNS Service is given by lancacheDNS and set up in combination with a Monolithic-Cache (Port 1234) in two docker containers on 192.168.178.11 (Host machine) in my local network.
Since I want to serve a Website(Port 8080) along with some independent APIs (Ports 8081, 8082 or whatsoever) I decided to use Nginx as a reverse Proxy.
The DNS does the following:
getr.me --> 192.168.178.11
The routing works completely fine and getr.me:8080 gives me my website as expected.
Now the tricky part (for me);
Set up Nginx such that:
website.getr.me --> serving website
api1.getr.me --> serving the API1
api2.getr.me --> serving the API2
For that I created a Newtwork "default_dash_nginx".
I edited the nginx to connect to that via:
networks: default: name: default_dash_nginx external: true
Also I connected my website serving container (dashboard) to the network via --network default_dash_nginx.
The serving website gets the IP 172.20.0.4 (received via docker inspect default_dash_nginx) and also the nginx server is connected to the network.
Nginx works and I can edit the admin page.
But unfortunaly event though I edited the proxyHost to the IP + Port of my website receiced from the network, the site is not available. Here the output of my network inspection: https://pastebin.com/jsuPZpqQ
I hope you have another Idea,
thanks in advance,
Maxi
Edit:
The nginx container is actually a NginxReverseProxyManager Container (I don´t know of it was unclear above or simply not important)
The Nginx container can actually Ping the website container ang also get the HTML files from Port 80 from it.
So it seems like the nginx itself isn´t working like it should.
The first answer got no results( I tried to save it as every of the mentioned files
here
Do I have missed something or am I just not smart enough?
nginx config, try and understand
server {
listen 80;
server_name api1.getr.me;
location / {
proxy_pass http://localhost:8081;
}
}
server {
listen 80;
server_name api2.getr.me;
location / {
proxy_pass http://localhost:8082;
}
}
server {
listen 80;
server_name some.getr.me;
location / {
proxy_pass http://localhost:XXXX;
}
}
I try to make a architecture like this:
docker_container_1 nginx: expose to public (network_mode:"bridge", port:80)
docker_container_2 web_serverI: as internal service (network_mode:"host", port:8080)
docker_container_2 web_serverII: as internal service (network_mode:"host", port:8081)
upstream server-i {
server 172.17.0.1:8080;
}
upstream server-ii {
server 172.17.0.1:8081;
}
server {
listen 80;
server_name localhost;
location /service-i {
proxy_pass http://server-i;
}
location /service-ii {
proxy_pass http://server-ii;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
i already set service i and ii network_mode:"host"
than i use "docker ps" to check, only nginx PORTS show
0.0.0.0:80->80/tcp, and other has nothing.
and i also use "docker stats" to check, NET I/O of all container only nginx has value other is zero.
and i found that still can access server_i server-ii from outside use: http://ip:port(8080 and 8081)
how can do ? do i miss something ?
If you specify network_mode: host, it disables all of Docker's networking functionality for that container. You can't remap a container port or prevent it from being visible on the host; you can't use the normal Docker inter-container networking to connect between containers. Your host-networking containers, from a network point of view, are indistinguishable from a non-Docker process running directly on the host.
Host networking is almost never necessary. In some very unusual cases -- if your service has thousands of ports it listens on, if you've measured the Docker NAT overhead to be significant with very-high-volume traffic -- it can get around some limitations of the Docker networking system. In almost all practical cases you should use the default (bridged) networking, and publish specific ports if you need them to be accessible from outside Docker space.
You should:
Disable host networking on all of your containers; use the standard bridge networking instead. (In a non-Compose context, you will need to docker network create a network with default settings.)
Update your Nginx configuration to use the other containers' names as host names; server docker_container_2 web_serverI:8080. See for example Networking in Compose for additional details.
If you don't want the back-end containers to be visible from outside of Docker space, remove their Compose ports: configuration or docker run -p option. Inter-container communication will still work without this.
I have a docker compose file with 2 services: joomla and phpmyadmin.
I need a reverse proxy which behaves like below:
path: / --> joomla service
path: /managedb --> phpmyadmin service
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://joomla;
}
location /managedb {
proxy_pass http://phpmyadmin;
}
}
Everthing works well, however I'd need to add load balancing to balance work between my 3 machines in docker swarm.
They all are VM on the same LAN with static IP 192.168.75.11/12/13.
The Nginx way to add load balancing should be the follow:
upstream joomla_app {
server 192.168.75.11;
server 192.168.75.12;
server 192.168.75.13;
}
upstream phpmyadmin_app {
server 192.168.75.11;
server 192.168.75.12;
server 192.168.75.13;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://joomla_app;
}
location /managedb {
proxy_pass http://phpmyadmin_app;
}
}
However, since the only exposed port is the Ngxinx 80 one because i need it as reverse proxy too, the code above is obviously not working.
So how can I add the load balancing in this scenario?
Thank you in advance!
In docker swarm, you don't need own load balancer, it has a built in one. Simply scale your services and that's all. Swarm name resolver will resolve joomla and phpmyadmin either to a virtual ip that will be a swarm lb for that service or if you configure service to work in dnsrr mode, will use a dns round-robin method when resolving servicename-hostname to container ip.
However, if you want to distribute services across nodes in swarm, that's a different thing. In this case, you can set placement restrictions for each service or set them to be "global" instead replicated - see https://docs.docker.com/engine/swarm/services/#control-service-placement
I have multiple NGNIX-uWSGI based Django Applications deployed using Docker and hosted in EC2 (currently at different ports like 81, 82, ...). Now I wish to add in sub-domains to this such that sub1.domain.com and sub2.domain.com will both work from the same EC2 instance.
I am fine with multiple ports, BUT they dont work via DNS settings.
sub1.domain.com -> 1.2.3.4:81
sub2.domain.com -> 1.2.3.4:82
What I cannot do
Multiple IPs ref: allocation of a new ip for each deployed sub-domain is not possible.
NGINX Proxy ref: This looks like the ideal solution BUT this is not maintained by an org like Docker or NGINX, so I am un-sure of the security and reliability.
What I am considering:
I am considering to write my own NGINX reverse proxy, similar to Apache Multiple Sub Domains With One IP Address BUT then the flow is will via multiple proxies since already there is an NGINX-uWSGI proxy via the Tech Stack
you can use nginx upstream
upstream backend {
server backend1.example.com weight=5;
server backend2.example.com:8080;
server unix:/tmp/backend3;
server backup1.example.com:8080 backup;
server backup2.example.com:8080 backup;
}
server {
server_name sub.test.com www.sub.test.com;
location / {
proxy_pass http://backend;
}
}
I am using nginx to do TCP forwarding based on hostname as discussed here: Nginx TCP forwarding based on hostname
When the upstream containers are taken down for a short period of time (5 or so mins), and then brought back up, nginx doesn't seem to re-resolve them (continue to get 111: connection refused error).
I've attempted to put a resolver in the server block of the nginx config:
server {
listen 443;
resolver x.x.x.x valid=30s
proxy_pass $name;
ssl_preread on;
}
I still get the same behaviour with this in place.
Like BMitch says, you can scale to 0 to ensure DNS remains available to Nginx.
But really, if you're using nginx in Swarm, I recommend using a Swarm-aware proxy solution that dynamically updates nginx/haproxy config's based on Services that have the proper labels. In those cases, when the service was removed, the config would also be removed from the proxy. One's I've used include:
Traefik
Docker Flow Proxy