I want to setup a failover for a webservice I've written.
Everywhere i read that docker swarm 1.12 does automatic failover but I think only for failed containers.
How should i configure public DNS for my service?
How does failover work if a host is down?
With normal dns round robin and IPs of the nodes it won't work. every nth request will fail for n servers. The docker route mesh doesn't help if one host is down. Or do i miss something here?
FYI: Currently I'm not using docker swarm but I'm planing to do so.
You got a couple of options:
If you are using a cloud provider like amazonaws, you can use ELB or what ever loadbalancer your cloudprovider provides, They usually support mode TCP and healthchecks. Then you will point your dns to load balancer. This way when ever a node fails, it will be removed from load balancer via healthchecks and will be back when it is healthy again.
You can use your own load balancer. HAproxy and Nginx support TCP mode and you can set healthcheck rules. when ever a node fails, it will be removed from load balancer via healthchecks and will be back when it is healthy again. Make sure HighAvailablity is provided via tools like keepalived so the loadbalancer wont become a point of failure.
Please note this applies to TCP traffic as many loadbalancers dont support UDP traffic.
Related
I have a docker container running haproxy inside, and while it seems to work well, I'm running into a problem where the client IP that reaches the frontend, and shows up on the haproxy logs, is always the same, just with a different port. This IP seems to be the same as the IPV4 IPAM Gateway of the network where the container is running inside (192.168.xx.xx).
The problem with this is that since every request that reaches the proxy has the same client IP, no matter the machine or network where it came from, it's easy for someone with bad intentions to trigger the security restrictions, which bans said IP and no request gets through until the proxy is reset, because every request seems to be coming from the same banned IP.
This is my current haproxy config: (I tried to reduce it to the bare minimum, without the restrictions rules, timeouts, etc, for ease of understanding. I'm testing with this setup and the problem is still present)
global
log stdout format raw local0 info
defaults
mode http
log global
option httplog
option forwardfor
frontend fe
bind :80
default_backend be
backend be
server foo_server $foo_server_IP_and_Port
backend be_abuse_table
stick-table type ip size 1m expire 15m store conn_rate(3s),conn_cur,gpc0,http_req_rate(15s),http_err_rate(20s)
I have tried setting and adding headers, I've also tried to put the container running in the host network, but the problem is that the request does not reach the backend server because it's in a different network, furthermore, I would like to keep the container in the network where it's at, alongside the other containers.
Also, does the backend server configuration influence in any way this problem I'm having? My understanding is that since the problem is already present when reaching the frontend, the backend configuration doesn't matter for this problem.
Any suggestions? This has been driving me crazy for 2 days now. Thank you so much!
Turns out the problem was I was using docker rootless mode. You can either use normal docker or you can use rootless and install the slirpn4netns package and change the port forwarder, following these steps (the section about changing the port forwarder): https://rootlesscontaine.rs/getting-started/docker/#changing-the-port-forwarder
I want a lead to the below problem.
My understanding:
Docker swarm incorporates an ingress and a DNS server that identifies services with their names. It also incorporates inbuilt robust load balancers on every node in the cluster.
We can hit any service running on different nodes which are participating in docker swarm mode using any machine's IP address. If a machine does not host service, the load balancer will route the request to a different machine that hosts that service.
For best practice, we can choose a load balancer container(NGINX/HAProxy) as a reverse proxy to route the requests on the basis of some predefined algorithms(round-robin/Hash/IP Hash/Least Connection/Least bandwidth, etc.).
Problem statement:
I want to make a cluster of two/three different machines where I will be deploying all the technical services which are required. A mini QA environment.
As a service is identified by its name, I can not create another service with the same name. Being a developer, I want to have a service up and running on my localHost which is also part of the docker swarm cluster. Obviously, I can not name it the same. So, let's say I name it as myIP_serviceName. Now the DNS entry which docker swarm has will be based on this name.
I want a mechanism where if I make a call for any service using my IP address as host, the load balancer will look for any service which is registered in DNS as myIP_serviceName, if it finds any service with such a name call should be routed to this service, if it doesn't, the call should follow the regular path. This should hold true for every consecutive request which is part of a round trip journey.
I have not explored Kubernetes yet, Please suggest if Kubernetes can be used here to achieve this goal more elegantly.
Please correct my understanding if I am wrong and do provide valuable suggestions.
HAProxy have written about HAProxy on Docker Swarm: Load Balancing and DNS Service Discovery maybe this will point you in the right direction.
Is it possible to cause docker load balancer which uses round robin to direct requests only one container of global docker service deployed on multiple hosts? If this container goes down, requests will be forwarded to other running containers.
The only way i can think of is using external load balancer like nginx, but requires additional docker service.
You can acheive the same result by using replica mode and only having one replica of the container running. In this case you rely on Docker to ensure that an instance is always available.
Alternatively, the recommended way is to use an external load balancer. Check Use swarm mode routing mesh to see the different usages.
This is probably a simple question but I am wondering how others do this.
I have a Docker Swarm Mode cluster with 3 managers. Lets say they have the IP's
192.168.1.10
192.168.1.20
192.168.1.30
I can access my web container through any of the IP's, through the internal routing mesh etc. But incase the host goes down with the IP I access my web container with I obviously won't get routed to the container any more. So I need some kind of single DNS record or something that points me to an active address in my cluster.
How do you guys do this? Should I add all three IP's as A records to the DNS? Or have a load balancer / reverse proxy infront of the cluster?
I suppose there is more than one solution, would be nice to hear how others solved this :-)
Thanks a lot
Assuming your web server was started something like this:
docker service create --replicas=1 -p 80:80 nginx:alpine
The cluster will attempt to reschedule that one replica should the replica go down for any reason. With only 1 replica, you may experience some downtime between the old task going away, in the new task coming up somewhere else.
Part of the design goal of the routing mesh feature was indeed to simplify load-balancing configuration. Because all members of the cluster listen on port 80 in my example, your load balancer can simply be configured to listen to all three ips. Assuming the load balancer can properly detect the one node going down, the task will get rescheduled somewhere in the cluster, and any of the two remaining nodes will be able to route traffic via the routing mesh.
I'm working on a project to set up a cloud architecture using docker-swarm. I know that with swarm I could deploy replicas of a service which means multiple containers of that image will be running to serve requests.
I also read that docker has an internal load balancer that manages this request distribution.
However, I need help in understanding the following:
Say I have a container that exposes a service as a REST API or say its a web app. And If I have multiple containers (replicas) deployed in the swarm and I have other containers (running some apps) that talk to this HTTP/REST service.
Then, when I write those apps which IP:PORT combination do I use? Is it any of the worker node IP's running these services? Will doing so take care of distributing the load appropriately even amongst other workers/manager running the same service?
Or should I call the manager which in turn takes care of routing appropriately (even if the manager node does not have a container running this specific service)?
Thanks.
when I write those apps which IP:PORT combination do I use? Is it any
of the worker node IP's running these services?
You can use any node that is participating in the swarm, even if there is no replica of the service in question exists on that node.
So you will use Node:HostPort combination. The ingress routing mesh will route the request to an active container.
One Picture Worth Ten Thousand Words
Will doing so take care of distributing the load appropriately even
amongst other workers/manager running the same service?
The ingress controller will do round robin by default.
Now The clients should use dns round robin to access the service on the docker swarm nodes. The classic DNS cache problem will occur. To avoid that we can use external load balancer like HAproxy.
An important additional information to the existing answer
The advantage of using a proxy (HAProxy) in-front of docker swarm is, swarm nodes can reside on a private network that is accessible to the proxy server, but that is not publicly accessible. This will make your cluster secure.
If you are using AWS VPC, you can create a private subnet and place your swarm nodes inside the private subnet and place the proxy server in public subnet which can forward the traffic to the swarm nodes.
When you access the HAProxy load balancer, it forwards requests to nodes in the swarm. The swarm routing mesh routes the request to an active task. If, for any reason the swarm scheduler dispatches tasks to different nodes, you don’t need to reconfigure the load balancer.
For more details please read https://docs.docker.com/engine/swarm/ingress/