I have put together an architecture that at high level is best described below
Five node docker swarm cluster
Have say 5 instances of my dockerized micro service running one copy on each of the swarm nodes
The service offers functionality via REST end points
One such functionality is downloads and they work perfectly, I wrote some code in Scala/Play framerwork, dockerized the service and deployed it.
I also know that since I use swarm , it internally does LB per request for me.
I have some questions on WebSocket and how load balancer does not ruin things during download.
I start a 5GB file download and it works. I am using HTTP stream or chunked I guess it does not matter. Now my question is once my REST end point for download is hit, the TCP connection remains open and since it is open until the server closes the connection, it is due to this that the swarm load balancing does not interfere? In short, each time a client requests a HTTP call, swarm load balances it but once the TCP socket is established as in case of specific download example, the request is served by one node as the connection is not re-stablished during the download process?
If a client opens a web socket, it will hit one of the nodes of swarm where the service is running and the websocket connection since it is open, the same service instance will push the notifications?
If for some reason the websocket dies, a new connection might be established by client but the request might end up on some other service instance and will remain like that until a new connection is again established?
Are above 3 points correct in my understanding? Is there some reading material/blogs I can find more on elaborating this?
Maybe using nginx like proxy LB, ip_hash mode
Specifies that a group should use a load balancing method where requests are distributed between servers based on client IP addresses. The first three octets of the client IPv4 address, or the entire IPv6 address, are used as a hashing key. The method ensures that requests from the same client will always be passed to the same server except when this server is unavailable. In the latter case client requests will be passed to another server. Most probably, it will always be the same server as well.
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash
Related
I want to display a maintenance page on an application running under Kubernetes whilst a deployment is in progress, in this “maintenance” window, I backup the database and then apply schema changes and then deploy the new version.
I thought maybe what I could do is change the service selector so that it would point to a nginx container serving up a simple maintenance page whilst the deployment progressed. Once the deployment had succeeded, I would switch back the selector to point to the pods that do the actual work.
My problem with this is approach is that unless I close and reopen the browser that is currently looking at the site then I never see the maintenance page; I’m guessing the browser is keeping a connection open. The public service address doesn’t change throughout this process.
I’m testing this locally on a Docker Kubernetes installation using a type of NodePort .
Any ideas on how to get it working or am I flogging a dead horse with this approach?
Regards
Lee
This happens due to a combination of how browsers and k8s services work.
Browsers cache TCP connections to servers: when requesting a page they will leave the TCP connection open, and if the user later requests more pages from the same domain, the browser will reuse the already-open TCP connection to save time.
The k8s service load balancing operates at the TCP layer. When a new TCP connection is received, it will be assigned to a pod from the Service, and it will keep talking to that pod for the entire TCP connection's lifetime.
So, the issue is your browser is keeping TCP connections open to your old pods, even if you modify the service.
How can we fix this?
Non-solution #1: have the browser not cache connections. As far as I know there's no way to do this, and you don't want it anyway because it'll make your site slower. Also, HTTP caching headers have no impact on this. Browsers always cache TCP connections. A no-cache header will make the browser request the page again, but over the already-open connection.
Non-solution #2: have k8s kill TCP connections when updating the service. This is not possible and is not desirable either because this behavior is what makes "graceful shutdown / request draining" deployment strategies work. See issue.
Solution #1: Use Layer 7 (HTTP) load balancing instead of Layer 4 (TCP) load balancing, such as nginx-ingress. L7 load balancing routes traffic to pods "per HTTP request", instead of "per TCP connection", so you won't have this problem even if browsers keep TCP connections open.
Solution #2: do this from your application instead of from k8s. For example, have an "in-maintenance" DB flag, check it on every request and serve the maintenance page if it's set.
Here is how services in Kubernetes work, they are basically a dummy loadbalancers forwarding requests to pods in a round robin fashion, and they select which pods to forward the requests to based on the labels as you have already figured out.
Now here is how http/tcp work, I open the browser to visit your website www.example.com the tcp takes it's round of syn,ack,syn-ack and I receive the data.
In your case once I open your website I get a reply from a certain pod based on how the service routed me, and that's it, no further communication is made.
Afterwards you remove the functional pods from the service and add the maintenance page, this will be only shown to the new clients connecting to your website.
I.E if I requested your website, and then you changed all the code and restarted NGINX, if I didn't refresh I would not receive new content
First of all make sure the content you are serving is not cached.
Second, make sure to close all open TCP connections when you shut down your pods. The steps should be as follows:
Change service selector to route traffic to maintenance pods
Gracefully shutdown running pods (this includes closing all open TCP connections)
Do maintenance
Change service selector back
As an alternative approach, you can use an ingress controller. That won't have this problem, because it doesn't maintain an open TCP connection to the pods.
I have a Docker container that is a REST API webserver. I want to use this webserver in a Docker Swarm. A couple of the REST API calls are used in an asynchronous pattern. That is, the first call provides data for processing, and is returned a request identifier. The second call uses the request identifier to check on the processing and get the results when processing is done. Since there is no connection between any of the webservers in the Docker Swarm, how can I force the second REST API call back to the Docker instance that was used in the first REST API call? Is there anyway to ensure webserver affinity for these two REST API calls in a Docker Swarm?
With Docker Swarm Mode and Ingress networking, connections are processed with round robin load balancing, and this isn't configurable. If the connection remains open, which is the case for most web browsers, you'll find that requests go back to the same instance.
You can use a reverse proxy in front of your application that is aware of each instance of the service. Docker has this with their HRM tool in the EE offering, and many of the other reverse proxies, like traefik, offer similar sticky session options.
If you can, a better design would be to utilize an external cache for any persistence, e.g. redis. This way you can perform a rolling update of your application without breaking all the sessions.
My question is related to microservices & service discovery of a service which is spread between several hosts.
The setup is as follows:
2 docker hosts (host A & host B)
a Consul server (service discovery)
Let’s say that I have 2 services:
service A
service B
Service B is deployed 10 times (with random ports): 5 times on host A and 5 times on host B.
When service A communicates with service B, for example, it sends a request to serviceB.example.com (hard coded).
In order to get an IP and a port, service A should query the Consul server for an SRV record.
It will get 10 ip:port pairs, for which the client should apply some load-balancing logic.
Is there a simpler way to handle this without me developing a client resolver (+LB) library for that matter ?
Is there anything like that already implemented somewhere ?
Am I doing it all wrong ?
There are a few options:
Load balance on client as you suggest for which you'll either need to find a ready-build service discovery library that works with SRV records and handles load balancing and circuit breaking. Another answer suggested Netflix' ribbon which I have not used and will only be interesting if you are on JVM. Note that if you are building your own, you might find it simpler to just use Consul's HTTP API for discovering services than DNS SRV records. That way you can "watch" for changes too rather than caching the list and letting it get stale.
If you don't want to reinvent that particular wheel, another popular and simple option is to use a HAProxy instance as the load balancer. You can integrate it with consul via consul-template which will automatically watch for new/failed instances of your services and update LB config. HAProxy then provides robust load balancing and health checking with a lot of options (http/tcp, different balancing algorithms, etc). One possible setup is to have a local HAProxy instance on each docker host and a fixed port assigned statically to each logical service (can store it in Consul KV) so you connect to localhost:1234 for service A for example and localhost:2345 for service B. Local instance means you don't pay for extra round trip to loadbalancer instance then to the actual service instance but this might not be an issue for you.
I suggest you to check out Kontena. It will solve this kind of problem out of the box. Every service will have an internal DNS that you can use in communication between services. Kontena has also built-in load balancer that is very easy to use making it very easy to create and scale micro services.
There are also lot's of built-in features that will help developing containerized applications, like private image registry, VPN access to running services, secrets management, stateful services etc.
Kontena is open source project and the code is visible on Github
If you look for a minimal setup, you can wrap the values you receive from Consul via ribbon, Netflix' client based load balancer.
You will find it as a module for Spring Cloud.
I didn't find an up-to-date standalone example, only this link to chrisgray's dropwizard-consul implementation that is using it in a Dropwizard context. But it might serve as a starting point for you.
I am implementing service with ZeroMQ (a good abstraction over networking). Ideally it would be cool if client can determine if server is what expected.
Example:
There are 3 services:
Sum service. (running on port 50001)
Multiply service. (running on port 50002)
Power service. (running on port 50003)
But there is a case, when services running not on default ports, because another services acquired these ports.
For better reliability of program this case should be handled.
How do I check if service is what expected? I mean a standard, best-ever algorithm (or keywords to find it).
My idea about the algorithm (can be much imperfect):
Client send handshake, that contain above all an identificator of service.
Client wait for response for timeout (for ex. 30 seconds)
Server responds properly.
Okay, client start communication.
Exceptions:
incorrect response
time out
The simple way to support multiple clients is to fork a process for each incoming connection and handle the client through that process and the other process then listens on the default port.
However I have heard that not only is the process forked, but the connection itself is redirected from the default port at which the server is listening to a different port on the server side. This improves performance. I am unable to find any reliable reference to this topic. Is this a standard procedure
This is not the case for most protocols on the internet. For example, public web servers will always listen on port 80 for every request. FTP servers on port 21 and so on. Where did you get your information? With more information on the context, someone might be able to help establish the circumstances you're talking about. What's your source?