I'm using docker swarm for deploying multiple services, which communicate between them via gRPC. Every service has multiple replicas, and I'm using client side load balancing to talk to them equally. I'm passing the services domain names as an environment variable like this: service-name-1,service-name-2,service-name-3 because if I rely on dockers default load balancer, I would only connect to one replica, as the channel remains open.
The problem I'm facing is that it's not very convenient to copy and paste the service (service-name-4) on my docker compose, in order to create a new replica. It would be better if I could just scale the services when needed with: docker service scale service-name=4. What I wanted to know is if there is a way of differentiating individual replicas in docker swarm.
I'm aware about the "look-aside server" for load balancing gRPC recommments. I just want to know if there's any straight forward solution to this problem.
TL;DR
If the number of clients is much smaller than the number of replicas, consider using a look-aside server. Otherwise, there's no problem leaving the job to docker and just increase the replica count like specified above.
Long description
I ended up using a look-aside server for my specific use case. However, what I described on the post above, wasn't such a big issue after all. Let me explain:
Let's say that there's service A, which talks to service B via gRPC. The issue would only exist if the number of clients of A is smaller than the number of available B replicas. For example: 2 replicas of A and 4 of B. It this were the case, A1 would connect (and keep an open connection) to B1, A2 would connect to B2, leaving B3 and B4 on idle.
On the other hand, if the number of clients A is greater or equal than the number of available B replicas, the services would distribute the load according to the load balancing strategy efficiently. For example 4 A replicas, and 2 B replicas. In this case, B1 would have open connections with A1 and A2; while leaving the rest to B2.
An argument could be made about dropped connections leaving uneven distribution on B services, however, that greatly depends on the load balancing strategy.
Related
I'm having 5 Docker nodes in a cluster [swarm].
Let's say I'll constraint NGINX [it is not about nginx, it is just an example] to be deployed only on Docker node 1.
Can I create that constraint in such a way that if Docker node 1 goes down the constraint to not be available anymore?
Like, having that constraint only when the node is reacheable, when it isn't, automatically remove the constraint?
Yes, you can use the placement-ref to place a spread stratergy to your node.hostname=your.node1.hostname as document here
https://docs.docker.com/engine/reference/commandline/service_create/#specify-service-placement-preferences---placement-pref.
If the nodes in one category (for example, those
with node.labels.datacenter=south) can’t handle their fair share of
tasks due to constraints or resource limitations, the extra tasks will
be assigned to other nodes instead, if possible.
The downside is that when your node 1 is back online, the service won't be update and rebalance until the service has been updated again (manually or the service is down).
Additionally, it's not a good design if your service has to be placed on a special node but it should be designed to be able to work every where so you can balance server load accross all nodes. Regarding, NGINX, it's stateless and you can deploy it to all of your nodes and let the docker routing mess to do the load balancing. If your service is statefull, even that it's re-deploy to a second node, your data will be not available and your total service is interupted too. So my real answer is that your question is possible but not the expectation of how Docker Swarm is designed and may be not good too.
If you have any good reason to stick with your question solution. You can think about a load balancer in front of your NGINX (or others) like another NGINX or HAProxy which will allow you more control to route your requests to a master node and use secondary or more node for backup purpose only and so on. The design will be that you have a stateless Load Balancer deploy in global mode, and your core service is running behind the LB. This design will give you no downtime when your node 1 is down or service is updating or relocating.
I have several Python application that all connect to a Redis server and consume messages using the pubsub mechanism. I have containerized the applications with Docker and I would like to scale each application by replicating the number of container instances. The challenge is that I don’t want each container to act as an independent subscriber to Redis, meaning I would essentially like to load balance the network traffic so that, when a message is published, only one container receives it per service.
Let’s take the simple example of two services, Service A and Service B. Both services need to be subscribed to the same topic so that each is notified upon a message published to that topic. Each service will process the message differently; in other words the same message will trigger two different outcomes, one executed by Service A and one by Service B. Now, I am trying to imagine an architecture in which these services consist of replicated containers, let’s call them workers. Say Service A consists of two workers A1 and A2, and Service B consists of three workers B1, B2, and B3 (maybe it requires more processing power per message than Service A, so it requires more workers for the same message load). So my use case requires that both Service A and Service B need to subscribe to the same topic so that they both receive updates as they come in, but I only want one worker to handle the message per service. Imagine that a message comes in and worker A1 handles it for Service A while B3 handles it for Service B.
Overall this feels like it should be pretty straightforward, I essentially have multiple applications, each of which needs to scale horizontally and should handle network traffic as if they were sitting behind a load balancer.
I am intending to deploy these applications with something like Amazon ECS, where each application is essentially a service with task replication and all services connect to a centralized Redis cache acting as a message broker. In a situation like this, from the limited research I’ve done, it would be nice to just put a network load balancer up in front of each service so that published messages would be directed to what looks like a single subscriber, but behind the scenes is a collection of workers acting like they’re pulling off a task queue.
I haven’t had much luck finding examples of this kind of architecture, or for that matter any examples of tasks that use something like Redis in the way I’m imagining. This is an architecture I’ve more or less dreamed up, so I could just be thinking about this all wrong, but at the same time it doesn’t seem like a crazy use case to me. I’m looking for any advice about how this could be accomplished and/or if what I’m talking about just sounds insane and there’s a better way.
I am new to docker-swarm and very much interested in knowing inner workings of how docker-swarm distributes the incoming requests for one single service.
For example, I deployed stack with one single service and 10 replicas across 2 nodes. when I brought up the node 5 containers did show up on node1 and other 5 on node2. Now, I make 10 http requests to the same service from 10 different browser instances, does each container end up with one request each? If it was a round-robin, I would think so. However, I am not observing the same behavior from the stack that I have just deployed.
I brought up the stack with configuration above and made 10 requests. When I did that load was mire distributed than concentrated but only 7 of the containers got 10 requests and 3 were free.
This tells me that it is not even-distribution round robin. If not, what algorithm does docker services api follow to determine which container will serve the next request?
When I searched for inner workings of docker swarm, I ended up with article here : https://docs.docker.com/engine/swarm/ingress/
which is interesting look into ingress and routing mesh but still does not answer my original question.
Anyone?
Our cloud application consists of 3 tightly coupled Docker containers, Nginx, Web and Mongo. Currently we run these containers on a single machine. However as our users are increasing we are looking for a solution to scale. Using Kubernetes we would form a multi container pod. If we are to replicate we need to replicate all 3 containers as a unit. Our cloud application is consumed by mobile app users. Our app can only handle approx 30000 users per Worker node and we intend to place a single pod on a single worker node. Once a mobile device is connected to worker node it must continue to only use that machine ( unique IP address )
We plan on using Kubernetes to manage the containers. Load balancing doesn't work for our use case as a mobile device needs to be tied to a single machine once assigned and each Pod works independently with its own persistent volume. However we need a way of spinning up new Pods on worker nodes if the number of users goes over 30000 and so on.
The idea is we have some sort of custom scheduler which assigns a mobile device a Worker Node ( domain/ IPaddress) depending on the number of users on that node.
Is Kubernetes a good fit for this design and how could we implement a custom pod scale algorithm.
Thanks
Piggy-Backing on the answer of Jonah Benton:
While this is technically possible - your problem is not with Kubernetes it's with your Application! Let me point you the problem:
Our cloud application consists of 3 tightly coupled Docker containers, Nginx, Web, and Mongo.
Here is your first problem: Is you can only deploy these three containers together and not independently - you cannot scale one or the other!
While MongoDB can be scaled to insane loads - if it's bundled with your web server and web application it won't be able to...
So the first step for you is to break up these three components so they can be managed independently of each other. Next:
Currently we run these containers on a single machine.
While not strictly a problem - I have serious doubt's what it would mean to scale your application and what the challenges that come with scalability!
Once a mobile device is connected to worker node it must continue to only use that machine ( unique IP address )
Now, this IS a problem. You're looking to run an application on Kubernetes but I do not think you understand the consequences of doing that: Kubernetes orchestrates your resources. This means it will move pods (by killing and recreating) between nodes (and if necessary to the same node). It does this fully autonomous (which is awesome and gives you a good night sleep) If you're relying on clients sticking to a single nodes IP, you're going to get up in the middle of the night because Kubernetes tried to correct for a node failure and moved your pod which is now gone and your users can't connect anymore. You need to leverage the load-balancing features (services) in Kubernetes. Only they are able to handle the dynamic changes that happen in Kubernetes clusters.
Using Kubernetes we would form a multi container pod.
And we have another winner - No! You're trying to treat Kubernetes as if it were your on-premise infrastructure! If you keep doing so you're going to fail and curse Kubernetes in the process!
Now that I told you some of the things you're thinking wrong - what a person would I be if I did not offer some advice on how to make this work:
In Kubernetes your three applications should not run in one pod! They should run in separate pods:
your webservers work should be done by Ingress and since you're already familiar with nginx, this is probably the ingress you are looking for!
Your web application should be a simple Deployment and be exposed to ingress through a Service
your database should be a separate deployment which you can either do manually through a statefullset or (more advanced) through an operator and also exposed to the web application trough a Service
Feel free to ask if you have any more questions!
Building a custom scheduler and running multiple schedulers at the same time is supported:
https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
That said, to the question of whether kubernetes is a good fit for this design- my answer is: not really.
K8s can be difficult to operate, with the payoff being the level of automation and resiliency that it provides out of the box for whole classes of workloads.
This workload is not one of those. In order to gain any benefit you would have to write a scheduler to handle the edge failure and error cases this application has (what happens when you lose a node for a short period of time...) in a way that makes sense for k8s. And you would have to come up to speed with normal k8s operations.
With the information provided, hard pressed to see why one would use k8s for this workload over just running docker on some VMs and scripting some of the automation.
I am thinking about the best strategy to scale with a cluster of servers. I know there is no hard and fast rules, but I am curious what people think about these scenarios:
cluster of combination app/db servers that are round robin (with failover) balanced using dnsmadeeasy. the db's are synced using replication. Has the advantage that capacity can be augmented easily by adding another server to the cluster, and it is naturally failsafe.
cluster of app servers, again round robin load balanced (with failover) using dnsmadeeasy, all reporting to a big DB server in the back. easy to add app servers, but the single db server creates a single failure point. Could possible add a hot standby with replication.
cluster of app servers (as above) using two databases, one handling reads only, and one handling writes only.
Also, if you have additional ideas, please make suggestions. The data is mostly denormalized and non relational, and the DBs are 50/50 read-write.
Take 2 physical machines and make them Xen servers
A. Xen Base alpha
B. Xen Base beta
In each one do three virtual machines:
"web" server for statics(css,jpg,js...) + load balanced proxy for dynamic request (apache+mod-proxy-balancer,nginx+fair)
"app" server (mongrel,thin,passenger) for dynamic requests
"db" server (mySQL, PostgreSQL...)
Then your distribution of functions can be like this:
A1 owns your public ip and handle requests to A2 and B2
B1 pings A1 and takes over if ping fails
A2 and B2 take dynamic request querying A3 for data
A3 is your dedicated data server
B3 backups A3 second to second and offer readonly access to make copies, backups etc.
B3 pings A3 and become master if A3 becomes unreachable
Hope this can help you some way, or at least give you some ideas.
It really depends on your application.
I've spent a bit of time with various techniques for my company and what we've settled on (for now) is to run a reverse proxy/loadbalancer in front of a cluster of web servers that all point to a single master DB. Ideally, we'd like a solution where the DB is setup in a master/slave config and we can promote the slave to master if there are any issues.
So option 2, but with a slave DB. Also for high availability, two reverse proxies that are DNS round robin would be good. I recommend using a load balancer that has a "fair" algorithm instead of simple round robin; you will get better throughput.
There are even solutions to load balance your DB but those can get somewhat complicated and I would avoid them until you need it.
Rightscale has some good documentation about this sort of stuff available here: http://wiki.rightscale.com/
They provide these types of services for the cloud hosting solutions.
Particularly useful I think are these two entries with the pictures to give you a nice visual representation.
The "simple" setup:
http://wiki.rightscale.com/1._Tutorials/02-AWS/02-Website_Edition/2._Deployment_Setup
The "advanced" setup:
http://wiki.rightscale.com/1._Tutorials/02-AWS/02-Website_Edition/How_do_I_set_up_Autoscaling%3f
I'm only going to comment on the database side:
With a normal RDBMS a 50/50 read/write load for the DB will make replication "expensive" in terms of overhead. For almost all cases having a simple failover solution is less costly than implementing a replicating active/active DB setup. Both in terms of administration/maintenance and licensing cost (if applicable).
Since your data is "mostly denormalized and non relational" you could take a look at HBase which is an OSS implementation of Google Bigtable, a column based key/value database system. HBase again is built on top of Hadoop which is an OSS implementation of Google GFS.
Which solution to go with depends on your expected capacity growth where Hadoop is meant to scale to potentially 1000s of nodes, but should run on a lot less as well.
I've managed active/active replicated DBs, single-write/many-read DBs and simple failover clusters. Going beyond a simple failover cluster opens up a new dimension of potential issues you'll never see in a failover setup.
If you are going for a traditional SQL RDBMS I would suggest a relatively "big iron" server with lots of memory and make it a failover cluster. If your write ratio shrinks you could go with a failover write cluster and a farm of read-only servers.
The answer lies in the details. Is your application CPU or I/O bound? Will you require terabytes of storage or only a few GB?