I'm trying to design a good way to bring up a fleet of docker containers that act as IOT devices, each with a slightly different configuration.
In each container I would haven an app that simulated some hardware device, e.g. temperature sensor, and each one would have a unique "identity". Example sensor1, sensor2, sensor3, etc and some other configurations that could vary.
I would also want to scale up and scale down the number of virtual devices based on the use case being tested.
The ways I can think of doing this would be to either pass unique properties to each container via a shell script, or have each container access a database or some other store where it would retrieve its unique configuration on startup from a pool of available configurations.
This way the same app would run in each container but its configuration would be unique. Then they would start sending data to some endpoint where I process their telemetry payloads as a data stream.
For a Kubernetes solution it would seem the containers would have to connect to some common datastore to get their configuration since it would not be possible to pass a unique set of properties to each container?
Regarding the identity, David already mentioned the StatefulSet.
Like a Deployment , a StatefulSet manages Pods that are based on an
identical container spec. Unlike a Deployment, a StatefulSet maintains
a sticky identity for each of their Pods. These pods are created from
the same spec, but are not interchangeable: each has a persistent
identifier that it maintains across any rescheduling.
Regarding the scaling. If I understand you correctly, you need a ReplicaSet.
A ReplicaSet is defined with fields, including a selector that
specifies how to identify Pods it can acquire, a number of replicas
indicating how many Pods it should be maintaining, and a pod template
specifying the data of new Pods it should create to meet the number of
replicas criteria. A ReplicaSet then fulfills its purpose by creating
and deleting Pods as needed to reach the desired number. When a
ReplicaSet needs to create new Pods, it uses its Pod template.
For the configuration/properties part, ConfigMaps are what you need.
ConfigMaps bind configuration files, command-line arguments,
environment variables, port numbers, and other configuration artifacts
to your Pods' containers and system components at runtime.
Please let me know if that helped.
Related
I am some kind of new to Kubernetes and Docker and I was studying the concept of statelessness and statefulness and I understand that stateless microservices don't store data on the host, whereas stateful microservices require some kind of storage on the host who serves the requests but if it's up to me I will always use a stateful one why should I ever use a stateless pod? what is the advantage of statelessness?
For a typical Kubernetes Pod, it will be managed by a higher-level controller like a Deployment. You might set the Deployment to have replicas: 3 so that if one of them fails the other two can pick up the load. On an update the existing Pods will get deleted and recreated. If there's heavy load, you can set up a HorizontalPodAutoscaler to increase that replica count for you, which will create more pods when needed.
All of this is really straightforward if your containers are stateless, and there are no consequences to kubectl delete pod.
The problem with a stateful pod is, well, the state. Kubernetes gives you some choices on where to store data, but most of them can only be used on one pod at a time; if you have multiple replicas then each generally needs its own local storage, and the application needs to know how to reconcile the multiple copies of it. (Or, if you can set up something like an NFS server, the application needs to know how to handle concurrent writes.) Operationally, you need to know how to back up and restore all of the individual little volumes that are getting created along the way.
A standard approach is to minimize the number of places where state is stored, and use network I/O from stateless applications to put things in places. The state doesn't even need to be in the cluster: if your application is running in AWS, you could have containers that principally store data in RDS hosted relational databases and Amazon's S3 object store but keep nothing locally, and you can then use normal backup and management approaches for those out-of-cluster stores.
I have a container (a machine learning application) which is capable of loading pre-trained ml-models stored in a persistent volume. I can ask the application to load a particular model by giving its name via its REST API.
Now I want to scale up this application so that, I can load whatever the model, in any of the replicas (not in all of them) and should be able to parse data from that model.
I know this can be done by having multiple deployments and multiple services pointing to each of those deployments so that each instance will have a separate node port and I can access the REST API of each instance through these different node ports. By doing this, I can keep a record of which model is loaded on which instance in my own.
Is there any recommended way to accomplish this requirement without having multiple deployments but through replicas? (Like maintaining a single deployment file with replicas and a manual load balancer at the service level)
As you mentioned, the preferred way to achieve this is through multiple deployments and services.
Unless created manually, replicasets are managed by the deployment and you won't be able to have a single deployment with multiple replicasets running different versions.
It will definitely be easier for you to have one service + deployment per version and a single ingress in front to route the traffic based on some piece of information. It could be a header, sub-domain, path, etc.
To generate all your deployments and services, you could have a look at kustomize.
Our cloud application consists of 3 tightly coupled Docker containers, Nginx, Web and Mongo. Currently we run these containers on a single machine. However as our users are increasing we are looking for a solution to scale. Using Kubernetes we would form a multi container pod. If we are to replicate we need to replicate all 3 containers as a unit. Our cloud application is consumed by mobile app users. Our app can only handle approx 30000 users per Worker node and we intend to place a single pod on a single worker node. Once a mobile device is connected to worker node it must continue to only use that machine ( unique IP address )
We plan on using Kubernetes to manage the containers. Load balancing doesn't work for our use case as a mobile device needs to be tied to a single machine once assigned and each Pod works independently with its own persistent volume. However we need a way of spinning up new Pods on worker nodes if the number of users goes over 30000 and so on.
The idea is we have some sort of custom scheduler which assigns a mobile device a Worker Node ( domain/ IPaddress) depending on the number of users on that node.
Is Kubernetes a good fit for this design and how could we implement a custom pod scale algorithm.
Thanks
Piggy-Backing on the answer of Jonah Benton:
While this is technically possible - your problem is not with Kubernetes it's with your Application! Let me point you the problem:
Our cloud application consists of 3 tightly coupled Docker containers, Nginx, Web, and Mongo.
Here is your first problem: Is you can only deploy these three containers together and not independently - you cannot scale one or the other!
While MongoDB can be scaled to insane loads - if it's bundled with your web server and web application it won't be able to...
So the first step for you is to break up these three components so they can be managed independently of each other. Next:
Currently we run these containers on a single machine.
While not strictly a problem - I have serious doubt's what it would mean to scale your application and what the challenges that come with scalability!
Once a mobile device is connected to worker node it must continue to only use that machine ( unique IP address )
Now, this IS a problem. You're looking to run an application on Kubernetes but I do not think you understand the consequences of doing that: Kubernetes orchestrates your resources. This means it will move pods (by killing and recreating) between nodes (and if necessary to the same node). It does this fully autonomous (which is awesome and gives you a good night sleep) If you're relying on clients sticking to a single nodes IP, you're going to get up in the middle of the night because Kubernetes tried to correct for a node failure and moved your pod which is now gone and your users can't connect anymore. You need to leverage the load-balancing features (services) in Kubernetes. Only they are able to handle the dynamic changes that happen in Kubernetes clusters.
Using Kubernetes we would form a multi container pod.
And we have another winner - No! You're trying to treat Kubernetes as if it were your on-premise infrastructure! If you keep doing so you're going to fail and curse Kubernetes in the process!
Now that I told you some of the things you're thinking wrong - what a person would I be if I did not offer some advice on how to make this work:
In Kubernetes your three applications should not run in one pod! They should run in separate pods:
your webservers work should be done by Ingress and since you're already familiar with nginx, this is probably the ingress you are looking for!
Your web application should be a simple Deployment and be exposed to ingress through a Service
your database should be a separate deployment which you can either do manually through a statefullset or (more advanced) through an operator and also exposed to the web application trough a Service
Feel free to ask if you have any more questions!
Building a custom scheduler and running multiple schedulers at the same time is supported:
https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
That said, to the question of whether kubernetes is a good fit for this design- my answer is: not really.
K8s can be difficult to operate, with the payoff being the level of automation and resiliency that it provides out of the box for whole classes of workloads.
This workload is not one of those. In order to gain any benefit you would have to write a scheduler to handle the edge failure and error cases this application has (what happens when you lose a node for a short period of time...) in a way that makes sense for k8s. And you would have to come up to speed with normal k8s operations.
With the information provided, hard pressed to see why one would use k8s for this workload over just running docker on some VMs and scripting some of the automation.
I'm trying to wrap my head around how/if Kubernetes manages multiple Pods in terms of a clustered client model. Based on this documentation Multi-container it sounds as though Kubernetes is only concerned with the health of a pod and the containers within it. This means that a single Kubernetes instance could manage multiple client's pods, which contain containers running that client's applications, microservices etc.
Is this correct?
Please see my diagram for a clearer idea of what I'm asking.
The diagram has the right idea, but not quite the right terminology.
The diagram would be more accurate if the "Pod" label was replaced with "Namespace", and the "Container" label was replaced with "Pod".
A single Kubernetes cluster is intended to be able to support multi-tenancy, where the workloads of individual clients can run with proper security, resource allocation, isolation, and other important tenancy management attributes.
The unit of tenancy, however, is a namespace- a logical layer of abstraction in which are deployed workloads, usually for an individual client- not a pod, and the unit of replication for workload processing is the pod (comprising one or more containers), not an individual container.
I Have the following setup in mind:
Kubernetes on Mesos (based on the kubernetes-mesos project) within a /16 network.
Each pod will have its own IP and I believe this will avail 64 000 pods.
The idea is to provide isolation for each app i.e. Each app gets its own mysql within the same pod - the app accesses mysql on localhost(within the pod).
If an additional service were needed, I'd use kubernetes rolling updates to add the service's container to the pod, the app will be able to access this new service on localhost as well.
Each application needs as much isolation as possible.
Are there any defects to such an implementation?
Do I have to use weave?
There's an option to specify the service-ip-range while running the kubernetes-mesos install.
One hole is how do I scale a service, is this really viable?
Is there a better way to do this? i.e. Offering isolated services
Thanks.
PS//I'm obviously a noobie at this and I'm trying to get the best possible setup running.
A common misconception is that a Pod should manage a vertical, multi-tier stack: for example a web tier + DB tier together.
It's interesting to read the Kubernetes design intent of Pods: they're for collecting 'helper' processes rather than composing a vertical stack.
To answer your questions, I'd recommend:
Define a Pod template for the web tier only. This can be scaled to any size required, using a replication controller (questions #1 and #3).
Define another Pod for MySQL.
Use the Service abstraction to locate these components.
This sort of design will work for small applications, but you're right that it'll be tough to scale up if you suddenly want two have a couple instances of a service hit the same mysql backend.
You may want to look into putting each service into a separate namespace. Then a service's DNS lookups will be scoped to its own namespace by default so that it won't find other services' resources unless it's explicitly looking for them. This would let you put mysql (and any other dependencies) in a separate pod so that the frontend could be scaled independently.