I am trying to deploy my set of microservices in different nodes. For installing kubeadm and creation of clusters I am following the below documentations.
https://medium.com/#SystemMining/setup-kubenetes-cluster-on-ubuntu-16-04-with-kubeadm-336f4061d929
https://medium.com/#Grigorkh/install-kubernetes-on-ubuntu-1ac2ef522a36
https://www.youtube.com/watch?v=b_fOIELGMDY&t=108s
I need one master with 2 worker machines. I got clear idea about how to create the kubernetes clusters.
My requirements: I have an application which has separate set of microservices. I need to deploy docker images for one set of microservices into one node1.And docker images for other set into node2. And 3 rd set of microservices in node3...etc...This is my planning of deployment. Please correct me if I am going in wrong direction, Since I only started exploration in docker, kubernetes and jenkins. Devop.
My confusions:
According to my requirement region wise deployment by nodes , Is this deployment strategy is possible by Kubernetes ? And is this one of the standard way ?
If I am using Jenkins for implementing CI/CD pipeline , then Do I need to install Jenkins in each Vm? Means master machine and also in machine which resides nodes?
These all are my confusion about this Kubernetes deployment. Please correct me if my thoughts are wrong, since I am only a beginner in DevOp world. How can I clarify my doubts about deployment by using Kubernetes ?
To answer your first question - you basically need to allocate each node for a tenant. If there are compliance/regulatory reasons then you should do it (Though it won't be very efficient). Here is how you can do it:
On the node1 add a taint:
kubectl taint nodes node1.compute.companyname.com reservedfor=tenant1:NoSchedule
What above means is that the node1 will only schedule pods which have a matching toleration and not any other pod. For the microservice which you need to schedule on node1, you will have to add a toleration to the pod YAML file like:
tolerations:
- key: "reservedfor"
operator: "Equal"
value: "tenant1"
effect: "NoSchedule"
The same logic can be extended - so that even if tenant1 needs 4 machines, then all the 4 machines can be tainted with above key value pair and then pods can be tolerated on those nodes. Check out the documentation here and blog with an example here
You can also use the pod/node affinity to achieve above.
Your second question on Jenkins - No, you don't need to install Jenkins on each node, but other than that more details are needed for that question.
Related
I've started learning kubernetes with docker and I've been thinking, what happens if master node dies/fails. I've already read the answers here. But it doesn't answer the remedy for it.
Who is responsible to bring it back? And how to bring it back? Can there be a backup master node to avoid this? If yes, how?
Basically, I'm asking a recommended way to handle master failure in kubernetes setup.
You should have multiple VMs serving as master node to avoid single point of failure.An odd number of 3 or 5 master nodes are recommended for quorum. Have a load balancer in-front of all the VMs serving as master node which can do load balancing and in case one master node dies loadbalancer should remove the VMs IP and make it as unhealthy and not send traffic to it.
Also ETCD cluster is the brain of a kubernetes cluster. So you should have multiple VMs serving as ETCD nodes. Those VMs can be same VMs as of master node or for reduced blast radius you can have separate VMs for ETCD. Again the odd number of VMs should should be 3 or 5. Make sure to take periodic backup of ETCD nodes data so that you can restore the cluster state to pervious state in case of a disaster.
Check the official doc on how to install a HA kubernetes cluster using Kubeadm.
In short, for Kubernetes you should keep master nodes to function properly all the time. There are different methods to make copies of master node, so it is available on failure. As example check this - https://kubernetes.io/docs/tasks/administer-cluster/highly-available-master/
Abhishek, you can run master node in high availability, you should set up the control plane aka master node behind Load balancer as first step. If you have plans to upgrade a single control-plane kubeadm cluster to high availability you should specify the --control-plane-endpoint to set the shared endpoint for all control-plane nodes. Such an endpoint can be either a DNS name or an IP address of a load-balancer.
By default because of security reasons the master node does not host PODS and if you want to enable hosting PODS on master node you can run the following command to do so.
kubectl taint nodes --all node-role.kubernetes.io/master
If you want to manually restore the master make sure you back up the etcd directory /var/lib/etcd. You can restore this on the new master and it should work. Read about high availability kubernetes over here.
We developed an application which consist out of a few go/java services, a mongoDB and a reverse proxy which forwards REST-calls to the specific service. Each service runs in an own docker container.
The whole app is deployable with a single docker-compose file.
We successfully managed to deploy the app in a kubernetes cluster.
Now the "tricky" part: We want to deploy one isolated instance of the app for each customer. (remember one instance consists of approximately 10 containers)
In the past we reached this goal by deploying multiple instances of the docker-compose file.
What is the recommended way in Kubernetes to reach this?
Thank you very much.
Applications can be separated via simple naming and labels or namespaces. Separation could go even further into restricting the nodes an instance may run on or even running separate clusters.
Network policies can be applied on top of deployment to improve network isolation. This would be needed to emulate the docker-compose "network bridge per instance" setup.
"Isolated" can mean a lot of things though as there are various layers where the term can be applied in various ways.
Naming
Many instances of a deployment can run intermingled on a cluster as long as the naming of each kubernetes resource doesn't clash. This includes the labels applied (and sometimes annotations) that are used to select or report on apps so you can uniquely identify a customers resources.
kubectl create -f deployment-customer1.yaml
kubectl create -f deployment-customer2.yaml
This type of naming is easier to manage with a deployment mechanism like helm. Helm "charts" describe a release and are built with the base concept of a variable "release name", so yaml templates can rely on variables. The average helm release would be:
helm install -f customer1-values.yaml customer1-app me/my-app-chart
helm install -f customer2-values.yaml customer2-app me/my-app-chart
Namespaces
A namespace is a logical grouping of resources in a cluster. By itself, a namespace only provides naming isolation but a lot of other k8s resources can then depend on a namespace to apply to:
Authorization/Role based access to k8s
Pod security policy
Resource quotas
A namespace per customer/instance may be useful, for example if you had a "premium" customer that get a bigger slice of resources quotas. It may also make labelling and selecting instances easier, which Network Policy will use.
Environments can be a good fit for a namespace, so a similar deployment can go to the dev/test/prod ns. If you are giving users access to manage or query Kubernetes resources themselves, namespaces make management much easier.
Managing namespaced resources might look like:
kubectl create ns customer1
kubectl create -f deployment.yaml -n customer1
kubectl create ns customer2
kubectl create -f deployment.yaml -n customer2
Again, helm is equally applicable to namespaced deployments.
DNS is probably worth a mention too, containers will look up host names in their own namespace by default. In namespace customer1, looking up the host name service-name will resolve to service-name.customer1.svc.cluster.local
Similarly in the namespace customer2: A lookup for service-name is service-name.customer2.svc.cluster.local
Nodes
Customers could be pinned to a particular nodes (VM or physical) to provide security and/or resource isolation from other customers.
Clusters
Cluster separation can provide full security, resource and network isolation without relying on kubernetes to manage it.
Large apps can often end up using a complete cluster per "grouping". This has a huge overhead of management for each cluster but allow closer to complete independence between instances. Security can be a big driver for this, as you can provide a layer of isolation between clusters outside of the Kubernetes masters.
Network Policy
A network policy lets you restrict network access between Pods/Services via label selectors. Kubernetes will actively manage the firewall rules wherever the Pods are scheduled in the cluster. This would be required to provide similar network isolation to docker-compose creating a network per instance.
The cluster will need to use a network plugin (CNI) that supports network policies, like Calico.
In kuberenetes, you can package all your resources into a helm chart (https://helm.sh/docs/topics/charts/) so that you deploy different instances of each and can manage its lifecycle. You can also pass parameters for each of the instances if required.
Another method is by deploying your application instance using kubernetes operators (https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). This also helps in managing your application components.
I’m trying to figure out and learn the patterns and best practices on moving a bunch of Docker containers I have for an application into Kubernetes. Things like, pod design, services, deployments, etc. For example, I could create a Pod with the single web and application containers in them, but that’d not be a good design.
Searching for things like architecture and design with Kubernetes just seems to yield topics on the product’s architecture or how to implement a Kubernetes cluster, and not the overlay of designing the pods, services, etc.
What does the community generally refer to this application later design in the Kubernetes world, and can anyone refer me to a 101 on this topic please?
Thanks.
Kubernetes is a complex system, and learning step by step is the best way to gain expertise. What I recommend you is documentation about Kubernetes, from where you can learn about each of components.
Another good option is to review 70 best K8S tutorials, which are categorized in many ways.
Designing and running applications with scalability, portability, and robustness in mind can be challenging. Here are great resources about it:
Architecting applications for Kubernetes
Using Kubernetes in production, lessons learned
Kubernetes Design Principles from Google
Well, there's no Kubernetes approach but rather a Cloud Native one: I would suggest you Designing Distributed Systems: patterns and paradigms by Brendan Burns.
It's really good because it provides several scenarios along with pattern approached and related code.
Most of the examples are obviously based on Kubernetes but I think that the implementation is not so important, since you have to understand why and when to use an Ambassador pattern or a FaaS according to the application needs.
The answer to this can be quite complex and that's why it is important that software/platform architects understand K8s well.
Mostly you will find an answer on that which tells you "put each application component in a single pod". And basically that's correct as the main reason for K8s is high availability, fault tolerance of the infrastructure and things like this. This leads us to, if you put every single component to a single pod and make it with a replica higher than 2 its will reach a batter availability.
But you also need to know why you want to go to K8s. At the moment it is a trending topic. But if you don't want to Ops a cluster and actually don't need HA or so, why you don't run on stuff like AWS ECS, Digital Ocean droplets and co?
Best answers you will currently find are all around how to design and cut microservices as each microservice could be represented in a pod. Also, a good starting point is from RedHat Principles of container-based Application Design
or InfoQ.
Un kubernetes cluster is composed of:
A master server called control plane
Nodes: nodes which execute the applications / Containers or pods
By design, a production kubernetes cluster must have at least a master server and 2 nodes according to the kubernetes documentation.
Here is a summary of the components of a kubernetes cluster:
Master = control plane:
kube-api-server: expose the kubernetes api
etcd: key values store for the cluster
kube-scheduler: distributed the pods on the nodes
kube-controller-manager: controller of nodes, pods, cluster components.
Nodes = Servers that run applications
Kubelet: runs on each node, It makes sure that the containers are running in a pod.
kube-proxy: Allows the pods to communicate in the cluster and outside
Runtine container: allows to run the containers / pods
Complementary modules = addons
DNS: DNS server that serves DNS records for Kubernetes services.
Webui: Graphical dashboard for the cluster
Container Resource Monitoring: Records metrics on containers in a central DB, provides UI to browse them
Cluster-level Logging: Records container logs in a central log with a search / browse interface.
I have 1 master kubernetes server and 9 nodes. In that, I want to run backend on 2 nodes and frontend on 2 nodes and DB on 3 nodes.
For all backend, frontend, DB I have ready DockerImage.
How to run an image using kubernetes on only desired(2 or 3).
Please share some ideas to achieve the same.
The Kubernetes scheduler most of the time will do a good job distributing the pods across the cluster. You may want to delegate that responsibility to the scheduler unless you have very specific requirements.
If you want to control this, you can use:
Node selectors
Node Affinity or Anti-Affinity
Directly specify the node name in the deployment spec
From these three, the recommended approach is to use node affinity or anti-affinity due to its flexibility.
Run the front end as a Deployment with desired replica count and let kubernetes manage it for you.
Run Backend as Deployment with desired number of replicas and Kubernetes will figure out how to run it. Use node selectors if you prefer specific nodes.
Run the DB as Deployment OR StatefulSet, Kubernetes will figure out how to run it.
https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/
Use network policies to restrict traffic.
You may use labels and nodeSelector. Here it is:
https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
I am trying to implement the CI/CD pipeline using Kubernetes , Jenkins with my private SVN repository. And I am planning to use Kubernetes cluster having 3 master and 15 worker machine/Node. And Using Jenkins to deploy the microservice developed using spring boot. So When I am deploying using Jenkins , How I can define which microservice need to deploy in which node in kubernetes cluster?. Do I need to specify in Pod ? Or Any other definition ?
How I can define which microservice need to deploy in which node in kubernetes cluster?. Do I need to specify in Pod ? Or Any other definition ?
As said in other answers you don't need to do this, but you can if there is any reason to do so using deprecated nodeSelector or preferable affinities. They are well worth the time to read since you can have some pods relating to specific services/microservices group together or away from each other across available nodes to allow for more flexible and resilient architecture and proper spread out. This way you are helping scheduler deciding where to place what to achieve desired layout. For most basic needs previously mentioned resource allocation can do the trick but for any fine graining you have affinity (and anti affinity) at your disposal. Documentation detailing this is here: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
Kubernetes figures out what nodes should run what pods. You don't have to do that. You have to indicate how much memory and cpu each pod needs, k8s to a first approximation figures out the rest.
That said, what you do have to do is figure out how to partition the full set of workloads you need to run- say, by environment (dev/stage/prod), or by tenant (team X/team Y/team Z or client X/client Y/client Z)- into namespaces, then figure out what workflow makes sense for that partitioning, then configure the CI to satisfy that workflow.