I need to setup an environment to host multiple websites (hundreds), each with their own domains and external IP addresses. I was thinking about using Kubernetes, Traefik and Letsencrypt for the SSL certs (in AWS). I have a couple of questions:
Is the Traefik\Kubernetes combination suitable?
Will I need a load balancer or can Traefik support multiple ingress IPs;
Opinion question, opinion answer: Yes. It's not as popular as the nginx ingress controller but a good solution nonetheless.
Generally speaking if you want to support thousands of IPs, you will need an additional load balancer that supports it together with a single ingress controller and also if you either expose your Ingress with a service type LoadBalancer or NodePort.
However, you could actually have multiple ingress controllers, one per IP address each with its own ingressClass option. You'll have to check if your cluster can scale up to thousands of ingress controllers.
Related
I want a lead to the below problem.
My understanding:
Docker swarm incorporates an ingress and a DNS server that identifies services with their names. It also incorporates inbuilt robust load balancers on every node in the cluster.
We can hit any service running on different nodes which are participating in docker swarm mode using any machine's IP address. If a machine does not host service, the load balancer will route the request to a different machine that hosts that service.
For best practice, we can choose a load balancer container(NGINX/HAProxy) as a reverse proxy to route the requests on the basis of some predefined algorithms(round-robin/Hash/IP Hash/Least Connection/Least bandwidth, etc.).
Problem statement:
I want to make a cluster of two/three different machines where I will be deploying all the technical services which are required. A mini QA environment.
As a service is identified by its name, I can not create another service with the same name. Being a developer, I want to have a service up and running on my localHost which is also part of the docker swarm cluster. Obviously, I can not name it the same. So, let's say I name it as myIP_serviceName. Now the DNS entry which docker swarm has will be based on this name.
I want a mechanism where if I make a call for any service using my IP address as host, the load balancer will look for any service which is registered in DNS as myIP_serviceName, if it finds any service with such a name call should be routed to this service, if it doesn't, the call should follow the regular path. This should hold true for every consecutive request which is part of a round trip journey.
I have not explored Kubernetes yet, Please suggest if Kubernetes can be used here to achieve this goal more elegantly.
Please correct my understanding if I am wrong and do provide valuable suggestions.
HAProxy have written about HAProxy on Docker Swarm: Load Balancing and DNS Service Discovery maybe this will point you in the right direction.
i have a handful of dockerized microservices, each is listening for http requests on a certain port, and i have these deployments formalized as kubernetes yaml files
however, i can't figure out a working strategy to expose my deployments on the interwebs (in terms of kubernetes services)
each deployment has multiple replicas, and so i assume each deployment should have a matching load balancer service to expose it to the outside
now i can't figure out a strategy to sanely expose these microservices to the internet... here's what i'm thinking:
the whole cluster is exposed on a domain name, and services are subdomains
say the cluster is available at k8s.mydomain.com
each loadbalancer service (which exposes a corresponding microservice) should be accessible by a subdomain
auth-server.k8s.mydomain.com
profile-server.k8s.mydomain.com
questions-board.k8s.mydomain.com
so requests to each subdomain would be load balanced to the replicas of the matching deployment
so how do i actually achieve this setup? is this desirable?
can i expose each load balancer as a subdomain? is this done automatically?
or do i need an ingress controller?
am i barking up the wrong tree?
i'm looking for general advice on how to expose a single app which is a mosaic of microservices
each service is exposed on the same ip/domain, but each gets its own port
perhaps the whole cluster is accessible at k8s.mydomain.com again
can i map each port to a different load balancer?
k8s.mydomain.com:8000 maps to auth-server-loadbalancer
k8s.mydomain.com:8001 maps to profile-server-loadbalancer
is this possible? it seems less robust and less desirable than strategy 1 above
each service is exposed on its own ip/domain?
perhaps each service specifies a static ip, and my domain has A records pointing each subdomain at each of these ip's in a manual way?
how do i know which static ip's to use? in production? in local dev?
maybe i'm conceptualizing this wrong? can a whole kubernetes cluster map to one ip/domain?
what's the simplest way to expose a bunch of microservies in kubernetes? on the other hand, what's the most robust/ideal way to expose microservices in production? do i need a different strategy for local development in minikube? (i was just going to edit /etc/hosts a lot)
thanks for any advice, cheers
I think the first option is by far the best.
Your Ingress might look like this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: name-virtual-host-ingress
spec:
rules:
- host: auth-server.k8s.mydomain.com
http:
paths:
- backend:
serviceName: service1
servicePort: 80
- host: profile-server.k8s.mydomain.com
http:
paths:
- backend:
serviceName: service2
servicePort: 80
- host: questions-board.k8s.mydomain.com
http:
paths:
- backend:
serviceName: service3
servicePort: 80
You can read more about it on Kubernetes docs regarding Ingress and Name based virtual hosting.
You can also use many Ingress Controllers depending where you will end up setting your cluster. You mentioned that you will be testing this on Minikube so I think nginx ingress will be a good choice here.
If you are thinking about managing your traffic you could consider istio .
Here is a nice guide Setting up HTTP(S) Load Balancing with Ingress and another once Configuring Domain Names with Static IP Addresses.
The first method is typically the format that everyone follows ie each microservice gets its own subdomain. You can achieve the same using Kubernetes ingress (for example Nginx Ingress https://kubernetes.github.io/ingress-nginx/)
They need not be in the same domain also ie you can have both *.example.com and *.example2.com
The second method doesn't scale up as you would have a limited number of available ports and running on non-standard ports comes with its own issues.
Use an ingress:
https://kubernetes.io/docs/concepts/services-networking/ingress/#types-of-ingress
With an ingress, you can assign subdomains to different services, or you can serve all the services under different context roots with some url rewriting.
I don't suggest exposing services using different ports. Nonstandard ports have other problems.
I have a docker-swarm set up in digital ocean.
For now there's only 1 master node but more will be added soon.
I use cloudflare dns for this one, and provided the master node's IP as A record IP.
It does work, I am not really sure that this is the correct way though.
Furthermore, I am wondering which IP should I provide to cloudflare when having multiple master nodes?
Any advice on this will be much appreciated, thanks.
You can use any IP you want and Swarm will do routing and load balancing for you. Because of mesh routing you automatically publish your service to all nodes which is part of the swarm network .
As your services will evolve you may need to have more custom rules for the routing. In such case you can introduce a new layer of load balancing done by nginx, haproxy, traefik or any similar tool.
The most simple setup after default swarm routing is to use nginx as load balancer.
I am setting up 2 VPC on GCP, I setup kubeadm on each, let's call them kubemaster and kubenode1. So I ran kubeadm on kubemaster and kubenode1 which :
kubeadm init on kubemaster
kubeadm join on kubenode1
When I was trying to kubectl apply -f (a deployment which contains a pod with simple webapps inside) and kubectl apply -f (a NodePort type of Service which target the deployment port)
After that I simply access the webapps from my browser (on my local machine not on GCP), it just does not work as what I tried on minikube (I setup minikube with same kubectl apply as above too). I dig some search and there are a lot of people saying regarding Ingress and network layer (flannel in kubernetes website example)
My question is what are these Ingress and flannel ? Which one is necessary or both are not necessary at all if I just want my webapp run ? How does each other works against others ? Because from my understanding the layering is as per below :
Traffic -> Services -> Deployments/Pods
Where are these ingress and flannel suits to ? If its not about them both, why my apps does not work as intended (i open all port in GCP setting so its not security issue I suppose), I tried setting up Kubernetes Dashboard-UI, run kubectl proxy and still my browser cannot access both services (my webapp inside the deployment and also Dashboard API), may be I am a little bit lost here.
The flannel and the Ingress are completely different things.
flannel is a CNI or Container Network Interface plugin which task is networking between containers. As coreOS says:
each container is assigned an IP address that can be used to
communicate with other containers on the same host. For communicating
over a network, containers are tied to the IP addresses of the host
machines and must rely on port-mapping to reach the desired container.
This makes it difficult for applications running inside containers to
advertise their external IP and port as that information is not
available to them.
flannel solves the problem by giving each container an IP that can be
used for container-to-container communication. It uses packet
encapsulation to create a virtual overlay network that spans the whole
cluster. More specifically, flannel gives each host an IP subnet (/24
by default) from which the Docker daemon is able to allocate IPs to
the individual containers.
The Kubernetes supports some other CNI plugins: Calico, weave, etc. They vary according to functionality ( e.g. supporting features like NetworkPolicy for restricting resources )
The Ingress is a Kubernetes object which is usually operate at the application layer of the network stack (HTTP) and allow you to expose your Service externally, it also provides a features such as HTTP requests routing, cookie-based session affinity, HTTPS traffic termination and so on. (just like a web server Nginx or Apache)
I want to add few more points along with exiting answers.
After that I simply access the webapps from my browser (on my local
machine not on GCP), it just does not work as what I tried on minikube
Did you open the security rules/firewall rules for the NodePort? On which instance did you open and which instance are you hitting to access your app?
My question is what are these Ingress and flannel?
I recommend you to read offical docs. But anyway, since you asked the question, I would like to tell few words.
Flannel is a overrelay network for containers which the subnet for the container can span across multiple nodes(Which is opposite to native docker networking-host n/w, NAT, etc). Each containers gets it own IP every time it spawn. The flannel is more like control plain for container network which is internal to K8s
Highly recommend you to read How Flannel N/W works
Ingress is smart router for the load balancer(Or simple for now, we can say it exposes the application to out side of K8s). It works at application level. Once you hit the "Ingress" enpoint, it will forward to service(which depends on ingress rules) and then to app pod.
A blog on Ingress - https://medium.com/#cashisclay/kubernetes-ingress-82aa960f658e
I see you were talking about ClusterIP. Generally, the the ClusterIP is the IP for the K8s service which is nothing but a magic of "IP Tables Rules". Kube-Proxy is responsible to write ip table rules in every node once you define "Service". These ip table rules or ClusterIP points to actual pod IP(The IP assigned by flannel daemon). I hope you can understand, how flannel and "Ingress" fit into the picture or work together or responsible for application traffic.(Please correct if I'm wrong..!!)
Can you paste ingress controller yaml content? What are the rules you defined?
Since you are using GCP, why don't you try GKE? I mean it is easy to deploy, besides you can access your application with LoadBalancer instead of depending on Ingress(Anyway, its none of my business :-) )
Said short, flannel or pod-to-pod networking layer in general, is what enables pods to talk to each other in Kubernetes. Ingress Controller on the other hand is what takes Ingress objects and turns them into rules for receiving and forwarding (mostly) HTTP(S) traffic to the backing services, over pod-to-pod network.
As you can see, technically, you need only the first one (pod-to-pod networking) as you can directly expose your service somewhere with NodePort or LoadBalancer service, it is very convenient though to use Ingress if you expose multiple services (pretty much like you do with vhosts on classic web server installations.
We are analyzing the integration of the Kubernetes service in our on premise environment. We have SaaS based services which can be exposed publicly.
We have doubts in setting up the external endpoints for the services. Is there any way to create the external endpoints for the services?
We have tried to setup the ExternalIP parameter in the services with the master node IP address. Not sure this is the correct way. Once we setup the external IP with the master node IP address we are able to access the services.
We have also tried with ingress controllers and also there we can access our services with the IP address of the node where the ingress controllers are running.
For Example :
Public IP : XXX.XX.XX.XX
Ideally, we would map the public IP with the load balancer virtual IP, but we cannot find such a setting in Kubernetes.
Is there any way to address this issue?
My suggestion is to use an Ingress Controller that acts as a proxy for all your services in kubernetes.
Of course your ingress controller has to be somehow exposed to the outside world. My suggestion is to use the hostNetwork setting for the ingress controller pod (this way, the pod will be listening on your host's physical interface, like any other "traditional" service).
A few resources:
Here details on how a pod can be reached from outside your k8s cluster).
Here a nice tutorial on how to setup an ingress controller on k8s.
If you have more than one minion in your cluster, you'll end up having problems with load balancing them. This question can be helpful about that.