I have a cluster of database nodes hosted in VMs or Bare Metal and I'd like to create additional database nodes (hosted in Kubernetes Pods) and have them join the existing cluster (ones hosted in VMs or bare metal).
In order to have them join the cluster, each database must be able to resolve the other via distinct IP and port. Within the Kubernetes network environment, there is no issue with this and no issue between the existing VM-hosted DBs. The sticking point is that I can't seem to see a way for the VM-hosted DBs to individually route to each POD-hosted DB. Is there a Kubernetes configuration that will allow me to have each pod/DB individually routable on specific ports while sharing the same NIC for the host running the cluster? Do I need to front each POD with it's own Service?
Here is the sort of configuration I'm trying to achieve with conceptual IP address spaces.
The approach I take personaly for a similar case is to actualy make it possible for nodes in the non-kubernetes environment to be able to talk to the pods them selves. Depending on your network configuration this might be quite easy to achieve.
In my case I simply have 2 additional elements running on VMs that need to access my k8s internals :
- flannel : this actually ties my VMs to the same flannel network as k8s pods operate in
- kube-proxy : translates service ips to pod ips using iptables (in cases where I need to access by service IP)
You could avoid setting this up on VMs or their hosts if you can solve this on a gateway level (ie. have flannel/proxy on your network gate and augment it with some SNAT rules).
Having NodePort/LB service per in-k8s db might work if your DB sticks to the IPs you give (not only use for discovery bootstraping where later on the IPs are replaced with actual IPs of DBs - iirc mongo usually does something like that)
Related
I've got a database running in a private network (say IP 1.2.3.4).
In my own computer, I can do these steps in order to access the database:
Start a Docker container using something like docker run --privileged --sysctl net.ipv4.ip_forward=1 ...
Get the container IP
Add a routing rule, such as ip route add 1.2.3.4/32 via $container_ip
And then I'm able to connect to the database as usual.
I wonder if there's a way to route traffic through a specific pod in Kubernetes for certain IPs in order to achieve the same results. We use GKE, by the way, I don't know if this helps in any way.
PS: I'm aware of the sidecar pattern, but I don't think this would be ideal for our use case, as our jobs are short-lived tasks, and we are not able to run multiple "gateway" containers at the same time.
I wonder if there's a way to route traffic through a specific pod in Kubernetes for certain IPs in order to achieve the same results. We use GKE, by the way, I don't know if this helps in any way.
You can start a GKE in a fully private network like this, then you run application that needs to be fully private in this cluster. Access to this cluster is only possible when explicitly granted; just like those commands you used in your question, but of course now you will use the cloud platform (eg. service control, bastion etc etc), there is no need to "route traffic through a specific pod in Kubernetes for certain IPs". But if you have to run everything in a cluster, then likely a fully private cluster will not work for you, in this case you can use network policy to control access to your database pod.
GKE doesn't support the use case you mentionned #gabriel Milan.
What's your requirement ? Do you need to know which IP the pod will use to reach the database so you can open a firewall for it ?
Replying here as the comments have limited character count
Unfortunately GKE doesn't support that use case.
However You have couple of options:
Option#1: Create a dedicated nodepool with couple of nodes, force the pods to be scheduled on these nodes using taints and tolerations [1]. Use the IP addresses of these nodes on your firewall
Option#2: Install a Service Mesh like Istio, Use the Egress gateway[2] to route traffic toward your onPrem system and force the gateways to be deployed on a specific set of nodes so you have a know IP address. This quite complicated as a solution
[1] https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
[2] https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway/
i would suggest using or creating the NAT gateway instead of using the container as a gateway option.
Using container or Istio is a good idea however it has its own limitations hard to implement, management and resources usage of that gateway containers.
Ultimately you want Single IP for your K8s cluster, instead request going out instead of Node's IP on which POD is scheduled.
Here terraform of GKE NAT gateway which you can use it.
https://registry.terraform.io/modules/GoogleCloudPlatform/nat-gateway/google/latest/examples/gke-nat-gateway
NAT gateway will forward all PODs traffic from a single VM and you can use that IP in the database to whitelist also.
After implementation, there will be single Egress point in your cluster.
GitHub Repo link - Click to deploy available GCP magic ;)
Not sure if this is a silly question. When the same app/service running in multiple containers, how do they report themselves to zookeeper/etcd and identify themselves? So that load balancers know the different instances and know who to talk to, where to probe and dispatch, etc..? Or the service instances would use some id from the container in their identification?
Thanks in advance
To begin with, let me explain in a few sentences how it works:
The basic building block starts with the Pod, which is just a resource that can be created and destroyed on demand. Because a Pod can be moved or rescheduled to another Node, any internal IPs that this Pod is assigned can change over time.
If we were to connect to this Pod to access our application, it would not work on the next re-deployment. To make a Pod reachable to external networks or clusters without relying on any internal IPs, we need another layer of abstraction. K8s offers that abstraction with what we call a Service Deployment.
This way, you can create a website that will be identified, for example, by a load balancer.
Services provide network connectivity to Pods that work uniformly across clusters. Service discovery is the actual process of figuring out how to connect to a service.
You can also find some information about Service in the official documentation:
An abstract way to expose an application running on a set of Pods as a network service.
With Kubernetes you don't need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them.
Kubernetes supports 2 primary modes of finding a Service - environment variables and DNS. You can read more about this topic here and here.
Currently I am working on a project where we have a single trusted master server, and multiple untrusted (physically in an unsecured location) hosts (which are all replicas of each other in different physical locations).
We are using Ansible to automate the setup and configuration management however I am very unimpressed in how big of a gap we have in our development and testing environments, and production environment, as well as the general complexity we have in configuration of the network as well as containers themselves.
I'm curious if Kubernetes would be a good option for orchestrating this? Basically, multiple unique copies of the same pod(s) on all untrusted hosts must be kept running, and communication should be restricted between the hosts, and only allowed between specific containers in the same host and specific containers between the hosts and the main server.
There's a little bit of a lack of info here. I'm going to make the following assumptions:
K8s nodes are untrusted
K8s masters are trusted
K8s nodes cannot communicate with each other
Containers on the same host can communicate with each other
Kubernetes operates on the model that:
all containers can communicate with all other containers without NAT
all nodes can communicate with all containers (and vice-versa) without NAT
the IP that a container sees itself as is the same IP that others see it as
Bearing this in mind, you're going to have some difficulty here doing what you want.
If you can change your physical network requirements, and ensure that all nodes can communicate with each other, you might be able to use Calico's Network Policy to segregate access at the pod level, but that depends entirely on your flexibility.
I am new to kubernetes, just wanted to know if my question is valid.
I had a question if a single POD can host 2 or more services.
And If it can host multiple services, how can it differentiate the traffic between the services.
Does it do a PORT mapping.
Please, let me know.
You can add multiple containers to the same pod, but that's only recommended if the services are tightly coupled, like if they need to communicate. For example, if you have a web server and a sql database, you would likely want them in the same pod.
If the services are distinct, you would likely want to put them in different pods, but deploy them to the same cluster of nodes. Then, you can have a LoadBalancer service on the cluster that can route different ports or paths to the right pod. In this way, the services can be scaled and managed separately (and without worrying about port conflicts), but they still draw from the same pool of resources
a single POD can host 2 or more services?
I am assuming that, by services you meant Docker containers. If that's not the case, please let me know.
Yes, Single pod can host more than one containers.
how can it differentiate the traffic between the services.
That's the catch. You need to deal with it using the ports. Expose one service on one a port and another on a different port. (If you want more than one service/container to be exposed from the same pod rethink about your design, it may be an ideal candidate for a different pod)
That being said, Now let's see the best practices,
When should I use 2 Docker containers in a pod
If your both services (docker containers) are tightly coupled and when you scale one service you need to scale the other along with the former. (Trust me, this very rare scenario). Usually these are referred as side-cars.
When should use different pod for different
If you want to scale each of them independent of the other.
Examples
Microservice -- database
Microservice -- Redis cache
Edge service -- Microservice
I'm trying to understand a good way to handle Kubernetes cluster where there are several nodes and a master.
I host the cluster within the cloud of my company, plain Ubuntu boxes (so no Google Cloud or AWS).
Each pod contains the webapp (which is stateless) and I run any number of pods via replication controllers.
I see that with Services, I can declare PublicIPs however this is confusing because after adding ip addresses of
my minion nodes, each ip only exposes the pod that it runs and it doesn't do any sort of load balancing. Due to this,
if a node doesn't have any active pod running (as created pods are random allocated among nodes), it simply timeouts and I end up some IP addresses that don't response. Am I understanding this wrong?
How can I truly do a proper external load balancing for my web app? Should I do load balancing on Pod level instead of using Service?
If so, pods are considered mortal and they may dynamically die and born, how I do track of this?
The PublicIP thing is changing lately and I don't know exactly where it landed. But, services are the ip address and port that you reference in your applications. In other words, if I create a database, I create it as a pod (with or without a replication controller). I don't connect to the pod, however, from another application. I connect to a service which knows about the pod (via a label selector). This is important for a number of reasons.
If the database fails and is recreated on a different host, the application accessing it still references the (stationary) service ip address, and the kubernetes proxies take care of getting the request to the correct pod.
The service address is known by all Kubernetes nodes. Any node can proxy the request appropriately.
I think a variation of the theme applies to your problem. You might consider creating an external load balancer which forwards traffic to all of your nodes for the specific (web) service. You still need to take the node out of the balancer's targets if the node goes down, but, I think that any node will forward the traffic for any service whether or not that service is on that node.
All that said, I haven't had direct experience with external (public) ip addresses load balancing to the cluster, so there are probably better techniques. The main point I was trying to make is the node will proxy the request to the appropriate pod whether or not that node has a pod.
-g