I have a nodejs app which connects to external db , the db will refuse the connection until I whitelist my ip or my pod will fail , so is it possible that my external ip for the service will be stuck on pending if the pod fails?
is it possible that my external ip for the service will be stuck on pending if the pod fails?
The Service and Pods are created separately. So if you're creating a LoadBalancer-type Service and your cluster is correctly configured, you should be able to get an externalIP: address for it even if the Pods aren't correctly starting up.
But:
I have a nodejs app which connects to external db , the db will refuse the connection until I whitelist my ip
The Service only accepts inbound connections. In a cloud environment like AWS, the externalIP: frequently is the address of a specific load balancer. Outbound requests to a database won't usually come from this address.
If your cluster is in the same network environment as the database, you probably need to allow every individual worker node in the database configuration. Tools like the cluster autoscaler can cause the node pool to change, so if you can configure the entire CIDR block containing the cluster that's easier. If the cluster is somewhere else and outbound traffic passes through a NAT gateway of some sort, then you need to allow that gateway.
Related
So right now we are trying to get a Bitnami Redis Sentinel cluster working, together with our Rails app + Sidekiq.
We tried different things, but it's not really clear to us, how we should specify the sentinels for Sidekiq (crutial part is here, that the sentinel nodes are READ ONLY, so we cannot use them for sidekiq, since job states get written).
Since on Kubernetes there are only 2 services available: "redis" and "redis-headless" (not sure how they differ?) - how can I specify the sentinels like this:
Sidekiq.configure_server do |config|
config.redis = {
url: "redis",
sentinels: [
{ host: "?", port: 26379 } # why do we have to specify it here seperately, since we should be able to get a unified answer via a service, or?
{ host: "?", port: 26379 }
{ host: "?", port: 26379 }
}
}
end
Would be nice if someone can shed some light on this. As far as I understood, the bitnami redis sentiel only returns the IP of the master and the application has to handle the corresponding writes to this master then (https://github.com/bitnami/charts/tree/master/bitnami/redis#master-replicas-with-sentinel) - but I really don't understand on how to do this with sidekiq?
Difference between a Kubernetes Service and a Headless Service
Let's get started by clarifying the difference between a Headless Service and a Service.
A Service allows one to connect to one Pod, while a headless Service returns the list of available IP addresses from all the available pods, allowing to auto-discover.
A better detailed explanation by Marco Luksa has been published on SO here:
Each connection to the service is forwarded to one randomly selected backing pod. But what if the client needs to connect to all of those pods? What if the backing pods themselves need to each connect to all the other backing pods. Connecting through the service clearly isn’t the way to do this. What is?
For a client to connect to all pods, it needs to figure out the the IP of each individual pod. One option is to have the client call the Kubernetes API server and get the list of pods and their IP addresses through an API call, but because you should always strive to keep your apps Kubernetes-agnostic, using the API server isn’t ideal
Luckily, Kubernetes allows clients to discover pod IPs through DNS lookups. Usually, when you perform a DNS lookup for a service, the DNS server returns a single IP — the service’s cluster IP. But if you tell Kubernetes you don’t need a cluster IP for your service (you do this by setting the clusterIP field to None in the service specification ), the DNS server will return the pod IPs instead of the single service IP. Instead of returning a single DNS A record, the DNS server will return multiple A records for the service, each pointing to the IP of an individual pod backing the service at that moment. Clients can therefore do a simple DNS A record lookup and get the IPs of all the pods that are part of the service. The client can then use that information to connect to one, many, or all of them.
Setting the clusterIP field in a service spec to None makes the service headless, as Kubernetes won’t assign it a cluster IP through which clients could connect to the pods backing it.
"Kubernetes in Action" by Marco Luksa
How to specify the sentinels
As the Redis documentation say:
When using the Sentinel support you need to specify a list of sentinels to connect to. The list does not need to enumerate all your Sentinel instances, but a few so that if one is down the client will try the next one. The client is able to remember the last Sentinel that was able to reply correctly and will use it for the next requests.
So the idea is to give what you have, and if you scale up the redis pods, then you don't need to re-configure Sidekiq (or Rails if you're using Redis for caching).
Combining all together
Now you just need a way to fetch the IP addresses from the headless service in Ruby, and configure Redis client sentinels.
Fortunately, since Ruby 2.5.0, the Resolv class is available and can do that for you.
irb(main):007:0> Resolv.getaddresses "redis-headless"
=> ["172.16.105.95", "172.16.105.194", "172.16.9.197"]
So that you could do:
Sidekiq.configure_server do |config|
config.redis = {
# This `host` parameter is used by the Redis gem with the Redis command
# `get-master-addr-by-name` (See https://redis.io/topics/sentinel#obtaining-the-address-of-the-current-master)
# in order to retrieve the current Redis master IP address.
host: "mymaster",
sentinels: Resolv.getaddresses('redis-headless').map do |address|
{ host: address, port: 26379 }
end
}
end
That will create an Array of Hashes with the IP address as host: and 26379 as the port:.
I have 5 microservices in 5 pods and have deployed each service using specific port using NODE PORT service.
I have a UI app as one service inside another pod which is also exposed using node port service.
Since I can't use pod IP to access urls in UI app as pods live and die so deployed as nodeport service and can I access all 5 services inside UI app seamlessly using respective node port?
Please advise - is this approach going to be reliable?
Yes, you can connect to those Node port services seamlessly.
But remember, you may need higher network bandwidth card and connection (to master nodes) if you get too much traffic to these services.
Also if you have a few master nodes, you can try dedicated master node-ip and nodeport for a service.(If you have 5 master nodes, each service is accessed from one master node's IP etc. This is not mandatory, you can connect to each service using any masterIP:nodeport)
Highly recommend to use load-balancer service for this. If you have baremetal cluster try using MetalLB.
Edit : (after Nagappa LM`s comment)
If its for QA, then no need to worry, but if they perform load test to all the services simultaneously could be a problematic.
Your code change means, only your k8 - deployment is changed, not Kubernetes service. k8 service is where you define nodeport
This question has been asked and answered before on stackoverflow but because I'm new to K8, I don't understand the answer.
Assuming I have two containers with each container in a separate POD (because I believe this is the recommend approach), I think I need to create a single service for my two pods to be apart of.
How does my java application code get the IP address of the service?
How does my java application code get the IP addresses of another POD/container (from the service)?
This will be a list of IP address because these are stateless and they might be replicated. Is this correct?
How do I select the least busy instance of the POD to communicate with?
Thanks
Siegfried
How does my java application code get the IP address of the service?
You need to create a Service to expose the Pod's port and then you just need to use the Service name and kube-dns will resolve the Pod's IP address
How does my java application code get the IP addresses of another
POD/container (from the service)?
Yes, using the service's name
This will be a list of IP address because these are stateless and they
might be replicated. Is this correct?
The Service will load balance between all pods that matches the selector, so it could be 0, 1 or any number of Pods
How do I select the least busy instance of the POD to communicate with?
Common way is round robin policy but here are other specific balancing policies
https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs
Cheers ;)
You don't need to get any IP, you use the service name (DNS). So if you called your service "java-service-1" and exposed port 80, you can access it this way from inside the cluster:
http://java-service-1
If the service is in a different namespace, you have to add that as well (see https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/)
You also don't select the least busy instance yourself, a service can be configured as LoadBalancer, Kubernetes does all of this for you (see https://kubernetes.io/docs/concepts/services-networking/)
I'm trying to get down and dirty with DevOps and I'm running into a health check request timed out failure. The problem is my Elastic Load Balancer sends a health check to my EC2 instance and gets a network timeout. I'm not sure what I did wrong. I am following this tutorial and I have completed all the steps up to and including "Using a Elastic Load Balancer". My EC2 instance seems to be working fine and I am able to successfully curl localhost on port 9292 from within the EC2 instance.
EC2 instance security group setup:
Elastic Load Balancer setup:
My target group for the ELB routing has port 9292 open via HTTP and here's a screenshot of the target in my target group that is unhealthy.
Health check config:
I have a VPC that my EC2 instance is a part of and my ELB is connected to the same VPC. I do not have Apache installed and I do not have nginx installed. To my understanding, I do not need these. I have a Rails Puma server running and I can send successful curl requests to the server.
My hunch is that my ELB is not allowed to reach my EC2 instance, resulting in a network timeout and a failed health check. I'm unable to find the cause for this. Any ideas? This SO post didn't help much. Are my security groups misconfigured? What else could potentially block a routing request from ELB to my EC2 instance?
Also, is there a way to view network requests / logs for my EC2 instance? I keep seeing VPC flow logging but I feel like there are simpler alternatives.
Here's something I posted in the AWS forums but to no avail.
UPDATE: I can curl the private IP of target just fine from within an EC2 instance. I don't think it's the target instance, I think it's something to do with the security group setup. I am unable to identify why though because I have basically allowed all traffic from the Load Balancer to the EC2 instance.
I made my mistake during the "Setup your VPC" step. I finished creating a subnet for an RDS instance. I proceeded to start an instance and the default subnet that AWS chose when I switched to my VPC was the subnet I made for my RDS, which was NOT a public subnet. Therefore, any attempts, from any EC2 instance or my load balancer, would not be able to reach it because I had only set up my public subnet to take requests.
The solution was to create a new instance and this time, pick the correct public subnet. My original EC2 instance was associated with a private subnet while the load balancer was pointing to the public subnet.
Here's a link to a hand drawn image that helped me pin point my problem, hopefully can help anyone else who's having trouble setting up. I didn't put image here directly because it's bigger than 2MB.
Glad to answer any further questions too!
Is it possible to know what range of external IP the DataFlow workers on GCP are using? The goal is to set-up some kind of IP filtering on an external service, so that only our DataFlow jobs running on GCP can access the service.
The best solution would be to upgrade so that you can use SSL or other mechanisms of strong authentication.
You can use the --network= option to control the GCE Network that the worker VMs are assigned to. Take a look at the GCE docs on networking for details on how to set up a VPN (like the comment from Elmar suggested). You could also look at setting up a single machine in the network with a static, external IP and using it as a proxy for the other VMs in the network.
This is not a use pattern we have tested, so there may be issues with latency or throughput of traffic through the proxy/VPN. You will likely need to be careful to only send your traffic through this proxy so that you don’t accidentally hijack the traffic used by each worker to communicate with the Dataflow service.