I have been building a distributed load testing application using Kubernetes and Locust (similar to this).
I currently have a multi-node cluster running on bare-metal (running on an Ubuntu 18.04 server, set up using Kubeadm, and with Flannel as my pod networking addon).
The architecture of my cluster is as follows:
I have a 'master instance' of the Locust application running on my master node.
I have 'slave instances' of the Locust application running on all of my other nodes. These slave instances must be able to bind to a port (5558 by default) of the master instance.
As of now, I don't believe that that is happening. My cluster shows that all of my deployments are healthy and running, however I am unable to access the logs of any of my slave instances which are running on nodes other than my master node. This leads me to believe that my pods are unable to communicate with each other across different nodes.
Is this an issue with my current networking or deployment setups (I followed the linked guides pretty-much verbatim)? Where should I start in debugging this issue?
How slaves instances try to join the master instance. You have to create master service (with labels) to access master pod. Also, make sure your SDN is up and master is reachable to slave instances. You can test using telnet to master pod IP from slave instances.
Based on your description of the problem I can guess that you have a connection problem caused by firewall or network misconfiguration.
From the network perspective, there are requirements mentioned in Kubernetes documentation:
all containers can communicate with all other containers without NAT
all nodes can communicate with all containers (and vice-versa) without NAT
the IP that a container sees itself as is the same IP that others see it as
From the firewall perspective, you need to ensure the cluster traffic can pass the firewall on the nodes.
Here is the list of ports you should have opened on the nodes provided by CoreOS website:
Master node inbound: TCP: 443 from Worker Nodes, API Requests, and End-Users
UDP: 8285,8472 from Master & Worker Nodes
Worker node inbound: TCP: 10250 from Master Nodes
TCP: 10255 from Heapster
TCP: 30000-32767 from External Application Consumers
TCP: 1-32767 from Master & Worker Nodes
TCP: 179 from Worker Nodes
UDP: 8472 from Master & Worker Nodes
UPD: 179 from Worker Nodes
Etcd node inbound: TCP: 2379-2380 from Master & Worker Nodes
see ip forwarding is enabled on all the nodes.
# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
if not enable it like this and test it.
echo 1 > /proc/sys/net/ipv4/ip_forward
Related
I am running into a peculiar problem.
I have kubernetes cluster, I setup no_proxy for the master node of the cluster (in docker systemd environment). In order to be able to run docker build/push to a registry that is running on docker on the master node.
Now I have a problem, as my containers cannot access the outside network (because the communication happens through k8s master node I presume).
Or if I choose not to set no_proxy for the master node in docker then I cannot push images to my registry through the external IP of the master, have to use (localhost) as push destination -> which breaks my app later on.
I use weave as my cni plugin
The network communication of containers running on your nodes has nothing to do with the network communication of your master to the outside world or it through a proxy.
Basically, the network communication for your containers running on a node goes through its own network interface, etc.
Having said that, are you running your workloads on your master? If yes, that could be affecting the communication of your master containers (if you set no_proxy for some hostnames). It could also be affecting the communication of your kube-controller-manager, kube-apiserver, core-dns, kubelet and network overlay on the master.
Are you configuring your docker client proxy correctly as per here?
I am using micro-service to access hadoop and hbase to get data but it's not accessible from pod.
It shows only:
INFO ipc.Client: Retrying connect to server: hdpcluster.internal/10.160.0.2:8020. Already tried 3 time(s); maxRetries=45
IP 10.160.0.2 is accessible from all nodes and they are on GCP.
You probably need to open a firewall rule to allow port 8020 on your Hbase nodes that your Kubernetes nodes can connect to them. Something like this on your Hbase firewall rules (for your Hbase nodes):
I would like to host a Kubernetes master node in AWS (or other cloud provider) and then add nodes from home to that cluster. I do however not have a static IP from my internet provider, so the question is: will this work and what happens when my IP address change?
Here could get some info about Master-Node communication in kubernetes.
For communication from Node to Mater, it will use kube-apiserver to do requests. So normally it should be work, and when your node IP is changed, node info in ETCD for your node will be update, and you could check your nodes status with command kubectl get nodes -o wide
But if some specific kubernetes feature may be affected, such as NodePort for Service.
Hope this could help !
I am currently trying to setup a Docker cluster in the following way:
NodeA: SwarmManager1 + Consul1
NodeB: SwarmManager2 + Consul2
NodeC: SwamNode1(advertising to Consul1) + Consul3
NodeD: SwarmNode2(adverting to Consul2)
I made some HA testing and found the follwing behavior:
I have restarted NodeB while monitoring the docker cluster info and I noticed that SwarmNode2 was disconnected from the cluster during the reboot time.
The explanation I have is that because Consul2 goes down and Node2 is configured to connect to that same Consul it becomes unavailable on the cluster perspective.
What is the correct way to setup the discovery service for the Swarm containers in order to avoid this problem?
I suggest creating a consul cluster, preferably stand-alone on different nodes.
Once the cluster is created, all consul clients should continue functioning properly as long as quorum is maintained.
I also suggest giving multiple consul server addresses with the -join flag, to ensure the agent will be able to rejoin in case it restarts while some of the consul servers are down.
I run my kubernetes cluster in three machines. I create a redis service and rc which runs a pod creating three replications. So three containers is running on two of the machines. But the redis slave which runs in a single node fails to connnect to the master which runs in the other.
node1 -> master and slave1
node2 -> slave2
The slave2 just complains like this:
Could not connect to Redis at 192.168.0.4:6379: Connection refused
Connecting to master failed. Waiting...
Error: Connection reset by peer
Could not connect to Redis at 192.168.0.4:6379: Connection refused
Connecting to master failed. Waiting...
Error: Connection reset by peer
Could not connect to Redis at 192.168.0.4:6379: Connection refused
Connecting to master failed. Waiting...
... (a lot of them...)
while the other slave(slave1) runs well in the node which runs the master.
So I don't know what is the problem.
Why slave2 tries to connect to this ip '192.168.0.4' instead of 127.0.0.1 (it's said that containers in a pod take the same ip.).
By the way, do containers have its own ip within a pod so that they can communicate with each other and separate themselves from each other.
Oh, I've got it. Containers running in a pod can't be separated into different nodes.
You need to use a service listing all the Redis nodes (master) to connect to them through the service. You can't just use IP as IP may change if the node goes down and the pod is restarted on another node.
Also, 127.0.0.1 is localhost, and that only works for containers within the same pod. (a pod links containers, and runs on a node. pods are deployed/replicated on multiple nodes.)