Having problem to access deployed application in multiclustering kubernetes environment in VirtualBox - docker

I have create multicluster kubernetes environment and my node details is:
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
16-node-121 Ready <none> 32m v1.14.1 192.168.0.121 <none> Ubuntu 16.04.6 LTS 4.4.0-142-generic docker://18.9.2
master-16-120 Ready master 47m v1.14.1 192.168.0.120 <none> Ubuntu 16.04.6 LTS 4.4.0-142-generic docker://18.9.2
And I created a service and exposed the service using following command:
$kubectl expose deployment hello-world --port=80 --target-port=8080
The is created and exposed. My service detail information is:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-world ClusterIP 10.105.7.156 <none> 80/TCP 33m
I exposed my deployment by following command:
kubectl expose deployment hello-world --port=80 --target-port=8080
service/hello-world exposed
Unfortunately when I try to access my service using curl command I'm getting timeout error:
My service details are following:
master-16-120#master-16-120:~$ kubectl describe service hello-world
Name: hello-world
Namespace: default
Labels: run=hello-world
Annotations: <none>
Selector: run=hello-world
Type: ClusterIP
IP: 10.105.7.156
Port: <unset> 80/TCP
TargetPort: 8080/TCP
Endpoints: 192.168.1.2:8080
Session Affinity: None
Events: <none>
curl http://10.105.7.156:80
curl: (7) Failed to connect to 10.105.7.156 port 80: Connection timed out
Here I'm using calico for my multicluster network which is :
wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
My Pod networking specification is:
sudo kubeadm init --pod-network-cidr=192.168.0.0/16

At last I have got the solution. Thanks to Daniel's comment which helps me to reach my solution.
I change my kubernetis pod network CIDR and calico as follow:
--pod-network-cidr=10.10.0.0/16
And also configure master which is master-16-120 Hosts (/etc/hosts):
master-16-120 192.168.0.120
16-node-121 192.168.0.121
And in the node which is 16-node-121 Hosts (/etc/hosts)
master-16-120 192.168.0.120
16-node-121 192.168.0.121
Now my kubernetes is ready to go.

Related

Kubernetes: Why my NodePort can not get an external ip?

Environment information:
Computer detail: One master node and four slave nodes. All are CentOS Linux release 7.8.2003 (Core).
Kubernetes version: v1.18.0.
Zero to JupyterHub version: 0.9.0.
Helm version: v2.11.0
Recently, I try to deploy "Zero to Jupyterhub" on kubernetes. My jupyterhub config file such below:
config.yaml
proxy:
secretToken: "2fdeb3679d666277bdb1c93102a08f5b894774ba796e60af7957cb5677f40706"
service:
type: NodePort
nodePorts:
http: 30080
https: 30443
singleuser:
storage:
dynamic:
storageClass: local-storage
capacity: 10Gi
Note: I set the service type as NodePort, because I not have any cloud provider(deploy on my lab servers cluster), and I try using nginx-ingress also then got failure, that reason why I do not using LoadBalance.
But when I using this config file to install jupyterhub via Helm, I can not access jupyterhub from browser, even all Pods running. These pods detail like below:
kubectl get pod --namespace jhub
NAME READY STATUS RESTARTS AGE
continuous-image-puller-8gxxk 1/1 Running 0 27m
continuous-image-puller-8tmdh 1/1 Running 0 27m
continuous-image-puller-lwdcx 1/1 Running 0 27m
continuous-image-puller-pszsr 1/1 Running 0 27m
hub-7b9cbbcf59-fbppq 1/1 Running 0 27m
proxy-6b699b54c8-2pxmb 1/1 Running 0 27m
user-scheduler-65f4cbb9b7-9vmfr 1/1 Running 0 27m
user-scheduler-65f4cbb9b7-lqfrh 1/1 Running 0 27m
and its services like this:
kubectl get service --namespace jhub
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hub ClusterIP 10.10.55.78 <none> 8081/TCP 28m
proxy-api ClusterIP 10.10.27.133 <none> 8001/TCP 28m
proxy-public NodePort 10.10.97.11 <none> 443:30443/TCP,80:30080/TCP 28m
Is seem to work well, right? (I guessed.) But the fact is that I can not use ip 10.10.97.11 to access the jupyter main page, and I did not get any external ip also.
So, my problems are:
Do my config have any wrong?
How to get an external ip?
Finally, thank you for save my day so much!
For NodePort service you will not get EXTERNAL-IP. You can not use the CLUSTER-IP to access it from outside the kubernetes cluster because CLUSTER-IP is for accessing it from inside the kubernetes cluster typically from another pod.For accessing from outside the kubernetes cluster you need to use NodeIP:NodePort where NodeIP is your kubernetes nodes IP address.

Can not ping to pod's ip of worker node in kubernetes

My cluster includes: 1 master and 2 worker nodes. I created a pod using deployment yaml. The pod running successfully on the worker node 1, I can ping the pod's ip on worker nodes but I can't ping the ip address of the pod on the master. I tried to disable firewarlld, restart docker but not successfully. Please see my commands
[root#k8s-master ~]# kubectl get pods -o wide | grep qldv
qldv-liberty-8499dfcf67-55njr 1/1 Running 0 6m42s 10.40.0.2 worker-node1 <none> <none>
[root#k8s-master ~]# ping 10.40.0.2
PING 10.40.0.2 (10.40.0.2) 56(84) bytes of data.
From 10.32.0.1 icmp_seq=1 Destination Host Unreachable
From 10.32.0.1 icmp_seq=2 Destination Host Unreachable
From 10.32.0.1 icmp_seq=3 Destination Host Unreachable
[root#k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 43d v1.15.0
worker-node1 Ready <none> 42d v1.15.0
worker-node2 Ready <none> 42d v1.15.0
[root#k8s-master ~]# kubectl describe pod qldv-liberty-8499dfcf67-55njr
Name: qldv-liberty-8499dfcf67-55njr
Namespace: default
Priority: 0
Node: worker-node1/192.168.142.130
Start Time: Sat, 17 Aug 2019 20:05:57 +0700
Labels: app=qldv-liberty
pod-template-hash=8499dfcf67
Annotations: <none>
Status: Running
IP: 10.40.0.2
Controlled By: ReplicaSet/qldv-liberty-8499dfcf67
Containers:
qldv-liberty:
Container ID: docker://03636fb62d4cca0e41f4ad9f5a94b50cf371089ab5a0813ed802d02f4ac4b07a
Image: qldv-liberty
Image ID: docker://sha256:bd0d7ce1c07da5b9d398131b17da7a6931a9b7ae0673d19a6ec0c409416afc69
Port: 9080/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 17 Aug 2019 20:06:23 +0700
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-vtphv (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-vtphv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-vtphv
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 119s default-scheduler Successfully assigned default/qldv-liberty-8499dfcf67-55njr to worker-node1
Normal Pulled 96s kubelet, worker-node1 Container image "qldv-liberty" already present on machine
Normal Created 95s kubelet, worker-node1 Created container qldv-liberty
Normal Started 91s kubelet, worker-node1 Started container qldv-liberty
I have another app, it also has a pod that running on the worker node 1, and I can ping the pod's ip from master. But I don't know why it is impossible with above case.
Please help me !
I doubt that the cluster still exists, therefore I'd better share some troubleshooting tips:
Check status of all control plane components and node status. Ensure kube-proxy and network addon (flannel/calico/waive/etc) Pods exist on each node and in Ready state.
kubectl get deployments,daemonsets,pods,svc -A -o wide
        There are several requirements for Kubernetes cluster, and it worth to check if they are satisfied.
        Some useful information could be found in the control-plane component logs using
kubectl logs kube-component-name-pod -n kube-system
        or kubelet logs using
journalctl -u kubelet
It's better to use well known images like nginx or mendhak/http-https-echo. They could be configured to listen any desired port and provide detailed information about requests in logs or in HTTP reply. It helps to exclude application/image related issues.
Check connectivity to Pod IP and Service ClusterIP within the same node first.
If worker node OS doesn't have necessary tools for troubleshooting (e.g container optimized images or coreOS), Pod with Ubuntu or Busybox image can be used for that. Creating Deployment or DaemonSet could help to schedule it on all nodes. Note that firewall or network issues can block kubectl exec connections to that pods.
If everything works fine within the same node, but connection to the Pod can't be established from another node it worth to check network addon status and nodes firewall configuration. Native Linux firewall helpers can interfere with iptables set of rules created by kube-proxy and block the connection.
        Clusters created in public clouds may require additional routing, peering, cloud firewall or security groups configuration to allow full IPIP connectivity between cluster nodes, especially if they are created in different VPCs.
The next thing that worth to check is coredns/kube-dns health. They suppose to resolve to correct IP address cluster Services' names like servicename.namespacename.svc.cluster.local if requested using their pod IP addresses or kube-dns Service (it usually has IP address 10.96.0.10 in default kubeadm cluster configuration).
Solution for each problem could be found in another answers on StackExchange sites. Official documentation is another great source of information and also contains good examples.

Routing between Kubernetes cluster and Docker container in the VM

I have setup Kubernates cluster in the VM (Ubuntu 18.04.1 LTS) on the Azure cloud using preconfigured scripts.
MongoDB docker container is running along with K8s cluster. My aim is to connect MongoDB to CMS container which is running inside the K8s.
Docker containers:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3883f7b397cf mongo "docker-entrypoint.s…" 5 hours ago Up 5 hours 0.0.0.0:27017->27017/tcp mongodb
299239d90cbb mirantis/kubeadm-dind-cluster:v1.12 "/sbin/dind_init sys…" 27 hours ago Up 27 hours 8080/tcp kube-node-2
34c8bd5fad2e mirantis/kubeadm-dind-cluster:v1.12 "/sbin/dind_init sys…" 27 hours ago Up 27 hours 8080/tcp kube-node-1
15a2d6521e6e mirantis/kubeadm-dind-cluster:v1.12 "/sbin/dind_init sys…" 27 hours ago Up 27 hours 127.0.0.1:32768->8080/tcp kube-master
Kubernates nodes:
$ kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 26h <none>
mycms LoadBalancer 10.97.53.114 <pending> 80:31664/TCP 18s app=mycms,tier=frontend
Kubernates service:
$ kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 26h <none>
mycms LoadBalancer 10.97.53.114 <pending> 80:31664/TCP 112s app=mycms,tier=frontend
Kubernates pods:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
mycms-dc4978ffc-khvj2 1/1 Running 0 4m8s 10.244.2.13 kube-node-1 <none>
MongoDB container's IP address is 172.17.0.2
Kubernates master container IP address is 10.192.0.2
Kubernates node 1 container IP address is 10.192.0.3
Kubernates node 2 container IP address is 10.192.0.4
As CMS pod is running on 10.244.2.13 which is inside the k8s container.
For testing, I have installed mongo-client on the host and test the connection which works well.
But CMS doesn't reach MongoDB container (I am passing Mongo String to pod in an environmental variable).
CMS pod's log
MongoError: failed to connect to server [172.17.0.2:27017] on first connect [MongoError: connect EHOSTUNREACH 172.17.0.2:27017]
How do I route MongoDB container and CMS container? Is anything wrong/missed in my approach?
Please let me know if you need further information. Thanks!
You need to use the IP address of the host where Docker is installed, not internal MongoDB container's IP address, to connect to MongoDB from the Kubernetes cluster or form any other host. According to the results of your docker ps -a, you have exposed 27017 port for MongoDB container, therefore <hostIP>:27017 should be used, not 172.17.0.2:27017.
By default in Kubernetes, there are no restrictions to connect outside the cluster.
Also, you may have firewall rules in Azure that forbid connections between hosts.

Kubernetes CoreDNS in CrashLoopBackOff

I understand that this question is asked dozen times, but nothing has helped me through internet searching.
My set up:
CentOS Linux release 7.5.1804 (Core)
Docker Version: 18.06.1-ce
Kubernetes: v1.12.3
Installed by official guide and this one:https://www.techrepublic.com/article/how-to-install-a-kubernetes-cluster-on-centos-7/
CoreDNS pods are in Error/CrashLoopBackOff state.
kube-system coredns-576cbf47c7-8phwt 0/1 CrashLoopBackOff 8 31m
kube-system coredns-576cbf47c7-rn2qc 0/1 CrashLoopBackOff 8 31m
My /etc/resolv.conf:
nameserver 8.8.8.8
Also tried with my local dns-resolver(router)
nameserver 10.10.10.1
Setup and init:
kubeadm init --apiserver-advertise-address=10.10.10.3 --pod-network-cidr=192.168.1.0/16
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
I tried to solve this with:
Editing the coredns: root#kub~]# kubectl edit cm coredns -n kube-system
and changing
proxy . /etc/resolv.conf
directly to
proxy . 10.10.10.1
or
proxy . 8.8.8.8
Also tried to:
kubectl -n kube-system get deployment coredns -o yaml | sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | kubectl apply -f -
And still nothing helps me.
Error from the logs:
plugin/loop: Seen "HINFO IN 7847735572277573283.2952120668710018229." more than twice, loop detected
The other thread - coredns pods have CrashLoopBackOff or Error state didnt help at all, becouse i havent hit any solutions that were described there. Nothing helped.
Even I have got such error and I successfully managed to work by below steps.
However, you missed 8.8.4.4
sudo nano /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
run following commands to restart daemon and docker service
sudo systemctl daemon-reload
sudo systemctl restart docker
If you are using kubeadm make sure you delete an entire cluster from master and provision cluster again.
kubectl drain <node_name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node_name>
kubeadm reset
Once You Provision the new cluster
kubectl get pods --all-namespaces
It Should give below expected Result
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-gldlr 2/2 Running 0 24s
kube-system coredns-86c58d9df4-lpnj6 1/1 Running 0 40s
kube-system coredns-86c58d9df4-xnb5r 1/1 Running 0 40s
kube-system kube-proxy-kkb7b 1/1 Running 0 40s
kube-system kube-scheduler-osboxes 1/1 Running 0 10s
$kubectl edit cm coredns -n kube-system
delete ‘loop’ ,save and exit
restart master node. It was work for me.
I faced the the same issue in my local k8s in Docker (KIND) setup. CoreDns pod gets crashloop backoff error.
Steps followed to make the pod into running state:
As Tim Chan said in this post and by referring the github issues link, I did the following
kubectl -n kube-system edit configmaps coredns -o yaml
modify the section
forward . /etc/resolv.conf with forward . 172.16.232.1 (mycase i set 8.8.8.8 for the timebeing)
Delete one of the Coredns Pods, or can wait for sometime - the pods will be in running state.
Usually happens when coredns can't talk to the kube-apiserver:
Check that your kubernetes service is in the default namespace:
$ kubectl get svc kubernetes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 130d
Then (you might have to create a pod):
$ kubectl -n kube-system exec -it <any-pod-with-shell> sh
# ping kubernetes.default.svc.cluster.local
PING kubernetes.default.svc.cluster.local (10.96.0.1): 56 data bytes
Also, try hitting port 443 from the port:
# telnet kubernetes.default.svc.cluster.local 443 # or
# curl kubernetes.default.svc.cluster.local:443
I got the error is:
connect: no route to host","time":"2021-03-19T14:42:05Z"}
crashloopbackoff
in the log showed by kubectl -n kube-system logs coredns-d9fdb9c9f-864rz
The issue is mentioned in https://github.com/coredns/coredns/tree/master/plugin/loop#troubleshooting-loops-in-kubernetes-clusters
tldr;
Reason: /etc/resolv.conf got updated somehow. The original one is at /run/systemd/resolve/resolv.conf:
e.g:
nameserver 172.16.232.1
Quick fix, edit Corefile:
$ kubectl -n kube-system edit configmaps coredns -o yaml
to replace forward . /etc/resolv.conf with forward . 172.16.232.1
e.g:
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . 172.16.232.1 {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2021-03-18T15:58:07Z"
name: coredns
namespace: kube-system
resourceVersion: "49996"
uid: 428a03ff-82d0-4812-a3fa-e913c2911ebd
Done, after that, may need to restart the docker
sudo systemctl restart docker
Update: it could be fixed by just sudo systemctl restart docker

openshift internal docker registry repo address is no same as docker-registry service cluster ip

My steps in one of my cluster Master server:
Create router:
# oadm router ose-router --replicas=1 --credentials='/etc/origin/master/openshift-router.kubeconfig' --images='openshift-register.com.cn:5000/openshift3/ose-${component}:v3.4' --service-account=router
Create docker registry:
# oadm registry --config=/etc/origin/master/admin.kubeconfig --service-account=registry --images='openshift-register.com.cn:5000/openshift3/ose-${component}:v3.4'
Router and registry are created successfully and we can see the docker-registry cluster-IP address is 172.30.182.170:
# oc get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
docker-registry 172.30.182.170 <none> 5000/TCP 22s
kubernetes 172.30.0.1 <none> 443/TCP,53/UDP,53/TCP 6d
ose-router 172.30.80.196 <none> 80/TCP,443/TCP,1936/TCP 1m
Query the iamge stems in namespace openshift, the Docker repo IP is 172.30.137.159:
# oc get is -n openshift
NAME DOCKER REPO TAGS
jenkins 172.30.137.159:5000/openshift/jenkins 2,1
mariadb 172.30.137.159:5000/openshift/mariadb 10.1
mongodb 172.30.137.159:5000/openshift/mongodb 3.2,2.6,2.4
mysql 172.30.137.159:5000/openshift/mysql 5.7,5.6,5.5
nodejs 172.30.137.159:5000/openshift/nodejs 4,0.10
perl 172.30.137.159:5000/openshift/perl 5.24,5.20,5.16
php 172.30.137.159:5000/openshift/php 5.5,7.0,5.6
postgresql 172.30.137.159:5000/openshift/postgresql 9.5,9.4,9.2
python 172.30.137.159:5000/openshift/python 3.4,3.3,2.7 +
more...
redis 172.30.137.159:5000/openshift/redis 3.2
ruby 172.30.137.159:5000/openshift/ruby 2.3,2.2,2.0
My concern and question are the docker repo IP address should be using my docker-registry service cluster IP by default, but why it generated a new IP address and I have no idea how/where to find this docker repo IP address? I'm really new openshift users, so if you can give any suggestion that would be appreciated.
[root#ocp-master01 ~]# oc get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
docker-registry 172.30.182.170 <none> 5000/TCP 49m
kubernetes 172.30.0.1 <none> 443/TCP,53/UDP,53/TCP 7d
ose-router 172.30.80.196 <none> 80/TCP,443/TCP,1936/TCP 50m
[root#ocp-master01 ~]# oc get svc -o yaml | grep IP
clusterIP: 172.30.182.170
sessionAffinity: ClientIP
type: ClusterIP
clusterIP: 172.30.0.1
sessionAffinity: ClientIP
type: ClusterIP
clusterIP: 172.30.80.196

Resources