Problem with minkube connecting to Node IP and getting timeout - docker

Trying to connect to IP of a node and getting a timeout error
I have started minikube
minikube start
* minikube v1.27.0 on Microsoft Windows 10 Pro 10.0.19042 Build 19042
! Kubernetes 1.25.0 has a known issue with resolv.conf. minikube is using a workaround that should work for most use cases.
! For more information, see: https://github.com/kubernetes/kubernetes/issues/112135
* Using the hyperv driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Restarting existing hyperv VM for "minikube" ...
* Preparing Kubernetes v1.25.0 on Docker 20.10.18 ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
! C:\Program Files\Docker\Docker\resources\bin\kubectl.exe is version 1.22.4, which may have incompatibilites with Kubernetes 1.25.0.
- Want kubectl v1.25.0? Try 'minikube kubectl -- get pods -A'
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Checked that the pods are up and running
PS C:\WINDOWS\system32> minikube kubectl -- get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default account-docker-kubernetes 1/1 Running 3 (2m59s ago) 3h43m
kube-system coredns-565d847f94-dh5qw 1/1 Running 4 (2m59s ago) 3h57m
kube-system etcd-minikube 1/1 Running 0 2m1s
kube-system kube-apiserver-minikube 1/1 Running 0 2m2s
kube-system kube-controller-manager-minikube 1/1 Running 4 (2m59s ago) 3h57m
kube-system kube-proxy-gs6pm 1/1 Running 4 (2m59s ago) 3h57m
kube-system kube-scheduler-minikube 1/1 Running 4 (2m59s ago) 3h57m
kube-system storage-provisioner 1/1 Running 6 (2m59s ago) 3h57m
Checked the service
PS C:\WINDOWS\system32> kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
account-docker-kubernetes NodePort 10.105.105.236 <none> 8082:30163/TCP 3h44m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h
Found the IP of the node
PS C:\WINDOWS\system32> minikube ip
172.25.177.1
so to connect to the service from outside Kubernetes its node ip plus service port
But http://172.25.177.1:30163/bank/health/ I get a timeout.
So i tried to connect to the Pod from inside first I get the pod IP.
PS C:\WINDOWS\system32> kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
account-docker-kubernetes 1/1 Running 3 (12m ago) 3h53m 172.18.0.2 minikube <none> <none>
Then exec in
PS C:\WINDOWS\system32>
PS C:\WINDOWS\system32> kubectl exec --stdin --tty account-docker-kubernetes -- /bin/bash
root#account-docker-kubernetes:/app#
Then ran the curl command on the pod returns 200
root#account-docker-kubernetes:/app# curl -v http://172.18.0.2:8082/bank/health/
* Trying 172.18.0.2:8082...
* TCP_NODELAY set
* Connected to 172.18.0.2 (172.18.0.2) port 8082 (#0)
> GET /bank/health/ HTTP/1.1
> Host: 172.18.0.2:8082
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 2
< Date: Sat, 01 Oct 2022 18:09:21 GMT
<
* Connection #0 to host 172.18.0.2 left intact
So I can connect from inside the pod but not from outside using the node IP.
I have also set up a firewall rule to allow connect to port 30163.

Related

Rancher cattle-cluster-agent Failed to connect to xxx port 443: Connection timed out)

This is my environment:
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
docker version
Client: Docker Engine - Community
Version: 20.10.13
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 10 14:07:51 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
kubectl version --short
Client Version: v1.23.5
Server Version: v1.23.5
sudo ufw status
Status: inactive
kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master,worker 34h v1.23.5
k8s-node-216 Ready worker 33h v1.23.5
kubectl cluster-info
Kubernetes control plane is running at https://k8s-api:6443
CoreDNS is running at https://k8s-api:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
sudo docker run -d \
-p 8888:80 -p 443:443 \
--name rancher \
--privileged \
--restart=unless-stopped \
-e CATTLE_BOOTSTRAP_PASSWORD=cquissE! \
-v /home/ngtl/soft/rancher:/var/lib/rancher \
rancher/rancher:latest
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-apiserver calico-apiserver-6f45746584-9bwjb 1/1 Running 0 33h
calico-apiserver calico-apiserver-6f45746584-cf9v6 1/1 Running 0 33h
calico-system calico-kube-controllers-67f85d7449-ddjmw 1/1 Running 0 33h
calico-system calico-node-22f4d 0/1 Running 0 33h
calico-system calico-node-szb5b 0/1 Running 0 33h
calico-system calico-typha-856cc5f7bb-cqj8h 1/1 Running 0 33h
cattle-system cattle-cluster-agent-7d8645db69-nkh4z 0/1 CrashLoopBackOff 22 (4m56s ago) 168m
kube-system coredns-6d8c4cb4d-hlvmx 1/1 Running 0 34h
kube-system coredns-6d8c4cb4d-ttttq 1/1 Running 0 34h
kube-system etcd-k8s-master 1/1 Running 2 34h
kube-system kube-apiserver-k8s-master 1/1 Running 2 34h
kube-system kube-controller-manager-k8s-master 1/1 Running 2 34h
kube-system kube-proxy-7fmwl 1/1 Running 0 33h
kube-system kube-proxy-bnd7k 1/1 Running 0 34h
kube-system kube-scheduler-k8s-master 1/1 Running 2 34h
tigera-operator tigera-operator-b876f5799-kc8cz 1/1 Running 0 33h
I have two nodes:
k8s-master => 192.168.2.85
k8s-node-216 => 192.168.2.216
And my rancher is running on k8s-master, rancher agent is running on k8s-node-216.
You can see that the Cattle-Cluster-Agent state is CrashLoopBackOff.
And it says:
kubectl -n cattle-system logs -f pod/cattle-cluster-agent-7d8645db69-nkh4z
INFO: Environment: CATTLE_ADDRESS=192.168.127.130 CATTLE_CA_CHECKSUM=ca19550fd4ff72e9daf89936fab0ecea12aa8d97af7b3b91108ebd862b784e57 CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://10.104.73.193:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://10.104.73.193:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.104.73.193 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://10.104.73.193:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.104.73.193 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.104.73.193 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY= CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=8ce09b15-0952-41c5-83d6-b74dee853902 CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-7d8645db69-nkh4z CATTLE_SERVER=https://192.168.2.85 CATTLE_SERVER_VERSION=v2.6.3
INFO: Using resolv.conf: nameserver 10.96.0.10 search cattle-system.svc.cluster.local svc.cluster.local cluster.local options ndots:5
ERROR: https://192.168.2.85/ping is not accessible (Failed to connect to 192.168.2.85 port 443: Connection timed out)
But telnet is work well:
telnet 192.168.2.85 443
Trying 192.168.2.85...
Connected to 192.168.2.85.
Escape character is '^]'.
What's wrong with my rancher?
The problem has been solved.
That's because calico calico-node not get correct IP address.
So just edit the custom-resources.yaml and specifie an approach to automatically detect node IPv4 addresses.
see:
https://projectcalico.docs.tigera.io/reference/installation/api#operator.tigera.io/v1.CalicoNetworkSpec

Kubernetes CoreDNS in CrashLoopBackOff

I understand that this question is asked dozen times, but nothing has helped me through internet searching.
My set up:
CentOS Linux release 7.5.1804 (Core)
Docker Version: 18.06.1-ce
Kubernetes: v1.12.3
Installed by official guide and this one:https://www.techrepublic.com/article/how-to-install-a-kubernetes-cluster-on-centos-7/
CoreDNS pods are in Error/CrashLoopBackOff state.
kube-system coredns-576cbf47c7-8phwt 0/1 CrashLoopBackOff 8 31m
kube-system coredns-576cbf47c7-rn2qc 0/1 CrashLoopBackOff 8 31m
My /etc/resolv.conf:
nameserver 8.8.8.8
Also tried with my local dns-resolver(router)
nameserver 10.10.10.1
Setup and init:
kubeadm init --apiserver-advertise-address=10.10.10.3 --pod-network-cidr=192.168.1.0/16
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
I tried to solve this with:
Editing the coredns: root#kub~]# kubectl edit cm coredns -n kube-system
and changing
proxy . /etc/resolv.conf
directly to
proxy . 10.10.10.1
or
proxy . 8.8.8.8
Also tried to:
kubectl -n kube-system get deployment coredns -o yaml | sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | kubectl apply -f -
And still nothing helps me.
Error from the logs:
plugin/loop: Seen "HINFO IN 7847735572277573283.2952120668710018229." more than twice, loop detected
The other thread - coredns pods have CrashLoopBackOff or Error state didnt help at all, becouse i havent hit any solutions that were described there. Nothing helped.
Even I have got such error and I successfully managed to work by below steps.
However, you missed 8.8.4.4
sudo nano /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
run following commands to restart daemon and docker service
sudo systemctl daemon-reload
sudo systemctl restart docker
If you are using kubeadm make sure you delete an entire cluster from master and provision cluster again.
kubectl drain <node_name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node_name>
kubeadm reset
Once You Provision the new cluster
kubectl get pods --all-namespaces
It Should give below expected Result
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-gldlr 2/2 Running 0 24s
kube-system coredns-86c58d9df4-lpnj6 1/1 Running 0 40s
kube-system coredns-86c58d9df4-xnb5r 1/1 Running 0 40s
kube-system kube-proxy-kkb7b 1/1 Running 0 40s
kube-system kube-scheduler-osboxes 1/1 Running 0 10s
$kubectl edit cm coredns -n kube-system
delete ‘loop’ ,save and exit
restart master node. It was work for me.
I faced the the same issue in my local k8s in Docker (KIND) setup. CoreDns pod gets crashloop backoff error.
Steps followed to make the pod into running state:
As Tim Chan said in this post and by referring the github issues link, I did the following
kubectl -n kube-system edit configmaps coredns -o yaml
modify the section
forward . /etc/resolv.conf with forward . 172.16.232.1 (mycase i set 8.8.8.8 for the timebeing)
Delete one of the Coredns Pods, or can wait for sometime - the pods will be in running state.
Usually happens when coredns can't talk to the kube-apiserver:
Check that your kubernetes service is in the default namespace:
$ kubectl get svc kubernetes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 130d
Then (you might have to create a pod):
$ kubectl -n kube-system exec -it <any-pod-with-shell> sh
# ping kubernetes.default.svc.cluster.local
PING kubernetes.default.svc.cluster.local (10.96.0.1): 56 data bytes
Also, try hitting port 443 from the port:
# telnet kubernetes.default.svc.cluster.local 443 # or
# curl kubernetes.default.svc.cluster.local:443
I got the error is:
connect: no route to host","time":"2021-03-19T14:42:05Z"}
crashloopbackoff
in the log showed by kubectl -n kube-system logs coredns-d9fdb9c9f-864rz
The issue is mentioned in https://github.com/coredns/coredns/tree/master/plugin/loop#troubleshooting-loops-in-kubernetes-clusters
tldr;
Reason: /etc/resolv.conf got updated somehow. The original one is at /run/systemd/resolve/resolv.conf:
e.g:
nameserver 172.16.232.1
Quick fix, edit Corefile:
$ kubectl -n kube-system edit configmaps coredns -o yaml
to replace forward . /etc/resolv.conf with forward . 172.16.232.1
e.g:
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . 172.16.232.1 {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2021-03-18T15:58:07Z"
name: coredns
namespace: kube-system
resourceVersion: "49996"
uid: 428a03ff-82d0-4812-a3fa-e913c2911ebd
Done, after that, may need to restart the docker
sudo systemctl restart docker
Update: it could be fixed by just sudo systemctl restart docker

Kubernetes: Container not able to ping www.google.com

I have kubernetes cluster running on 4 Raspberry-pi devices, out of which 1 is acting as master and other 3 are working as worker i.e w1, w2, w3. I have started a daemon set deployment, so each worker is running a pod of 2 containers.
w2 is running pod of 2 container. If I exec into any container and ping www.google.com from the container, I get the response. But if I do the same on w1 and w3 it says temporary failure in name resolution. All the pods in kube-system are running. I am using weave for networking. Below are all the pods for kube-system
NAME READY STATUS RESTARTS AGE
etcd-master-pi 1/1 Running 1 23h
kube-apiserver-master-pi 1/1 Running 1 23h
kube-controller-manager-master-pi 1/1 Running 1 23h
kube-dns-7b6ff86f69-97vtl 3/3 Running 3 23h
kube-proxy-2tmgw 1/1 Running 0 14m
kube-proxy-9xfx9 1/1 Running 2 22h
kube-proxy-nfgwg 1/1 Running 1 23h
kube-proxy-xbdxl 1/1 Running 3 23h
kube-scheduler-master-pi 1/1 Running 1 23h
weave-net-7sh5n 2/2 Running 1 14m
weave-net-c7x8p 2/2 Running 3 23h
weave-net-mz4c4 2/2 Running 6 22h
weave-net-qtgmw 2/2 Running 10 23h
If I am starting the containers using the normal docker container command but not from the kubernetes deployment then I do not see this issue. I think this is because of kube-dns. How can I debug this issue.?
You can start by checking if the dns is working
Run the nslookup on kubernetes.default from inside the pod, check if it is working.
[root#metrics-master-2 /]# nslookup kubernetes.default
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
Check the local dns configuration inside the pods:
[root#metrics-master-2 /]# cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5
At last, check the kube-dns container logs while you run the ping command, It will give you the possible reasons why the name is not resolving.
kubectl logs kube-dns-86f4d74b45-7c4ng -c kubedns -n kube-system
Hope this helps.
This might not be applicable to your scenario, but I wanted to document the solution I found. My issues ended up being related to a flannel network overlay setup on our master nodes.
# kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
coredns-qwer 1/1 Running 0 4h54m
coredns-asdf 1/1 Running 0 4h54m
etcd-h1 1/1 Running 0 4h53m
etcd-h2 1/1 Running 0 4h48m
etcd-h3 1/1 Running 0 4h48m
kube-apiserver-h1 1/1 Running 0 4h53m
kube-apiserver-h2 1/1 Running 0 4h48m
kube-apiserver-h3 1/1 Running 0 4h48m
kube-controller-manager-h1 1/1 Running 2 4h53m
kube-controller-manager-h2 1/1 Running 0 4h48m
kube-controller-manager-h3 1/1 Running 0 4h48m
kube-flannel-ds-amd64-asdf 1/1 Running 0 4h48m
kube-flannel-ds-amd64-qwer 1/1 Running 1 4h48m
kube-flannel-ds-amd64-zxcv 1/1 Running 0 3h51m
kube-flannel-ds-amd64-wert 1/1 Running 0 4h54m
kube-flannel-ds-amd64-sdfg 1/1 Running 1 4h41m
kube-flannel-ds-amd64-xcvb 1/1 Running 1 4h42m
kube-proxy-qwer 1/1 Running 0 4h42m
kube-proxy-asdf 1/1 Running 0 4h54m
kube-proxy-zxcv 1/1 Running 0 4h48m
kube-proxy-wert 1/1 Running 0 4h41m
kube-proxy-sdfg 1/1 Running 0 4h48m
kube-proxy-xcvb 1/1 Running 0 4h42m
kube-scheduler-h1 1/1 Running 1 4h53m
kube-scheduler-h2 1/1 Running 1 4h48m
kube-scheduler-h3 1/1 Running 0 4h48m
tiller-deploy-asdf 1/1 Running 0 4h28m
If I exec'd into any container and ping'd google.com from the container, I get a bad address response.
# ping google.com
ping: bad address 'google.com'
# ip route
default via 10.168.3.1 dev eth0
10.168.3.0/24 dev eth0 scope link src 10.168.3.22
10.244.0.0/16 via 10.168.3.1 dev eth0
ip route varies from ip route run from the master node.
altering my pods deployment configuration to include the hostNetwork: true allowed me to ping outside my container.
my newly running pod ip route
# ip route
default via 172.25.10.1 dev ens192 metric 100
10.168.0.0/24 via 10.168.0.0 dev flannel.1 onlink
10.168.1.0/24 via 10.168.1.0 dev flannel.1 onlink
10.168.2.0/24 via 10.168.2.0 dev flannel.1 onlink
10.168.3.0/24 dev cni0 scope link src 10.168.3.1
10.168.4.0/24 via 10.168.4.0 dev flannel.1 onlink
10.168.5.0/24 via 10.168.5.0 dev flannel.1 onlink
172.17.0.0/16 dev docker0 scope link src 172.17.0.1
172.25.10.0/23 dev ens192 scope link src 172.25.11.35 metric 100
192.168.122.0/24 dev virbr0 scope link src 192.168.122.1
# ping google.com
PING google.com (172.217.6.110): 56 data bytes
64 bytes from 172.217.6.110: seq=0 ttl=55 time=3.488 ms
Update 1
My associate and I found a number of different websites which advise against setting hostNetwork: true. We then found this issue and are currently investigating it as a possible solution, sans hostNetwork: true.
Usually you'd do this with the '--ip-masq' flag to flannel which is 'false' by default and is defined as "setup IP masquerade rule for traffic destined outside of overlay network". Which sounds like what you want.
Update 2
It turns out that our flannel network overlay was misconfigured. We needed to ensure that our configmap for flannel had net-conf\.json.network matching our networking.podSubnet (kubeadm config view). Changing these networks to match alleviated our networking woes. We were then able to remove hostNetwork: true from our deployments.

Coredns in pending state in Kubernetes cluster

I am trying to configure a 2 node Kubernetes cluster. First I am trying to configure the master node of the cluster on a CentOS VM. I have initialized the cluster using 'kubeadm init --apiserver-advertise-address=172.16.100.6 --pod-network-cidr=10.244.0.0/16' and deployed the flannel network to the cluster. But when I do 'kubectl get nodes', I get the following output ----
[root#kubernetus ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubernetus NotReady master 57m v1.12.0
Following is the output of 'kubectl get pods --all-namespaces -o wide ' ----
[root#kubernetus ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
kube-system coredns-576cbf47c7-9x59x 0/1 Pending 0 58m <none> <none> <none>
kube-system coredns-576cbf47c7-l52wc 0/1 Pending 0 58m <none> <none> <none>
kube-system etcd-kubernetus 1/1 Running 2 57m 172.16.100.6 kubernetus <none>
kube-system kube-apiserver-kubernetus 1/1 Running 2 57m 172.16.100.6 kubernetus <none>
kube-system kube-controller-manager-kubernetus 1/1 Running 1 57m 172.16.100.6 kubernetus <none>
kube-system kube-proxy-hr557 1/1 Running 1 58m 172.16.100.6 kubernetus <none>
kube-system kube-scheduler-kubernetus 1/1 Running 1 57m 172.16.100.6 kubernetus <none>
coredns is in a pending state for a very long time. I have removed docker and kubectl, kubeadm, kubelet a no of times & tried to recreate the cluster, but every time it shows the same output. Can anybody help me with this issue?
Try to install Pod network add-on (Base on this guide).
Run this line:
kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml
Unable to update cni config: No networks found in /etc/cni/net.d .....
Oct 02 19:21:32 kubernetus kubelet[19007]: E1002 19:21:32.886170 19007
kubelet.go:2167] Container runtime network not ready:
NetworkReady=false reason:NetworkPluginNotReady message:docker:
network plugin is not ready: cni config uninitialized
According to this error, you forgot to initialize a Kubernetes Pod network add-on. Looking at your settings, I suppose it should be Flannel.
Here is the instruction from the official Kubernetes documentation:
For flannel to work correctly, you must pass
--pod-network-cidr=10.244.0.0/16 to kubeadm init.
Set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running
sysctl net.bridge.bridge-nf-call-iptables=1 to pass bridged IPv4
traffic to iptables’ chains. This is a requirement for some CNI
plugins to work, for more information please see here.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
Note that flannel works on amd64, arm, arm64 and ppc64le, but until
flannel v0.11.0 is released you need to use the following manifest
that supports all the architectures:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/kube-flannel.yml
For more information, you can visit this link.
For the Kubernetes cluster to be available, the cluster should have a Container Networking Interface (CNI). A pod-network is required to be configured for the dns pod to be functional.
Install any of the CNI Providers like:
- Flannel
- Calico
- Canal
- WeaveNet, etc.,
Without this, the hosted Kubernetes cluster would have the master in the NotReady State.
Check if docker and kubernetes are using the same cgroup driver.
I faced the same issue (CentOS 7, kubernetes v1.14.1), and setting same cgroup driver (systemd) fixed it.
I installed kubernetes with 1 master + 1 work-node.
After I made kubeadm init ..., I faced two issues:
On the master node, the coredns were pending.
On the work-node, kubectl command didn't work out
On the work-node, I did the following and fixed the both issues:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/kubelet.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config**
For me, I've restarted the system and re-applied calico.yaml, coredns and calico pods started creating.
take this solution at least priority and try changing instance type (preferably higher cpu core/ram)
in my case i have changed linux instance t3.micro to t2.medium and its works

vswitch not found hyperV while running minikube start

I had created a Virtual switch with the name "Minikube2" . Previously I had created Virtual switch with the name "minikube" ,but deleted it later as there was config issue.
Did all the required configuration -"sharing on ethernet .."
Now when I try to run
minikube start --kubernetes-version="v1.10.3" --vm-driver="hyperv" --hyperv-virtual-switch="minikube2"
it downloads the ISO , but fails to configure the switch -
it says vswitch "minikube2" not found
Short answer is to delete C:\Users\%USERNAME%\.minikube and try again.
Below is my investigation:
First I have created Virtual Switch "minikube", started the cluster and it worked as expected.
Then I stopped minikube, created new "Minikube2" switch and started minikube
minikube start --kubernetes-version="v1.10.3" --vm-driver="hyperv" --hyperv-virtual-switch="minikube2" --v=9
Appeared issue:
Starting local Kubernetes v1.10.3 cluster... Starting VM... [executing
==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state [stdout =====>] : Off
[stderr =====>] : [executing ==>] :
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile
-NonInteractive Hyper-V\Start-VM minikube [stdout =====>] : [stderr =====>] : Hyper-V\Start-VM : 'minikube' failed to start. Synthetic Ethernet Port (Instance ID AF9D08DC-2625-4F24-93E5-E09BAD904899):
Error 'Insufficient system resources exist to complete the requested
service.'. Failed to allocate resources while connecting to a virtual
network. The Ethernet switch may not exist. 'minikube' failed to
start. (Virtual machine ID 863D6558-78EC-4648-B712-C1FDFC907588)
'minikube' Synthetic Ethernet Port: Failed to finish reserving
resources with Error 'Insufficient system resources exist to complete
the requested service.' (0x800705AA). (Virtual machine ID
863D6558-78EC-4648-B712-C1FDFC907588) 'minikube' failed to allocate
resources while connecting to a virtual network: Insufficient system
resources exist to complete the requested service. (0x800705AA)
(Virtual Machine ID 863D6558-78EC-4648-B712-C1FDFC907588). The
Ethernet switch may not exist. Could not find Ethernet switch
'minikube'. At line:1 char:1
+ Hyper-V\Start-VM minikube
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-VM], VirtualizationException
+ FullyQualifiedErrorId : Unspecified,Microsoft.HyperV.PowerShell.Commands.StartVM
E1022 12:50:43.384867 6216 start.go:168] Error starting host: Error
starting stopped host: exit status 1.
Retrying. E1022 12:50:43.398832 6216 start.go:174] Error starting
host: Error starting stopped host: exit status 1 PS
C:\Windows\system32>
Then I deleted C:\Users\%USERNAME%\.minikube , minikube vm inside Hyper-V and started again:
C:\Windows\system32> minikube start --kubernetes-version="v1.10.3" --vm-driver="hyperv" --hyperv-virtual-switch="minikube2" --v=9
Result:
Starting local Kubernetes v1.10.3 cluster... Starting VM...
Downloading Minikube ISO
170.78 MB / 170.78 MB [============================================] 100.00% 0s
Creating CA: C:\Users\Vitalii.minikube\certs\ca.pem
Creating client certificate: C:\Users\Vitalii.minikube\certs\cert.pem
----- [stderr =====>] : Using switch "Minikube2"
----- Moving files into cluster...
Downloading kubeadm v1.10.3
Downloading kubelet v1.10.3 Finished
Downloading kubeadm v1.10.3 Finished
Finished Downloading kubelet v1.10.3
Setting up certs... Connecting to
cluster... Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
PS C:\Windows\system32> kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-c4cffd6dc-cjzsm 1/1 Running 0 1m
kube-system etcd-minikube 1/1 Running 0 56s
kube-system kube-addon-manager-minikube 1/1 Running 0 13s
kube-system kube-apiserver-minikube 1/1 Running 0 41s
kube-system kube-controller-manager-minikube 1/1 Running 0 1m
kube-system kube-dns-86f4d74b45-w62rv 2/3 Running 0 1m
kube-system kube-proxy-psgss 1/1 Running 0 1m
kube-system kube-scheduler-minikube 1/1 Running 0 21s
kube-system kubernetes-dashboard-6f4cfc5d87-jz266 1/1 Running 0 1m
kube-system storage-provisioner 1/1 Running 0 1m
It looks like HyperV driver has some problems running Minikube. For me when trying to run it on Windows it was much simpler to use the docker driver. Just run:
minikube start --driver=docker
and it works without problems. Make sure your docker daemon is running e.g. with Docker Desktop. For reference about drivers look here: https://kubernetes.io/docs/setup/learning-environment/minikube/#specifying-the-vm-driver

Resources