I am trying to follow docs to setup a one node Kubernetes cluster with Centos 7.
kubeadm init will return no error but kubectl get nodes will return:
NAME STATUS AGE VERSION
[MY_IP] NotReady 22s v1.6.4
system log repeats:
Jun 6 16:21:48 localhost kubelet: W0606 16:21:48.064388 11520 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 6 16:21:48 localhost kubelet: E0606 16:21:48.064537 11520 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
I can only find info about this in Kubernetes github logs but they talk about a bug and I haven't found a workaround. Thanks
you can run this command
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Looks like you need a pod network. Have you completed step 3 in the guide here? If you install one of the network overlays (listed at https://kubernetes.io/docs/concepts/cluster-administration/addons/), you should be good to go.
Related
So my new task at work involves setting up a Kubernetes cluster with Windows nodes for GitLab CI/CD on a .NET Framework project. I’ve never done anything with Kubernetes, so this week has been quite a journey, and now I can’t manage to get my cluster to run the GitLab agent and connect to the GitLab instance.
My guess is that it’s because the “Adding windows nodes” guide told me to use Flannel, but GitLab Agent is trying to use cni net.d? Is this the problem? Is there a way to fix this, or maybe a simpler way to do all of this?
Thanks in advance.
I’ve managed to set up a control-plane node on Ubuntu 20.04, add some Ubuntu and Windows worker nodes, but I got stuck with setting up the GitLab Agent (connecting the cluster and GitLab instance). I’ve used this tutorial for setting up the cluster, except I changed the Flannel install step with the patched version from this documentation page. The latter page is what I used to add the Windows nodes to the cluster.
So I’m trying to install the GitLab K8s agent using this page. The GitLab test server is a Docker Omnibus install on a separate server with Ubuntu. I’m trying to use the one-liner method since I don’t understand the advanced methods, but after I run the one-liner my GitLab Agent pod gets stuck in an ImagePullBackOff.
Here’s the looped output of “journalctl -xeu kubelet” on the control node:
`Mar 31 12:30:37 k8s-ctrl-ubu1 kubelet[4594]: I0331 12:30:37.208235 4594 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 31 12:30:40 k8s-ctrl-ubu1 kubelet[4594]: E0331 12:30:40.052911 4594 kubelet.go:2347] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 31 12:30:42 k8s-ctrl-ubu1 kubelet[4594]: I0331 12:30:42.208568 4594 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 31 12:30:45 k8s-ctrl-ubu1 kubelet[4594]: E0331 12:30:45.061648 4594 kubelet.go:2347] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 31 12:30:47 k8s-ctrl-ubu1 kubelet[4594]: I0331 12:30:47.208848 4594 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 31 12:30:50 k8s-ctrl-ubu1 kubelet[4594]: E0331 12:30:50.070691 4594 kubelet.go:2347] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 31 12:30:52 k8s-ctrl-ubu1 kubelet[4594]: I0331 12:30:52.209618 4594 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"`
Here’s the output of “kubectl describe pod” on the gitlab-kubernetes-agent pod:
`Events:
Type Reason Age From Message
Normal Scheduled 5m28s default-scheduler Successfully assigned gitlab-kubernetes-agent/gitlab-agent-6bb8c4f4d5-wc64z to k8s-win5
Normal Pulling 4m1s (x4 over 5m27s) kubelet Pulling image "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable"
Warning Failed 4m (x4 over 5m25s) kubelet Error: ErrImagePull
Normal BackOff 20s (x19 over 5m24s) kubelet Back-off pulling image "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable"`
I struggle to initialize a simple Kubernetes cluster using Ubuntu on Virtualbox. I tried both server and desktop version, following the official documentation:
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
I also tried to follow some other ones, thinking the issue was because i'm using Virtualbox VM's, like this one:
https://medium.com/#gunjangarge/create-kubernetes-cluster-using-kubeadm-on-ubuntu-virtualbox-step-by-step-68a3eeb1f74c
But everytime I have the same issue with port 6443 not being exposed. Sometimes the process starts correctly, giving me the join command:
kubeadm init --pod-network-cidr=192.168.0.0/16
W1029 08:47:53.841460 11540 configset.go:348] WARNING: kubeadm cannot validate component configs
for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.1.192:6443 --token ztnoww.t8ng5a3jo2kx5cb2 \
--discovery-token-ca-cert-hash
sha256:907dde6cc6d72ed4cd7fe7e7f252e2cf657dd3256fba6ee5ec92027132a9c5af
Sometimes it's not starting at all and timeouting:
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
Anyway, even when it's starting, port 6443 is never exposed, and kubelet is not happy with it:
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Thu 2020-10-29 08:48:15 CET; 20s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 13262 (kubelet)
Tasks: 14 (limit: 4666)
CGroup: /system.slice/kubelet.service
└─13262 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-contai
Okt 29 08:48:22 master kubelet[13262]: E1029 08:48:22.588386 13262 controller.go:136] failed to ensure node lease exists, will retry in 800ms, error: Get
"https://192.168.1.192:6443/apis/coordination.k8s.io/v1/names
Okt 29 08:48:22 master kubelet[13262]: E1029 08:48:22.785951 13262 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.1.192:644
Okt 29 08:48:23 master kubelet[13262]: I1029 08:48:23.022354 13262 kubelet_node_status.go:70] Attempting to register node master
Okt 29 08:48:24 master kubelet[13262]: I1029 08:48:24.188510 13262 request.go:645] Throttling request took 1.097264312s, request: POST:https://192.168.1.192:6443/api/v1/namespaces/kube-system/pods
Okt 29 08:48:25 master kubelet[13262]: I1029 08:48:25.678880 13262 kubelet_node_status.go:108] Node master was previously registered
Okt 29 08:48:25 master kubelet[13262]: I1029 08:48:25.679004 13262 kubelet_node_status.go:73] Successfully registered node master
Okt 29 08:48:25 master kubelet[13262]: W1029 08:48:25.765981 13262 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
Okt 29 08:48:27 master kubelet[13262]: E1029 08:48:27.148246 13262 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: c
Okt 29 08:48:30 master kubelet[13262]: W1029 08:48:30.767511 13262 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
Okt 29 08:48:32 master kubelet[13262]: E1029 08:48:32.164211 13262 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: c
I have to say I don't know what to do now. I tried for hours with different Ubuntu versions, trying to find solutions on the Internet but I didn't found any solution. I also went trough the logs and found that maybe the config file is not created correctly for any reason:
failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml
but I found nothing about it, except "try to init the cluster again", which I did several times..
Thank you in advance for your help!
OK, I think I finally found the problem. I tried the same process on another PC and everything worked smoothly, so for anyway of you having a similar issue, just don't try to use VirtualBox and WSL at the same time (even if wsl is shut off)
I just did what's explained here: https://stackoverflow.com/a/63229718/2428805 and now everything's fine...
I try to run a sample application in my Ubuntu 18 vm.
I have installed Docker client and server version of 18.06.1-ce. I already have VirtualBox running.
I use below link and install kubectl 1.14 too: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux
I have Minikube v1.0.1 also installed. But Minikube start command stuck in Waiting for pods: apiserver and timeout
harshana#-Virtual-Machine:~$ sudo minikube start
😄 minikube v1.0.1 on linux (amd64)
🤹 Downloading Kubernetes v1.14.1 images in the background ...
⚠️ Ignoring --vm-driver=virtualbox, as the existing "minikube" VM was created using the none driver.
⚠️ To switch drivers, you may create a new VM using `minikube start -p <name> --vm-driver=virtualbox`
⚠️ Alternatively, you may delete the existing VM using `minikube delete -p minikube`
🔄 Restarting existing none VM for "minikube" ...
⌛ Waiting for SSH access ...
📶 "minikube" IP address is xxx.xxx.x.xxx
🐳 Configuring Docker as the container runtime ...
🐳 Version of container runtime is 18.06.1-ce
⌛ Waiting for image downloads to complete ...
✨ Preparing Kubernetes environment ...
💾 Downloading kubeadm v1.14.1
💾 Downloading kubelet v1.14.1
🚜 Pulling images required by Kubernetes v1.14.1 ...
🔄 Relaunching Kubernetes v1.14.1 using kubeadm ...
⌛ Waiting for pods: apiserver
sudo minikube logs:
May 19 08:11:40 harshana-Virtual-Machine kubelet[10572]: E0519 08:11:40.825465 10572 kubelet.go:2244] node "minikube" not found
May 19 08:11:40 harshana-Virtual-Machine kubelet[10572]: E0519 08:11:40.895848 10572 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)minikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
I got the same behaviour because I have created a first VM using kvm. I have followed the instructions and deleted the VM. Run the below :
1- minikube delete -p minikube
2- minikube start
I am following all the steps from this link : https://github.com/justmeandopensource/kubernetes
after running the join command in the worker node it's getting added to master, but the status of the worker node is getting changed to ready.
From the logs I got the following :
Container runtime network not ready: NetworkReady=false
reason:NetworkPluginNotReady message:dock
Unable to update cni config: No networks found in /etc/cni/net.d
kubelet.go:2266 -- node "XXXXXXXXX" not found. (xxxxx is the masters
host/node name)
To establish CNI I am using flannel and also tried with weave and many other
CNI networks but the results are the same
points to ponder:
---> worker node kubelet status is healthy
---> trying to run kubeadm init command in the worker node,its showing the status of kubelet might be unhealthy. (Not able to make worker node master by running the kubeadm init command but kubeadm join command is working.After joining kubectl get nodes is showing the worker node but status is notready)
Thank you for the help
I cannot reproduce your issue. I followed exactly the instructions on github`s site you shared, and did not face similar error.
The only extra steps I needed to do, to suppress errors, detected by pre-flight checks of kubeadm init:
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
was to set appropriate flag by running:
echo '1' > /proc/sys/net/ipv4/ip_forward
State of my cluster nodes:
NAME STATUS ROLES AGE VERSION
centos-master Ready master 18h v1.13.1
centos-worker Ready <none> 18h v1.13.1
I verified cluster condition by deploying&exposing sample application and everything seems to be working fine:
kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node
kubectl expose deployment hello-node --port=8080
I`m getting valid response from hello-world node.js app:
curl 10.100.113.255:8080
Hello World!#
What IP address you have put to your /etc/hosts files ?
KubeletNotReady
runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady
message:docker: network plugin is not ready: cni config uninitialized
I don't know how to make the network plugin ready
While you run kubectl describe node <node_name>
In the Conditions table, the Ready type will contain this message if you did not initialized cni. Proper initialization can be obtained by installing network addon. I will point you to 2 most used: Weave and Flannel
1) Weave
$ export kubever=$(kubectl version | base64 | tr -d '\n')
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"
After executing those two commands you should see node in status "Ready"
$ kubectl get nodes
You could also check status
$ kubectl get cs
2) Flannel
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml
3) Kubernetes documentation will explain how install other network addons. In this article each CNI provider have a short description.
In my case, update systemd from 30.el7_3.9 to 67.el7_7.4 solved this.