Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
During the installation of kubernetes, an error is reported when I initialize the master node. I am using the arm platform server and the operating system is centos-7.6 aarch64. Does kubernetes support deploying master nodes on the arm platform?
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
6月 30 22:53:04 master kubelet[54238]: W0630 22:53:04.188966 54238 pod_container_deletor.go:75] Container "51615bc1d926dcc56606bca9f452c178398bc08c78a2418a346209df28b95854" not found in pod's containers
6月 30 22:53:04 master kubelet[54238]: E0630 22:53:04.189353 54238 kubelet.go:2248] node "master" not found
6月 30 22:53:04 master kubelet[54238]: I0630 22:53:04.218672 54238 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
6月 30 22:53:04 master kubelet[54238]: E0630 22:53:04.236484 54238 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://192.168.1.112:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 192.168.1.112:6443: connect: connection refused
6月 30 22:53:04 master kubelet[54238]: E0630 22:53:04.238898 54238 certificate_manager.go:400] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Post https://192.168.1.112:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.1.112:6443: connect: connection refused
6月 30 22:53:04 master kubelet[54238]: I0630 22:53:04.260520 54238 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
6月 30 22:53:04 master kubelet[54238]: E0630 22:53:04.289516 54238 kubelet.go:2248] node "master" not found
6月 30 22:53:04 master kubelet[54238]: E0630 22:53:04.389666 54238 kubelet.go:2248] node "master" not found
6月 30 22:53:04 master kubelet[54238]: E0630 22:53:04.436810 54238 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://192.168.1.112:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.112:6443: connect: connection refused
6月 30 22:53:04 master kubelet[54238]: E0630 22:53:04.489847 54238 kubelet.go:2248] node "master" not found
To start kubernetes cluster, make sure you have minimum requirement of kubernetes platfrom.
If you want kubernetes cluster with low compute you could discus with me in seperatly.
You need :
Docker
Compute Node at least 4GB Memory 2CPU.
I will write answer depends on your node.
Docker
On each of your machines, install Docker. Version 19.03.11 is recommended, but 1.13.1, 17.03, 17.06, 17.09, 18.06 and 18.09 are known to work as well. Keep track of the latest verified Docker version in the Kubernetes release notes.
Use the following commands to install Docker on your system:
Install required packages
yum install -y yum-utils device-mapper-persistent-data lvm2
Add the Docker repository
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Install Docker CE
yum update -y && yum install -y \
containerd.io-1.2.13 \
docker-ce-19.03.11 \
docker-ce-cli-19.03.11
Create /etc/docker
mkdir /etc/docker
Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
Restart Docker
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
Kubernetes
As a requirement for your Linux Node's iptables to correctly see bridged traffic, you should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config, e.g.
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
Make sure that the br_netfilter module is loaded before this step. This can be done by running lsmod | grep br_netfilter. To load it explicitly call sudo modprobe br_netfilter.
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
systemctl daemon-reload
systemctl restart kubelet
Initializing your control-plane node
The control-plane node is the machine where the control plane components run, including etcd (the cluster database) and the API Server (which the kubectl command line tool communicates with).
Master
Init kubernetes cluster (Running this on master node)
kubeadm init --pod-network-cidr 192.168.0.0/16
Note : I will calico here. so the cidr use 192.168.0.0/16
Move kube config to user directory (assume root)
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Worker Node
Join other nodes (Running below command from your worker node)
kubeadm join <IP_PUBLIC>:6443 --token <TOKEN> \
--discovery-token-ca-cert-hash sha256:<HASH>
Note : you will get this when you successfully init master
Master Node
Applying calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
Verify cluster
kubectl get nodes
Reference : https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
Related
I intend to test a non-trivial Kubernetes setup as part of CI and wish to run the full system before CD. I cannot run --privileged containers and am running the docker container as a sibling to the host using docker run -v /var/run/docker.sock:/var/run/docker.sock
The basic docker setup seems to be working on the container:
linuxbrew#03091f71a10b:~$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
However, minikube fails to start inside the docker container, reporting connection issues:
linuxbrew#03091f71a10b:~$ minikube start --alsologtostderr -v=7
I1029 15:07:41.274378 2183 out.go:298] Setting OutFile to fd 1 ...
I1029 15:07:41.274538 2183 out.go:345] TERM=xterm,COLORTERM=, which probably does not support color
...
...
...
I1029 15:20:27.040213 197 main.go:130] libmachine: Using SSH client type: native
I1029 15:20:27.040541 197 main.go:130] libmachine: &{{{<nil> 0 [] [] []} docker [0x7a1e20] 0x7a4f00 <nil> [] 0s} 127.0.0.1 49350 <nil> <nil>}
I1029 15:20:27.040593 197 main.go:130] libmachine: About to run SSH command:
sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname
I1029 15:20:27.040992 197 main.go:130] libmachine: Error dialing TCP: dial tcp 127.0.0.1:49350: connect: connection refused
This is despite the network being linked and the port being properly forwarded:
linuxbrew#51fbce78731e:~$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
93c35cec7e6f gcr.io/k8s-minikube/kicbase:v0.0.27 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes 127.0.0.1:49350->22/tcp, 127.0.0.1:49351->2376/tcp, 127.0.0.1:49348->5000/tcp, 127.0.0.1:49349->8443/tcp, 127.0.0.1:49347->32443/tcp minikube
51fbce78731e 7f7ba6fd30dd "/bin/bash" 8 minutes ago Up 8 minutes bpt-ci
linuxbrew#51fbce78731e:~$ docker network ls
NETWORK ID NAME DRIVER SCOPE
1e800987d562 bridge bridge local
aa6b2909aa87 host host local
d4db150f928b kind bridge local
a781cb9345f4 minikube bridge local
0a8c35a505fb none null local
linuxbrew#51fbce78731e:~$ docker network connect a781cb9345f4 93c35cec7e6f
Error response from daemon: endpoint with name minikube already exists in network minikube
The minikube container seems to be alive and well when trying to curl from the host and even sshis responding:
mastercook#linuxkitchen:~$ curl https://127.0.0.1:49350
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:49350
mastercook#linuxkitchen:~$ ssh root#127.0.0.1 -p 49350
The authenticity of host '[127.0.0.1]:49350 ([127.0.0.1]:49350)' can't be established.
ED25519 key fingerprint is SHA256:0E41lExrrezFK1QXULaGHgk9gMM7uCQpLbNPVQcR2Ec.
This key is not known by any other names
What am I missing and how can I make minikube properly discover the correctly working minikube container?
Because minikube does not complete the cluster creation, running Kubernetes in a (sibling) Docker container favours kind.
Given that the (sibling) container does not know enough about its setup, the networking connections are a bit flawed. Specifically, a loopback IP is selected by kind (and minikube) upon cluster creation even though the actual container sits on a different IP in the host docker.
To correct the networking, the (sibling) container needs to be connected to the network actually hosting the Kubernetes image. To accomplish this, the procedure is illustrated below:
Create a kubernetes cluster:
linuxbrew#324ba0f819d7:~$ kind create cluster --name acluster
Creating cluster "acluster" ...
✓ Ensuring node image (kindest/node:v1.21.1) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-acluster"
You can now use your cluster with:
kubectl cluster-info --context kind-acluster
Thanks for using kind! 😊
Verify if the cluster is accessible:
linuxbrew#324ba0f819d7:~$ kubectl cluster-info --context kind-acluster
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 127.0.0.1:36779 was refused - did you specify the right host or port?
3.) Since the cluster cannot be reached, retrieve the control planes master IP. Note the "-control-plane" addition to the cluster name:
linuxbrew#324ba0f819d7:~$ export MASTER_IP=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' acluster-control-plane)
4.) Update the kube config with the actual master IP:
linuxbrew#324ba0f819d7:~$ sed -i "s/^ server:.*/ server: https:\/\/$MASTER_IP:6443/" $HOME/.kube/config
5.) This IP is still not accessible by the (sibling) container and to connect the container with the correct network retrieve the docker network ID:
linuxbrew#324ba0f819d7:~$ export MASTER_NET=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.NetworkID}}{{end}}' acluster-control-plane)
6.) Finally connect the (sibling) container ID (which should be stored in the $HOSTNAME environment variable) with the cluster docker network:
linuxbrew#324ba0f819d7:~$ docker network connect $MASTER_NET $HOSTNAME
7.) Verify whether the control plane accessible after the changes:
linuxbrew#324ba0f819d7:~$ kubectl cluster-info --context kind-acluster
Kubernetes control plane is running at https://172.18.0.4:6443
CoreDNS is running at https://172.18.0.4:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
If kubectl returns Kubernetes control plane and CoreDNS URL, as shown in the last step above, the configuration has succeeded.
You can run minikube in docker in docker container. It will use docker driver.
docker run --name dind -d --privileged docker:20.10.17-dind
docker exec -it dind sh
/ # wget https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
/ # mv minikube-linux-amd64 minikube
/ # chmod +x minikube
/ # ./minikube start --force
...
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
/ # ./minikube kubectl -- run --image=hello-world
/ # ./minikube kubectl -- logs pod/hello
Hello from Docker!
...
Also, note that --force is for running minikube using docker driver as root which we shouldn't do according minikube instructions.
All kubernetes forums and articles ask to work with minikube that gives you only a single-node kubernetes cluster.
What options are available to work with multi node kubernetes cluster on a Windows environment?.
The problem is that Windows node may only act as a worker node.
You can only create a hybrid cluster and have Windows workloads running in Windows pods, talking to Linux workloads running in Linux pods.
Intro to Windows support in Kubernetes:
The Kubernetes control plane, including the master components, continues to run on Linux. There are no plans to have a Windows-only Kubernetes cluster.
Full list of limitations can be found in official docs
Control Plane limitations:
Windows is only supported as a worker node in the Kubernetes
architecture and component matrix. This means that a Kubernetes
cluster must always include Linux master nodes, zero or more Linux
worker nodes, and zero or more Windows worker nodes.
Resource management and process isolation:
Linux cgroups are used as a pod boundary for resource controls in
Linux. Containers are created within that boundary for network,
process and file system isolation. The cgroups APIs can be used to
gather cpu/io/memory stats. In contrast, Windows uses a Job object per
container with a system namespace filter to contain all processes in a
container and provide logical isolation from the host. There is no way
to run a Windows container without the namespace filtering in place.
This means that system privileges cannot be asserted in the context of
the host, and thus privileged containers are not available on Windows.
Containers cannot assume an identity from the host because the
Security Account Manager (SAM) is separate.
On my windows-10 laptop, I used virtualbox to create 2 ubuntu VMs (Each VM -> 3 GB RAM and 50 GB dynamically sized virtual disks). I used microk8s from https://microk8s.io. Very simple one line installation on each VM: sudo snap install microk8s --classic
Follow instructions at https://microk8s.io/docs/clustering .... one VM becomes the master k8s node and the other VM becomes the worker node joined to the master.
Once that is setup, you may want to setup alias like: alias k='microk8s.kubectl'. Then you can simply do: k apply -f
I was able to create a multi node kubernetes cluster on my windows box using Oracle virtual box.!
Hope this might help.
I created 4 * centos 8 VMs within Virtual Box hosted on Windows 10.
Among the 4* VMs, one VM is set up as master and the rest worker nodes.
Below is my step-by-step set up procedure.
Preparation
1.1 Preparation for basic VM template (node-master-centOS-1)
1.1.1 (Host) Download centOS 8 image (CentOS-8.1.1911-x86_64-dvd1.iso) from http://isoredirect.centos.org/centos/8/isos/x86_64/
1.1.2 Install Oracle VM Box from https://www.virtualbox.org/wiki/Downloads
1.2 Create and Configure a template VM (node-master-centOS-1) in VirtualBox
1.2.1 (VM Box) File->Host Network Manager -> Create a Host-only Ethernet Adapter with Manual address (e.g. 192.168.56.1/24, DHCP server # 192.168.56.100/24, DHCP range 101-254)
1.2.2 (VM Box) Pre-configure VM instance
1.2.2.1 (VM Box) System (Memory= 4096MB, Boot Order= Hard Disk -> Optical, Processor=2)
1.2.2.2 (VM Box) Storage (delete IDE controller; under SATA controller, add Optical Drive pointing to centOS-8.x.xxxx-arch-dvdx.iso downloaded at step 1.1.1)
1.2.2.3 (VM Box) Network (Adapter 1= Enable, attached to= NAT; Adapter 2 = Enable, attach to = Host-only Adapter, Name= VirtualBox Host-Only Ethernet Adapter.) Note the Adapter 2 created at step 1.2.1
1.2.2.4 (Host) Settings -> Firewall & network Protection -> Advanced Setting -> In-bound rules -> New Rule -> Custom -> All Programs -> Any port & protocol -> Local IP set as 192.168.56.1 (virtualbox host-only adapter)
-> remote IP set as a range from 192.168.56.2 - 192.168.56.99 (or as needed)
1.2.2.5 (Host) Settings -> Network and Internet -> Network Connections -> Properties for the adapter which has internet connection -> get working DNS address (e.g. 192.168.1.1)
1.2.2.6 Start VM instance
1.2.3 (Remote VM) Set up network
1.2.3.1 (Remote VM) Settings -> Network -> Ethernet (enp0s3): ipv4 (manual, 10.0.2.20/24, DNS 10.0.2.3)
1.2.3.2 (Remote VM) Settings -> Network -> Ethernet (enp0s8): ipv4 (manual, 192.168.56.20/24, DNS 192.168.1.1 or as obtained at step 1.2.2.5 so that remote VM inherits internet DNS of the host machine)
1.2.3.3 (Remote VM) Terminal -> sudo ifdown(then ifup) Profile_1 (or enp0s3) -> sudo ifdown(then ifup) Profile_2 (or enp0s8) -> systemctl restart network (if not working: systemctl restart NetworkManager.service)
1.2.4 (Remote VM) Set up hostname
1.2.4.1 (Remote VM) hostnamectl set-hostname node-master-centos-1 (i.e. {node_1})
1.2.5 Verify Connectivity
1.2.5.1 (Host) Ping: ping 192.168.56.20 (i.e. {ip_node_1}) success
1.2.5.2 (Host) SSH: ssh root#192.168.56.20 success -> (SSH) wget www.google.com success (indicates network and DNS is working. If DNS at steps 1.2.2.5 and 1.2.3.2 is not set up, DNS may not work although ip-based internet may be working well.
1.3 Prepare VM environment
1.3.1 Optional (Remote VM SSH)
-> yum install vim git wget bzh
-> sh -c "$(wget -O- https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" (ohmyzsh gives colored scheme to bash)
-> vi .zshrc -> change to ZSH_THEME = "bira" -> source .zshrc (this changes the bash color scheme)
1.3.4 Turn off selinux (Remote VM SSH)
-> setenforce 0
->
1.3.5 Install JDK 8
-> (Remote VM SSH): yum install java-1.8.0-openjdk-devel
-> (Remote VM SSH):
-> vi /etc/profile, add "export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64" and "export PATH=$JAVA_HOME/bin:$PATH"
-> source /etc/profile (to avoid duplicated path setting, better skip this step, if 1.3.6 is to be performed)
-> (Remote VM SSH): to verify, run javac -version; java -version; which javac; which java; echo $JAVA_HOME; echo $PATH;
1.3.6 Install Apache Maven
-> (Remote VM SSH):
-> cd /opt
-> wget https://www.strategylions.com.au/mirror/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
-> tar xzvf apache-maven-3.6.3-bin.tar.gz
-> vi /etc/profile
-> add "export PATH=/opt/apache-maven-3.6.3/bin:$PATH"
-> source /etc/profile (once is enough)
-> to verify, mvn -v
1.3.7 Install Python, Virtual Env, Tensorflow
-> (Remote VM SSH) Install Python3
-> yum update -y (update all installed packages)
-> yum install gcc openssl-devel bzip2-devel libffi-devel -y
-> verify python3: python3
-> (Remote VM SSH) Install VirtualEnv and Tensorflow
-> python3 -m venv --system-site-packages ./venv
-> source ./venv/bin/activate # sh, bash, or zsh
-> pip install --upgrade pip
-> pip install --upgrade requests bs4 numpy torch scipy (and so on)
-> pip install tenflow==1.15 (tf2.3.x does not work well on my platform)
1.3.8 Install Kubenetes and Docker (Remote VM SSH)
-> Turn off selinux
-> setenforce 0
-> sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config (sed -i "s/old text/new text/g" file)
-> Stop and Disable Firwall
-> systemctl stop firewalld
-> systemctl disable firewalld
-> Disable devices and files for paging and swapping
-> swapoff -a
-> yes | cp /etc/fstab /etc/fstab_bak ( create a bak file)
-> cat /etc/fstab_bak| grep -v swap > /etc/fstab (keep everything back the line with 'swap' to delete swap)
-> Re-configure network adaptor
-> enable br_netfilter
-> vi /etc/modules-load.d/k8s.conf
-> insert "br_netfilter"
-> modprobe br_netfilter
-> set sysctl settings
-> vi /etc/sysctl.d/k8s.conf
-> net.bridge.bridge-nf-call-ip6tables = 1
-> net.bridge.bridge-nf-call-iptables = 1
-> sysctl --system
-> Firwall (k8s use 6443, 2379-2380, 10250-10255 TCP which need to be enabled)
-> systemctl enable firewalld
-> systemctl start firewalld
-> firewall-cmd --permanent --add-port=6443/tcp
-> firewall-cmd --permanent --add-port=2379-2380/tcp
-> firewall-cmd --permanent --add-port=10250-10255/tcp
-> firewall-cmd –reload
-> Enable network modules
-> vi /etc/sysconfig/modules/ipvs.modules
-> insert
-> modprobe -- ip_vs
-> modprobe -- ip_vs_rr
-> modprobe -- ip_vs_wrr
-> modprobe -- ip_vs_sh
-> modprobe -- nf_conntrack_ipv4
-> modprobe -- ip_vs
-> modprobe -- ip_vs_rr
-> modprobe -- ip_vs_wrr
-> modprobe -- ip_vs_sh
-> modprobe -- nf_conntrack_ipv4
-> verify: cut -f1 -d " " /proc/modules | grep -e ip_vs -e nf_conntrack_ipv4 (shows 5 rows)
-> Install Kubenetes
-> Set up repository
-> vi /etc/yum.repos.d/kubernetes.repo, and insert:
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
-> Install K8s
-> yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
-> systemctl enable kubelet
-> systemctl start kubelet
-> systemctl status kubelet (error255)
-> journalctl -xe (missing yaml file /var/lib/kubelet/config.yaml which is expected. )
-> Install Docker
-> Set up repository
-> yum install -y yum-utils
-> yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
-> Install & Run Docker
-> yum install docker-ce docker-ce-cli containerd.io
-> systemctl enable docker
-> systemctl start docker
-> verify: docker run helloworld
-> verify: docker run -it ubuntu bash
-> Update Docker Cgroup
-> docker info | grep Cgroup (shows cgroup driver: cgroupfs. This needs updated to align with K8s)
-> vi /etc/docker/daemon.json, insert:
{
"exec-opts":["native.cgroupdriver=systemd"]
}
-> systemctl restart docker
-> verify: docker info | grep Cgroup
-> Install node.JS and npm
-> yum install epel-release (access to the EPEL repository)
-> yum install nodejs (it installs nodeJS and npm)
-> verify: node --version (v10.21.0)
-> verify: npm version (v6.14.4)
1.4 Create a cluster of 4 VMs by applying the basic template (node-worker-centOS-1,node-worker-centOS-2,node-worker-centOS-3)
-> (VM Box): Clone node-master-centOS-1 for three times, each with new MAC
-> (Remote VM): update enp0s3 with ipv4 = 10.0.2.21/22/23, respectively.
-> (Remote VM): update enp0s8 with ipv4 = 192.168.56.21/22/23, respectively.
-> (Remote VM): update hostname = node-worker-centos-1/2/3, respectively.
-> (Remote VM SSH): add host mapping (192.168.20.20/21/22/23 node-master/worker-centos-1/2/3) to /etc/hosts for all nodes.
1.5 Set up Kubernetes Cluster (1* Master, 3* Workers)
-> Init Master Node
-> (root#node-master-centos-1 ~) kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.56.20
pod-network-cide=10.244.0.0 is chosen due to the k8s flannel addon used which in its yaml specifies this ip block for pods.
below results are shown:
# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.56.20
[init] Using Kubernetes version: v1.20.0
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node-master-centos-1] and IPs [10.96.0.1 192.168.56.20]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node-master-centos-1] and IPs [192.168.56.20 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node-master-centos-1] and IPs [192.168.56.20 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 12.004852 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node-master-centos-1 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node node-master-centos-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: m5ohft.9xi6nyvgu73sxu68
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.56.20:6443 --token m5ohft.9xi6nyvgu73sxu68 \
--discovery-token-ca-cert-hash sha256:b04371eb9c969f27a0d8f39761e99b7fb88b33c4bf06ba2e0faa0c1c28ac3be0
-> (root#node-master-centos-1 ~) vi /etc/kubernetes/admin.conf, and edit to replace "192.168.56.20" to "node-master-centos-1" (use hostname instead of ip address)
-> (root#node-master-centos-1 ~) sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
-> (root#node-master-centos-1 ~) sudo chown $(id -u):$(id -g) $HOME/.kube/config
-> (root#node-master-centos-1 ~) kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-master-centos-1 NotReady control-plane,master 4m3s v1.20.0
-> root#node-master-centos-1 ~) kubeadm token create --print-join-command (to obtain the command to be run on workers)
-> By now, the k8s master is initialized which sets pod network to be 10.244.0.0/16 with api server at HTTPS://node-master-centos-1:6443.
At this stage, the node-master-centos-1 node is NotReady because Pod Network is not yet deployed which we need to use flannel.yaml (one of addons for podnetwork)
-> Join Worker Nodes
-> synchronize system time to avoid X509 certification error duruing kubeadm join. Below updates time offsets and adjust systime in one step.
-> (root#node-worker-centos-1/2/3 ~) chronyc -a 'burst 4/4'
-> (root#node-worker-centos-1/2/3 ~) chronyc -a makestep
-> join the worker to cluster
-> (root#node-worker-centos-1/2/3 ~) kubeadm join node-master-centos-1:6443 --token cjxoym.okfgvzd8t241grea --discovery-token-ca-cert-hash sha256:b04371eb9c969f27a0d8f39761e99b7fb88b33c4bf06ba2e0faa0c1c28ac3be0 --v=2
-> check node worker status on Master
-> (root#node-master-centos-1 ~) kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node-master-centos-1 Ready control-plane,master 4h12m v1.20.0 192.168.56.20 <none> CentOS Linux 8 4.18.0-147.el8.x86_64 docker://20.10.0
node-worker-centos-1 Ready <none> 162m v1.20.0 192.168.56.21 <none> CentOS Linux 8 4.18.0-147.el8.x86_64 docker://20.10.0
node-worker-centos-2 Ready <none> 142m v1.20.0 192.168.56.22 <none> CentOS Linux 8 4.18.0-147.el8.x86_64 docker://20.10.0
node-worker-centos-3 Ready <none> 4m41s v1.20.0 192.168.56.23 <none> CentOS Linux 8 4.18.0-147.el8.x86_64 docker://20.10.0
-> (root#node-master-centos-1 ~) kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-74ff55c5b-sfjvd 1/1 Running 0 112m
kube-system coredns-74ff55c5b-whjrs 1/1 Running 0 112m
kube-system etcd-node-master-centos-1 1/1 Running 0 112m
kube-system kube-apiserver-node-master-centos-1 1/1 Running 0 112m
kube-system kube-controller-manager-node-master-centos-1 1/1 Running 0 112m
kube-system kube-flannel-ds-dmqmw 1/1 Running 0 61m
kube-system kube-flannel-ds-hqwqt 1/1 Running 0 2m51s
kube-system kube-flannel-ds-qr9ml 1/1 Running 0 22m
kube-system kube-proxy-4dpk9 1/1 Running 0 22m
kube-system kube-proxy-6tltc 1/1 Running 0 2m51s
kube-system kube-proxy-t6k24 1/1 Running 0 112m
kube-system kube-scheduler-node-master-centos-1 1/1 Running 0 112m
By Now, the kubernetes cluster is set up. As the VMs are not always run, the differences of system time between VMs may cause X509 or other errors.
It may be therefore necessary to set up auto-sync scripts runnable on OS startup.
I try to run a sample application in my Ubuntu 18 vm.
I have installed Docker client and server version of 18.06.1-ce. I already have VirtualBox running.
I use below link and install kubectl 1.14 too: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux
I have Minikube v1.0.1 also installed. But Minikube start command stuck in Waiting for pods: apiserver and timeout
harshana#-Virtual-Machine:~$ sudo minikube start
😄 minikube v1.0.1 on linux (amd64)
🤹 Downloading Kubernetes v1.14.1 images in the background ...
⚠️ Ignoring --vm-driver=virtualbox, as the existing "minikube" VM was created using the none driver.
⚠️ To switch drivers, you may create a new VM using `minikube start -p <name> --vm-driver=virtualbox`
⚠️ Alternatively, you may delete the existing VM using `minikube delete -p minikube`
🔄 Restarting existing none VM for "minikube" ...
⌛ Waiting for SSH access ...
📶 "minikube" IP address is xxx.xxx.x.xxx
🐳 Configuring Docker as the container runtime ...
🐳 Version of container runtime is 18.06.1-ce
⌛ Waiting for image downloads to complete ...
✨ Preparing Kubernetes environment ...
💾 Downloading kubeadm v1.14.1
💾 Downloading kubelet v1.14.1
🚜 Pulling images required by Kubernetes v1.14.1 ...
🔄 Relaunching Kubernetes v1.14.1 using kubeadm ...
⌛ Waiting for pods: apiserver
sudo minikube logs:
May 19 08:11:40 harshana-Virtual-Machine kubelet[10572]: E0519 08:11:40.825465 10572 kubelet.go:2244] node "minikube" not found
May 19 08:11:40 harshana-Virtual-Machine kubelet[10572]: E0519 08:11:40.895848 10572 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)minikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
I got the same behaviour because I have created a first VM using kvm. I have followed the instructions and deleted the VM. Run the below :
1- minikube delete -p minikube
2- minikube start
What is the equivalent command for minikube delete in docker-for-desktop on OSX
As I understand, minikube creates a VM to host its kubernetes cluster but I do not understand how docker-for-desktop is managing this on OSX.
Tear down Kubernetes in Docker for OS X is quite an easy task.
Go to Preferences, open Reset tab, and click Reset Kubernetes cluster.
All object that have been created with Kubectl before that will be deleted.
You can also reset docker VM image (Reset disk image) and all settings (Reset to factory defaults) or even uninstall Docker.
In recent Docker Edge versions for Mac ( 2.1.7 ) Preferences design has been changed. Now you can reset Kubernetes cluster and other docker aspects by switching to the bug plane in the top right of Preferences window:
Note: You are able to reset Kubernetes cluster only if it's enabled. If you uncheck "Enable Kubernetes" checkbox, "Reset Kubernetes cluster" button becomes inactive.
For convenience "Reset Kubernetes cluster" is also present on the Kubernetes tab in the main Preferences plane:
To reset Docker-desktop Kubernetes cluster using command line, put the following content to a file (dd-reset.sh) and mark it executable ( chmod a+x dd-reset.sh )
#!/bin/bash
dr='docker run -it --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i'
${dr} sh -c 'export PATH=$PATH:/containers/services/docker/rootfs/usr/bin:/containers/services/docker/rootfs/usr/local/bin:/var/lib/kube-binary-cache/ && \
if [ ! -e /var/run/docker.sock ] ; then ln -s /containers/services/docker/rootfs/var/run/docker.sock /var/run/docker.sock ; fi && \
kube-reset.sh'
sleep 3
echo "cluster resetted. restarting docker-desktop..."
osascript -e 'quit app "Docker"'
open --background -a Docker
echo "docker-desktop started. Wait 3-5 mins for kubernetes to start."
Explanation:
This method uses internal scripts from Docker-desktop VM. To make it work, some preparation of user environment is required.
I wasn't able to start Kubernetes cluster using kube-start.sh script from inside the VM, so I've used MacOS commands to restart Docker application instead.
This method works even if your Kubernetes cluster is not enabled in Docker preferences at the moment, but it's required to enable Kubernetes at least once to use the script.
It was tested on Docker Edge for MacOS v2.2.2.0 (43066)
There is no guarantee that it will be compatible with earlier or later versions.
This version of Docker uses kubeadm to initialize Kubernetes cluster. Scripts are located in the folder /containers/services/docker/rootfs/usr/bin:
kube-pull.sh (brings kubernetes binaries to VM)
kube-reset.sh (runs kube-stop.sh and do kubeadm reset + some rm stuff)
kube-restart.sh (runs kube-stop.sh and kube-start.sh)
kube-start.sh (runs kube-pull.sh and kubelet.sh)
kube-stop.sh (kills kubelet and kube-apiserver processes, and all k8s containers)
kubeadm-init.sh (initializes Kubernetes cluster)
kubelet.sh (runs kubeadm-init.sh and starts kubelet binary)
Cluster configuration is located in the file /containers/services/docker/lower/etc/kubeadm/kubeadm.yaml
Resources used:
Restart Docker from command line
Use nsenter in priviledged container
It's really under the hood in the code. Docker for Mac uses these components: Hyperkit, VPNkit and DataKit
Kubernetes runs in the same Hyperkit VM created for docker and the kube-apiserver is exposed.
You can connect to the VM with this:
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
Then you can see all the Kubernetes processes in the VM:
linuxkit-025000000001:~# ps -Af | grep kube
1251 root 0:00 /usr/bin/logwrite -n kubelet /usr/bin/kubelet.sh
1288 root 0:51 kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --cgroups-per-qos=false --enforce-node-allocatable= --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cadvisor-port=0 --kube-reserved-cgroup=podruntime --system-reserved-cgroup=systemreserved --cgroup-root=kubepods --hostname-override=docker-for-desktop --fail-swap-on=false
3564 root 0:26 kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf
3616 root 1:45 kube-controller-manager --cluster-signing-key-file=/run/config/pki/ca.key --address=127.0.0.1 --root-ca-file=/run/config/pki/ca.crt --service-account-private-key-file=/run/config/pki/sa.key --kubeconfig=/etc/kubernetes/controller-manager.conf --cluster-signing-cert-file=/run/config/pki/ca.crt --leader-elect=true --use-service-account-credentials=true --controllers=*,bootstrapsigner,tokencleaner
3644 root 1:59 kube-apiserver --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --service-account-key-file=/run/config/pki/sa.pub --secure-port=6443 --insecure-port=8080 --insecure-bind-address=0.0.0.0 --requestheader-client-ca-file=/run/config/pki/front-proxy-ca.crt --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra- --advertise-address=192.168.65.3 --service-cluster-ip-range=10.96.0.0/12 --tls-private-key-file=/run/config/pki/apiserver.key --enable-bootstrap-token-auth=true --requestheader-allowed-names=front-proxy-client --tls-cert-file=/run/config/pki/apiserver.crt --proxy-client-key-file=/run/config/pki/front-proxy-client.key --proxy-client-cert-file=/run/config/pki/front-proxy-client.crt --allow-privileged=true --client-ca-file=/run/config/pki/ca.crt --kubelet-client-certificate=/run/config/pki/apiserver-kubelet-client.crt --kubelet-client-key=/run/config/pki/apiserver-kubelet-client.key --authorization-mode=Node,RBAC --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/run/config/pki/etcd/ca.crt --etcd-certfile=/run/config/pki/apiserver-etcd-client.crt --etcd-keyfile=/run/config/pki/apiserver-etcd-client.key
3966 root 0:01 /kube-dns --domain=cluster.local. --dns-port=10053 --config-dir=/kube-dns-config --v=2
4190 root 0:05 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf
4216 65534 0:03 /sidecar --v=2 --logtostderr --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,SRV --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,SRV
4606 root 0:00 /compose-controller --kubeconfig --reconciliation-interval 30s
4905 root 0:01 /api-server --kubeconfig --authentication-kubeconfig --authorization-kubeconfig --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/etc/docker-compose/etcd/ca.crt --etcd-certfile=/etc/docker-compose/etcd/client.crt --etcd-keyfile=/etc/docker-compose/etcd/client.key --secure-port=9443 --tls-ca-file=/etc/docker-compose/tls/ca.crt --tls-cert-file=/etc/docker-compose/tls/server.crt --tls-private-key-file=/etc/docker-compose/tls/server.key
So if you uncheck the following box (unclear from the docs what command it uses):
You can see that the processes are removed:
linuxkit-025000000001:~# [ 6616.856404] cni0: port 2(veth5f6c8b28) entered disabled state
[ 6616.860520] device veth5f6c8b28 left promiscuous mode
[ 6616.861125] cni0: port 2(veth5f6c8b28) entered disabled state
linuxkit-025000000001:~#
linuxkit-025000000001:~# [ 6626.816763] cni0: port 1(veth87e77142) entered disabled state
[ 6626.822748] device veth87e77142 left promiscuous mode
[ 6626.823329] cni0: port 1(veth87e77142) entered disabled state
linuxkit-025000000001:~# ps -Af | grep kube
linuxkit-025000000001:~#
On docker desktop version 3.5.2 (engine version 20.10.7), the reset button has been moved inside the docker preferences.
You can get there by following the below steps:
Click on the docker icon in the menu bar and choose 'Preferences'.
Go to the Kubernetes tab.
Click on the Reset Kubernetes CLuster button. This is the red color button.
This will delete all pods and reset the kubernetes. You can execute the docker ps command at terminal to verify that there are no containers running.
Just delete the vm that holds the kubernetes resources.
$ minikube delete
Followed official guide to install Kubernetes cluster with kubeadm on Vagrant.
https://kubernetes.io/docs/getting-started-guides/kubeadm/
master
node1
node2
Master
# kubeadm init --apiserver-advertise-address=192.168.33.200
# sudo cp /etc/kubernetes/admin.conf $HOME/
# sudo chown $(id -u):$(id -g) $HOME/admin.conf
# export KUBECONFIG=$HOME/admin.conf
# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# kubectl apply -f kube-flannel.yaml
Node1 and Node2
# kubeadm join --token <token> 192.168.33.200:6443
...
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
Node join complete:
* Certificate signing request sent to master and response
received.
* Kubelet informed of new secure connection details.
Run 'kubectl get nodes' on the master to see this machine join.
Until now all success.
But when check kubectl get nodes on master host, retunes only one node:
# kubectl get nodes
NAME STATUS AGE VERSION
localhost.localdomain Ready 25m v1.6.4
Sometimes, it retunes:
# kubectl get nodes
Unable to connect to the server: net/http: TLS handshake timeout
Edit
Add hostname to all the hosts.
Then check kubectl get nodes again from master:
[root#master ~]# kubectl get nodes
NAME STATUS AGE VERSION
localhost.localdomain Ready 4h v1.6.4
master Ready 12m v1.6.4
Just added a new current host name.