To try out LXC on CentOS 6.4 host, I followed the steps in http://wiki.centos.org/HowTos/LXC-on-CentOS6, but it fails with ERROR: internal error 'cpuacct', 'devices' & 'memory' cgroups controllers must be mounted, when install vm using virt-install.
You need to run /etc/init.d/cgconfig daemon to mount the cgroups controllers.
In my case I had to do
cgroup_dirlist=('devices' 'memory' 'cpuacct' 'cpuset' 'cpu')
for d in "${cgroup_dirlist[#]}"
do
if [ -e /dev/cgroup/$d ]
then
mount -t cgroup cgroup -o $d /dev/cgroup/$d
fi
done
which was buried in my /etc/init.c cgroup init file - but also I had
'cgroup_disable=memory,blkio'
enabled in my /proc/cmdline - so that had to be removed in grub config also
hth
Related
In a docker container I want to run k8s.
When I run kubeadm join ... or kubeadm init commands I see sometimes errors like
\"modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could
not open moddep file
'/lib/modules/3.10.0-1062.1.2.el7.x86_64/modules.dep.bin'.
nmodprobe:
FATAL: Module configs not found in directory
/lib/modules/3.10.0-1062.1.2.el7.x86_64",
err: exit status 1
because (I think) my container does not have the expected kernel header files.
I realise that the container reports its kernel based on the host that is running the container; and looking at k8s code I see
// getKernelConfigReader search kernel config file in a predefined list. Once the kernel config
// file is found it will read the configurations into a byte buffer and return. If the kernel
// config file is not found, it will try to load kernel config module and retry again.
func (k *KernelValidator) getKernelConfigReader() (io.Reader, error) {
possibePaths := []string{
"/proc/config.gz",
"/boot/config-" + k.kernelRelease,
"/usr/src/linux-" + k.kernelRelease + "/.config",
"/usr/src/linux/.config",
}
so I am bit confused what is simplest way to run k8s inside a container such that it consistently past this getting the kernel info.
I note that running docker run -it solita/centos-systemd:7 /bin/bash on a macOS host I see :
# uname -r
4.9.184-linuxkit
# ls -l /proc/config.gz
-r--r--r-- 1 root root 23834 Nov 20 16:40 /proc/config.gz
but running exact same on a Ubuntu VM I see :
# uname -r
4.4.0-142-generic
# ls -l /proc/config.gz
ls: cannot access /proc/config.gz
[Weirdly I don't see this FATAL: Module configs not found in directory error every time, but I guess that is a separate question!]
UPDATE 22/November/2019. I see now that k8s DOES run okay in a container. Real problem was weird/misleading logs. I have added an answer to clarify.
I do not believe that is possible given the nature of containers.
You should instead test your app in a docker container then deploy that image to k8s either in the cloud or locally using minikube.
Another solution is to run it under kind which uses docker driver instead of VirtualBox
https://kind.sigs.k8s.io/docs/user/quick-start/
It seems the FATAL error part was a bit misleading.
It was badly formatted by my test environment (all on one line.
When k8s was failing I saw the FATAL and assumed (incorrectly) that was root cause.
When I format the logs nicely I see ...
kubeadm join 172.17.0.2:6443 --token 21e8ab.1e1666a25fd37338 --discovery-token-unsafe-skip-ca-verification --experimental-control-plane --ignore-preflight-errors=all --node-name 172.17.0.3
[preflight] Running pre-flight checks
[WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.4.0-142-generic
DOCKER_VERSION: 18.09.3
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.3. Latest validated version: 18.06
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-142-generic/modules.dep.bin'\nmodprobe: FATAL: Module configs not found in directory /lib/modules/4.4.0-142-generic\n", err: exit status 1
[discovery] Trying to connect to API Server "172.17.0.2:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.17.0.2:6443"
[discovery] Failed to request cluster info, will try again: [the server was unable to return a response in the time allotted, but may still be processing the request (get configmaps cluster-info)]
There are other errors later, which I originally though were a side-effect of the nasty looking FATAL error e.g. .... "[util/etcd] Attempt timed out"]} but I now think root cause is Etcd part times out sometimes.
Adding this answer in case someone else puzzled like I was.
What is the equivalent command for minikube delete in docker-for-desktop on OSX
As I understand, minikube creates a VM to host its kubernetes cluster but I do not understand how docker-for-desktop is managing this on OSX.
Tear down Kubernetes in Docker for OS X is quite an easy task.
Go to Preferences, open Reset tab, and click Reset Kubernetes cluster.
All object that have been created with Kubectl before that will be deleted.
You can also reset docker VM image (Reset disk image) and all settings (Reset to factory defaults) or even uninstall Docker.
In recent Docker Edge versions for Mac ( 2.1.7 ) Preferences design has been changed. Now you can reset Kubernetes cluster and other docker aspects by switching to the bug plane in the top right of Preferences window:
Note: You are able to reset Kubernetes cluster only if it's enabled. If you uncheck "Enable Kubernetes" checkbox, "Reset Kubernetes cluster" button becomes inactive.
For convenience "Reset Kubernetes cluster" is also present on the Kubernetes tab in the main Preferences plane:
To reset Docker-desktop Kubernetes cluster using command line, put the following content to a file (dd-reset.sh) and mark it executable ( chmod a+x dd-reset.sh )
#!/bin/bash
dr='docker run -it --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i'
${dr} sh -c 'export PATH=$PATH:/containers/services/docker/rootfs/usr/bin:/containers/services/docker/rootfs/usr/local/bin:/var/lib/kube-binary-cache/ && \
if [ ! -e /var/run/docker.sock ] ; then ln -s /containers/services/docker/rootfs/var/run/docker.sock /var/run/docker.sock ; fi && \
kube-reset.sh'
sleep 3
echo "cluster resetted. restarting docker-desktop..."
osascript -e 'quit app "Docker"'
open --background -a Docker
echo "docker-desktop started. Wait 3-5 mins for kubernetes to start."
Explanation:
This method uses internal scripts from Docker-desktop VM. To make it work, some preparation of user environment is required.
I wasn't able to start Kubernetes cluster using kube-start.sh script from inside the VM, so I've used MacOS commands to restart Docker application instead.
This method works even if your Kubernetes cluster is not enabled in Docker preferences at the moment, but it's required to enable Kubernetes at least once to use the script.
It was tested on Docker Edge for MacOS v2.2.2.0 (43066)
There is no guarantee that it will be compatible with earlier or later versions.
This version of Docker uses kubeadm to initialize Kubernetes cluster. Scripts are located in the folder /containers/services/docker/rootfs/usr/bin:
kube-pull.sh (brings kubernetes binaries to VM)
kube-reset.sh (runs kube-stop.sh and do kubeadm reset + some rm stuff)
kube-restart.sh (runs kube-stop.sh and kube-start.sh)
kube-start.sh (runs kube-pull.sh and kubelet.sh)
kube-stop.sh (kills kubelet and kube-apiserver processes, and all k8s containers)
kubeadm-init.sh (initializes Kubernetes cluster)
kubelet.sh (runs kubeadm-init.sh and starts kubelet binary)
Cluster configuration is located in the file /containers/services/docker/lower/etc/kubeadm/kubeadm.yaml
Resources used:
Restart Docker from command line
Use nsenter in priviledged container
It's really under the hood in the code. Docker for Mac uses these components: Hyperkit, VPNkit and DataKit
Kubernetes runs in the same Hyperkit VM created for docker and the kube-apiserver is exposed.
You can connect to the VM with this:
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
Then you can see all the Kubernetes processes in the VM:
linuxkit-025000000001:~# ps -Af | grep kube
1251 root 0:00 /usr/bin/logwrite -n kubelet /usr/bin/kubelet.sh
1288 root 0:51 kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --cgroups-per-qos=false --enforce-node-allocatable= --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cadvisor-port=0 --kube-reserved-cgroup=podruntime --system-reserved-cgroup=systemreserved --cgroup-root=kubepods --hostname-override=docker-for-desktop --fail-swap-on=false
3564 root 0:26 kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf
3616 root 1:45 kube-controller-manager --cluster-signing-key-file=/run/config/pki/ca.key --address=127.0.0.1 --root-ca-file=/run/config/pki/ca.crt --service-account-private-key-file=/run/config/pki/sa.key --kubeconfig=/etc/kubernetes/controller-manager.conf --cluster-signing-cert-file=/run/config/pki/ca.crt --leader-elect=true --use-service-account-credentials=true --controllers=*,bootstrapsigner,tokencleaner
3644 root 1:59 kube-apiserver --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --service-account-key-file=/run/config/pki/sa.pub --secure-port=6443 --insecure-port=8080 --insecure-bind-address=0.0.0.0 --requestheader-client-ca-file=/run/config/pki/front-proxy-ca.crt --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra- --advertise-address=192.168.65.3 --service-cluster-ip-range=10.96.0.0/12 --tls-private-key-file=/run/config/pki/apiserver.key --enable-bootstrap-token-auth=true --requestheader-allowed-names=front-proxy-client --tls-cert-file=/run/config/pki/apiserver.crt --proxy-client-key-file=/run/config/pki/front-proxy-client.key --proxy-client-cert-file=/run/config/pki/front-proxy-client.crt --allow-privileged=true --client-ca-file=/run/config/pki/ca.crt --kubelet-client-certificate=/run/config/pki/apiserver-kubelet-client.crt --kubelet-client-key=/run/config/pki/apiserver-kubelet-client.key --authorization-mode=Node,RBAC --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/run/config/pki/etcd/ca.crt --etcd-certfile=/run/config/pki/apiserver-etcd-client.crt --etcd-keyfile=/run/config/pki/apiserver-etcd-client.key
3966 root 0:01 /kube-dns --domain=cluster.local. --dns-port=10053 --config-dir=/kube-dns-config --v=2
4190 root 0:05 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf
4216 65534 0:03 /sidecar --v=2 --logtostderr --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,SRV --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,SRV
4606 root 0:00 /compose-controller --kubeconfig --reconciliation-interval 30s
4905 root 0:01 /api-server --kubeconfig --authentication-kubeconfig --authorization-kubeconfig --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/etc/docker-compose/etcd/ca.crt --etcd-certfile=/etc/docker-compose/etcd/client.crt --etcd-keyfile=/etc/docker-compose/etcd/client.key --secure-port=9443 --tls-ca-file=/etc/docker-compose/tls/ca.crt --tls-cert-file=/etc/docker-compose/tls/server.crt --tls-private-key-file=/etc/docker-compose/tls/server.key
So if you uncheck the following box (unclear from the docs what command it uses):
You can see that the processes are removed:
linuxkit-025000000001:~# [ 6616.856404] cni0: port 2(veth5f6c8b28) entered disabled state
[ 6616.860520] device veth5f6c8b28 left promiscuous mode
[ 6616.861125] cni0: port 2(veth5f6c8b28) entered disabled state
linuxkit-025000000001:~#
linuxkit-025000000001:~# [ 6626.816763] cni0: port 1(veth87e77142) entered disabled state
[ 6626.822748] device veth87e77142 left promiscuous mode
[ 6626.823329] cni0: port 1(veth87e77142) entered disabled state
linuxkit-025000000001:~# ps -Af | grep kube
linuxkit-025000000001:~#
On docker desktop version 3.5.2 (engine version 20.10.7), the reset button has been moved inside the docker preferences.
You can get there by following the below steps:
Click on the docker icon in the menu bar and choose 'Preferences'.
Go to the Kubernetes tab.
Click on the Reset Kubernetes CLuster button. This is the red color button.
This will delete all pods and reset the kubernetes. You can execute the docker ps command at terminal to verify that there are no containers running.
Just delete the vm that holds the kubernetes resources.
$ minikube delete
When running docker run hello-world I get an "Incorrect Usage" error (full output pasted below). I'm running the following:
Docker 17.05.0-ce, build 89658be
docker-containerd 0.2.3 (commit 9048e5e)
runc v1.0.0-rc4
Linux kernel 4.1.15
Using buildroot 2017.11 (commit 1f1a242) to generate custom toolchain/rootfs
systemd 234
Seems as though I can pull the hello-world image down properly, as it is included in docker images output. Wondering if there is an incompatibility between docker/containerd/runc? Or maybe something obvious? First time working with docker.
Additionally, I've run a docker check-config.sh script I found that states the only kernel configuration features I'm missing are optional. They are CONFIG_CGROUP_PIDS, CONFIG_CGROUP_HUGETLB, CONFIG_AUFS_FS, /dev/zfs, zfs command, and zpool command. Everything else, including all required, are enabled.
Output:
# docker run hello-world
[ 429.332968] device vethc0d83d1 entered promiscuous mode
[ 429.359681] IPv6: ADDRCONF(NETDEV_UP): vethc0d83d1: link is not ready
Incorrect Usage.
NAME:
docker-runc create - create a container
USAGE:
docker-runc create [command options] <container-id>
Where "<container-id>" is your name for the instance of the container that you
are starting. The name you provide for the container instance must be unique on
your host.
DESCRIPTION:
The create command creates an instance of a container for a bundle. The bundle
is a directory with a specification file named "config.json" and a root
filesystem.
The specification file includes an args parameter. The args parameter is used
to specify command(s) that get run when the container is started. To change the
command(s) that get executed on start, edit the args parameter of the spec. See
"runc spec --help" for more explanation.
OPTIONS:
--bundle value, -b value path to the root of the bundle directory, defaults to the current directory
--console-socket value path to an AF_UNIX socket which will receive a file descriptor referencing the master end of the console's pseudoterminal
--pid-file value specify the file to write the process id to
--no-pivot do not use pivot root to jail process inside rootfs. This should be used whenever the rootfs is on top of a ramdisk
--no-new-keyring do not create a new session keyring for the container. This will cause the container to inherit the calling processes session key
--preserve-fds value Pass N additional file descriptors to the container (stdio + $LISTEN_FDS + N in total) (default: 0)
flag provided but not defined: -console
[ 429.832198] docker0: port 1(vethc0d83d1) entered disabled state
[ 429.849301] device vethc0d83d1 left promiscuous mode
[ 429.859317] docker0: port 1(vethc0d83d1) entered disabled state
docker: Error response from daemon: oci runtime error: flag provided but not defined: -console.
The -console option was replaced with --console-socket in runc Dec 2016 for v1.0.0-rc4.
So I would guess you need an older version of runc or a newer version of Docker.
If you are building Docker yourself, use Docker 17.09.0-ce or an older release of runc. I'm not sure if that's v0.1.1 or just an earlier 1.0 like v1.0.0-rc2
If you were upgrading packages, something has gone wrong with the install. Probably purge everything and reinstall Docker.
I have a custom boot2docker with this configuration attached to the .iso
config.vm.provider "virtualbox" do |v|
v.customize ['storageattach', :id, '--storagectl', 'SATA', '--port', 0, '--device', 0, '--type', 'dvddrive', '--medium', File.expand_path("../boot2docker.iso", __FILE__)]
v.customize ['modifyvm', :id, '--nictype1', 'virtio']
end
config.vm.network "private_network", ip: "192.168.10.10", id: "default-network", nic_type: "virtio"
My files in the directory
mycompany/
dockerhost/
Vagrantfile
Vagrantfile
The vagrantfile of the docker-host that replaces the above.
## This is required with the plugin winnfsd
config.vm.network "private_network", type: "dhcp"
config.vm.synced_folder "../", "/vagrant", type: "nfs"
When I start up the dockerhost with vagrant up it works well and the NFS is setup correctly, but when I start a service container vagrant up myservice
shows the error
==> myservice: Docker host is required. One will be created if necessary...
myservice: Docker host VM is already ready.
==> myservice: Syncing folders to the host VM...
dockerhost: Mounting shared folders...
dockerhost: /var/lib/docker/docker_1472079332_51007 => C:/Users/myuser/Desktop/mycompany
Vagrant was unable to mount VirtualBox shared folders. This is usually because the filesystem "vboxsf" is not available. This filesystem is made available via the VirtualBox Guest Additions and kernel module.
Please verify that these guest additions are properly installed in the guest. This is not a bug in Vagrant and is usually caused by a faulty
Vagrant box. For context, the command attemped was:
set -e
mount -t vboxsf -o uid=`id -u docker`,gid=`getent group docker | cut -d: -f3` b5973a5087 /var/lib/docker/docker_1472079332_51007 mount -t vboxsf -o uid=`id -u docker`,gid=`id -g docker` b5973a5087 /var/lib/docker/docker_1472079332_51007
The error output from the command was:
mount: mounting b5973a5087 on /var/lib/docker/docker_1472079332_51007 failed: Protocol error
It looks like Vagrant mount another volumen when starts a container.
Any idea about how to fix it?
or why vagrant mounts another for my?
Tks
Vagrant and VirtualBox version incompatibility problems
This fix my problem:
Vagrant 1.8.5 should be compatible with the latest VirtualBox 5.1.2 release, however VirtualBox Guest Additions version we currently have is 5.0.20 and the latest boot2docker release is at 5.0.24. Both will most likely have issues with VirtualBox 5.1.2.
So for now avoid VirtualBox 5.1.x and stick with:
Vagrant 1.7.4 - 1.8.4
VirtualBox 5.0.x
Reference:
https://github.com/blinkreaction/boot2docker-vagrant/issues/83
I'm trying to get my head around something that's been working on a Centos+Vagrant, but not on our providers RHEL (Red Hat Enterprise Linux Server release 6.5 (Santiago)). A sudo service docker restart hands this:
Stopping docker: [ OK ]
Starting cgconfig service: Error: cannot mount cpuset to /cgroup/cpuset: Device or resource busy
/sbin/cgconfigparser; error loading /etc/cgconfig.conf: Cgroup mounting failed
Failed to parse /etc/cgconfig.conf [FAILED]
Starting docker: [ OK ]
The service starts okey enough, but images cannot run. A mounting failed error is shown when I try. And the startup-log also gives a warning or two. Regarding the kernelwarning, centos gives the same and has no problems as Epel should resolve this:
WARNING: You are running linux kernel version 2.6.32-431.17.1.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0.
2014/08/07 08:58:29 docker daemon: 1.1.2 d84a070; execdriver: native; graphdriver:
[1233d0af] +job serveapi(unix:///var/run/docker.sock)
[1233d0af] +job initserver()
[1233d0af.initserver()] Creating server
2014/08/07 08:58:29 Listening for HTTP on unix (/var/run/docker.sock)
[1233d0af] +job init_networkdriver()
[1233d0af] -job init_networkdriver() = OK (0)
2014/08/07 08:58:29 WARNING: mountpoint not found
Anyone had any success overcoming this problem or should I throw in the towel and wait for the provider to update to RHEL 7?
I have the same issue.
(1) check cgconfig status
# /etc/init.d/cgconfig status
if it stopped, restart it
# /etc/init.d/cgconfig restart
check cgconfig is running
(2) check cgconfig is on
# chkconfig --list cgconfig
cgconfig 0:off 1:off 2:off 3:off 4:off 5:off 6:off
if cgconfig is off, turn it on
(3) if still does not work, may be some cgroups modules is missing. In the kernel .config file, make menuconfig, add those modules into kernel and recompile and reboot
after that, it should be OK
I ended up asking the same question at Google Groups and in the end finding a solution with some help. What worked for me was this:
umount cgroup
sudo service cgconfig start
The project of making Docker work was put on halt all the same. Later a problem of network connection for the containers. This took to much time to solve and had to give up.
So I spent the whole day trying to rig docker to work on my vps. I was running into this same error. Basically what it came down to was the fact that OpenVZ didn't support docker containers up until a couple months ago. Specifically this RHEL update:
https://openvz.org/Download/kernel/rhel6/042stab105.14
Assuming this is your problem, or some variation of it, the burden of solving it is on your host. They will need to follow these steps:
https://openvz.org/Docker_inside_CT
In my case
/etc/rc.d/rc.cgconfig start
was generating
Starting cgconfig service: Error: cannot mount cpu,cpuacct,memory to
/cgroup/cpu_and_mem: Device or resource busy /usr/sbin/cgconfigparser;
error loading /etc/cgconfig.conf: Cgroup mounting failed Failed to
parse /etc/cgconfig.conf
i had to use:
/etc/rc.d/rc.cgconfig restart
and it automagicly umouted and mounted groups
Stopping cgconfig service: Starting cgconfig service:
it seems like the cgconfig service not running,so check it!
# /etc/init.d/cgconfig status
# mkdir -p /cgroup/cpuacct /cgroup/memory /cgroup/devices /cgroup/freezer net_cls /cgroup/blkio
# cat /etc/cgconfig.conf |tail|grep "="|awk '{print "mount -t cgroup -o",$1,$1,$NF}'>cgroup_mount.sh
# sh ./cgroup_mount.sh
# /etc/init.d/cgconfig restart
# /etc/init.d/docker restart
This situation occurs when the kernel is booted with cgroup_disable=memory and /etc/cgconfig.conf contains memory = /cgroup/memory;
This causes only /cgroup/cpuset to be mounted instead of the full set.
Solution: either remove cgroup_disable=memory from your kernel boot options or comment out memory = /cgroup/memory; from cgconfig.conf.
The cgconfig service startup uses mount and umount which requires an extra privilege bump from docker.
See the --privileged=true flag here for more info.
I was able to overcome this issue by starting my container with:
docker run -it --privileged=true my-image.
Tested in Centos6, Centos6.5.