I have configured to use docker on Centos 6.5. Quite often the containers keep losing internet. In such instances I have to restart Docker on the host. Recently, I tried to run an yum update from inside a container which failed. Following is the log from /var/log/docker
2014/07/15 10:33:36 GET /v1.12/containers/update_test/json
[b601ba8c] +job container_inspect(update_test)
[b601ba8c] -job container_inspect(update_test) = OK (0)
2014/07/15 10:33:36 POST /v1.12/containers/update_test/attach?stderr=1&stdin=1&stdout=1&stream=1
[b601ba8c] +job container_inspect(update_test)
2014/07/15 10:33:36 POST /v1.12/containers/update_test/start
[b601ba8c] +job start(update_test)
[b601ba8c] -job container_inspect(update_test) = OK (0)
[b601ba8c] +job attach(update_test)
[b601ba8c] +job allocate_interface(5a5c0247441ef5872b531ba720ba1f7d8af2df1cbd47b4a98b84a7b995384d8b)
[b601ba8c] -job allocate_interface(5a5c0247441ef5872b531ba720ba1f7d8af2df1cbd47b4a98b84a7b995384d8b) = OK (0)
[b601ba8c] -job start(update_test) = OK (0)
2014/07/15 10:33:36 POST /v1.12/containers/update_test/resize?h=37&w=165
[b601ba8c] +job resize(update_test, 37, 165)
[b601ba8c] -job resize(update_test, 37, 165) = OK (0)
[b601ba8c] +job release_interface(5a5c0247441ef5872b531ba720ba1f7d8af2df1cbd47b4a98b84a7b995384d8b)
[b601ba8c] -job release_interface(5a5c0247441ef5872b531ba720ba1f7d8af2df1cbd47b4a98b84a7b995384d8b) = OK (0)
[error] container.go:492 5a5c0247441ef5872b531ba720ba1f7d8af2df1cbd47b4a98b84a7b995384d8b: Error closing terminal: invalid argument
[b601ba8c] -job attach(update_test) = OK (0)
As mentioned above restarting Docker on the host solves the issue. I don't want to keep restarting the docker as I am planning to run a production application through docker. Anybody have any idea in this regard?
Please let me know if you need more information in this regard.
It is my bad that I did not mention that the host was hosted in the Rackspace. My apologies for not clarifying that (at that time I thought that it was irrelevant). It was the Rackspace's automated routine that kept messing up the iptables which obviously affected the docker routing. Rackspace did suggest to create a lock file somewhere in the /etc to prevent their automated routine touching the iptables which I have forgotten now; should not be difficult for anybody to get this from them if they experience the issue.
As suggested by Maduraiveeran, to prevent Rackspace from overwriting Docker's custom iptables rules you need to create a lock file in /etc. The file should be named rackconnect-allow-custom-iptables:
touch /etc/rackconnect-allow-custom-iptables
Once the lockfile has been created, restart your docker daemon with: sudo systemctl restart docker
https://support.rackspace.com/how-to/how-to-prevent-rackconnect-from-overwriting-custom-iptables-rules-on-linux-cloud-servers/
If you don't set a DNS at the host /etc/resolv.conf , then you can face some internet issues; If you still have this issue, consider using the --dns 209.244.0.3 at your docker run settings; the solution might look like this:
docker run -d --dns 209.244.0.3 centos webapp.sh
Another useful param you would maybe require at some point in this matter is --add-host which adds a host to the containers' /etc/hosts file
Related
There are many question around this topic, but not the specific info I am after.
Host OS is Mac, and recently had to uninstall Docker Desktop due to their licensing change. So instead we have moved to minikube, and it is all working great with VirtualBox driver.
But ideally we would like to use the hyperkit driver, as it requires less resources than virtualbox, and is (anecdotally) faster. This also all works great until we connect to our VPN (using cisco anyconnect) and then all outbound networking from within the minikube VM stops working. e.g.
k8> minikube ssh "traceroute 8.8.8.8"
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 46 byte packets
1 host.minikube.internal (192.168.64.1) 0.154 ms 0.181 ms 0.151 ms
2 * * *
Everything else is is fine, inbound networking via ingress is all good. And maven-docker-plugin is happily creating images with the minikube docker daemon. Just nothing outbound.
So figured I'd try to work with VPNKit as I have read it is meant to address this issue. But cannot find a lot of detailed documentation, and so am struggling.
We have tried starting VPNKit with minimal config:
vpnkit --ethernet /tmp/vpkit-ethernet.socket --debug
And then attempt to start minikube, but it fails:
k8> minikube delete
🔥 Deleting "minikube" in hyperkit ...
💀 Removed all traces of the "minikube" cluster.
k8> minikube start --driver=hyperkit --hyperkit-vpnkit-sock=/tmp/vpnkit-ethernet.socket
😄 minikube v1.25.1 on Darwin 10.15.7
✨ Using the hyperkit driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating hyperkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔥 Deleting "minikube" in hyperkit ...
🤦 StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: hyperkit crashed! command line:
hyperkit loglevel=3 console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes random.trust_cpu=on hw_rng_model=virtio base host=minikube
🔥 Creating hyperkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
😿 Failed to start hyperkit VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: hyperkit crashed! command line:
hyperkit loglevel=3 console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes random.trust_cpu=on hw_rng_model=virtio base host=minikube
❌ Exiting due to PR_HYPERKIT_CRASHED: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: hyperkit crashed! command line:
hyperkit loglevel=3 console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes random.trust_cpu=on hw_rng_model=virtio base host=minikube
💡 Suggestion: Hyperkit is broken. Upgrade to the latest hyperkit version and/or Docker for Desktop. Alternatively, you may choose an alternate --driver
🍿 Related issues:
▪ https://github.com/kubernetes/minikube/issues/6079
▪ https://github.com/kubernetes/minikube/issues/5780
And in the vpnkit log we see:
time="2022-02-14T06:07:57Z" level=debug msg="usernet: accepted vmnet connection"
time="2022-02-14T06:07:57Z" level=warning msg="Uwt: Pipe.listen: rejected ethernet connection: EOF"
time="2022-02-14T06:08:07Z" level=debug msg="usernet: accepted vmnet connection"
time="2022-02-14T06:08:07Z" level=warning msg="Uwt: Pipe.listen: rejected ethernet connection: EOF"
So kind of implies something is not right with how I started vpnkit. Have played with IP args to ensure it all matches, but does not help.
My guess is that the --ethernet=path arg is not the right type of socket. I have seen there is also --vsock-path=path but specifying this does not appear to create the socket file like --ethernet=path does. Do I have to create this some other way?
Or are there other config options I need to mess with. e.g. I thought --gateway-forwards=path could help, but can find no documentation on file format or contents.
So, I guess two main questions:
Is what we are trying even possible? Is it the the right way to go about it? Or is it much more complicated than simply running the vpnkit command?
If we are on the right track, does anyone have experience with this, and know how to set up the socket for minikube+vpnkit+hyperkit? What args, config, or other setup is required?
And just to note: --hyperkit-vpnkit-sock=auto is not an option for us, as we do not have docker installed, and so the docker socket file does not exist.
And just in case its a version issue:
k8> minikube version
minikube version: v1.25.1
commit: 3e64b11ed75e56e4898ea85f96b2e4af0301f43d
k8> vpnkit --version
854498c13b1884d4a48d84f3569eb34681af2126
k8> hyperkit -v
hyperkit: 0.20200908
Homepage: https://github.com/docker/hyperkit
License: BSD
My environment:
CentOS7 linux
/etc/hosts:
192.168.0.106 master01
192.168.0.107 node02
192.168.0.108 node01
On master01 machine:
/etc/hostname:
master01
On master01 machine I execute commands as follows:
1)yum install docker-ce kubelet kubeadm kubectl
2)systemctl start docker.service
3)vim /etc/sysconfig/kubelet
EDIT the file:
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
4)systemctl enable docker kubelet
5)kubeadm init --kubernetes-version=v1.12.0 --pod-network-cidr=10.244.0.0/16 servicecidr=10.96.0.0/12 --ignore-preflight-errors=all
THEN
The first error message:
unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
kubelet.go:2236] node "master01" not found
kubelet_node_status.go:70] Attempting to register node master01
Oct 2 23:32:35 master01 kubelet: E1002 23:32:35.974275 49157
kubelet_node_status.go:92] Unable to register node "master01" with API server: Post https://192.168.0.106:6443/api/v1/nodes: dial tcp 192.168.0.106:6443: connect: connection refused
l don't know why node master01 not found?
l have tried a lot of ways but can't solve the problem.
Thank you!
Your issue also might be caused by firewall rules, restricting tcp connection to 6443 port.
So you can temporary disable firewall on master node to validate this:
systemctl stop firewalld
and then try to perform kubeadm init once again.
Hope it helps.
I'm not sure that the problem is always related to firewall rules.
Consider the steps below:
1 ) Try to resolve your error related to: /etc/kubernetes/pki/ca.crt.
Run kubeadm init again without the --ignore-preflight-errors=all and see if all actions in the [certs] phase took place - for example:
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
.
.
2 ) Try running re-installing kubeadm or running kubeadm init with other version (use the --kubernetes-version=X.Y.Z flag).
3 ) Try restarting kubelet with systemctl restart kubelet.
4 ) Check if the docker version need to be updated.
5 ) Open specific ports which are needed (if you decide to stop firewalld remember to start it right after you resolve the issue).
I'm trying to use Docker Swarm, to do that I need to start the Docker daemon with the -H flag on each node using this command:
docker -H tcp://0.0.0.0:2375 -d
When doing this on my node (Debian 8, Docker 1.6.0) the command never stops, even if it displays that the daemon has completed initialization.
The complete output:
INFO[0000] +job init_networkdriver()
INFO[0000] +job serveapi(tcp://0.0.0.0:2375)
INFO[0000] Listening for HTTP on tcp (0.0.0.0:2375)
INFO[0000] /!\ DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING /!\
INFO[0000] -job init_networkdriver() = OK (0)
WARN[0000] mountpoint for memory not found
INFO[0000] Loading containers: start.
INFO[0000] Loading containers: done.
INFO[0000] docker daemon: 1.6.0 4749651; execdriver: native-0.2; graphdriver: aufs
INFO[0000] +job acceptconnections()
INFO[0000] -job acceptconnections() = OK (0)
INFO[0000] Daemon has completed initialization
After this last line nothing happens and I'm not able to write another command.
I also ran the command using screen to be able to run a command after the first one but I have a error message when running a Docker command:
FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
This command clearly states that the daemon didn't start correctly. How could I have a Docker daemon that starts and ensures that remote API on Swarm Agents is available over TCP for the Swarm Manager?
This commands states the client cannot talk to the docker daemon/engine/server. According to logs, your server is running.
With only the -H tcp://0.0.0.0:2375, if you didn't export DOCKER_HOST=127.0.0.1:2375, the docker client won't be able to talk to the daemon. You have 2 ways to handle this :
Exporting DOCKER_HOST
# Exporting DOCKER_HOST when you want to talk to it
$ export DOCKER_HOST=127.0.0.1:2375
$ docker ps
Or update your server options to also bind to the socket, like this
# docker -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock -d
$ docker ps
I am working on a fresh VM delivered by Host Europe that matches the description on
https://docs.docker.com/installation/ubuntulinux/#ubuntu-trusty-1404-lts-64-bit
(so Ubuntu Trusty 14.04 (LTS) (64-bit), 3.13.0 Linux kernel).
After installing the docker.io package docker ps fails with
"Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
When running docker -d I get:
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] +job init_networkdriver()
inappropriate ioctl for device
INFO[0000] -job init_networkdriver() = ERR (1)
FATA[0000] inappropriate ioctl for device
Apparently this error happens as well when the docker service tries to start via upstart.
I also tried it with the latest docker package according to "Docker-maintained Package Installation" in the above-mentioned description.
Here is the more detailed ouptput using docker -D -d:
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
DEBU[0000] libdevmapper(3): ioctl/libdm-iface.c:363 (-1) /dev/mapper/control: open failed: Operation not permitted
DEBU[0000] libdevmapper(3): ioctl/libdm-iface.c:415 (-1) Failure to communicate with kernel device-mapper driver.
DEBU[0000] libdevmapper(3): ioctl/libdm-iface.c:417 (-1) Check that device-mapper is available in the kernel.
DEBU[0000] Using graph driver vfs
DEBU[0000] Creating images graph
DEBU[0000] Restored 0 elements
DEBU[0000] Creating repository list
INFO[0000] +job init_networkdriver()
DEBU[0000] Creating bridge docker0 with network 172.17.42.1/16
DEBU[0000] setting bridge mac address = true
inappropriate ioctl for device
INFO[0000] -job init_networkdriver() = ERR (1)
FATA[0000] inappropriate ioctl for device
Ideas anybody? Thanks in advance. (Seems like a "deadend" to me after lots of successfull "dockerizing" on local VMs.)
Most probably your hoster doesn't provide cgroups. That happens sometimes depending on the kind virtualization they use.
I have the same problem with www.stratro.de
Thats when cat /proc/cgroups returns an empty table.
You can see more here: https://mannlinstones.wordpress.com/2014/08/12/docker-v-server-strato-final-results/
Did you checked the runtime dependencies from Docker -> Check runtime dependencies? It is defentliy a problem with your filesystem maybe it is related to this problem.
From Docker:
a properly mounted cgroupfs hierarchy (having a single, all-encompassing "cgroup" mount point is not sufficient)
I'm trying to get my head around something that's been working on a Centos+Vagrant, but not on our providers RHEL (Red Hat Enterprise Linux Server release 6.5 (Santiago)). A sudo service docker restart hands this:
Stopping docker: [ OK ]
Starting cgconfig service: Error: cannot mount cpuset to /cgroup/cpuset: Device or resource busy
/sbin/cgconfigparser; error loading /etc/cgconfig.conf: Cgroup mounting failed
Failed to parse /etc/cgconfig.conf [FAILED]
Starting docker: [ OK ]
The service starts okey enough, but images cannot run. A mounting failed error is shown when I try. And the startup-log also gives a warning or two. Regarding the kernelwarning, centos gives the same and has no problems as Epel should resolve this:
WARNING: You are running linux kernel version 2.6.32-431.17.1.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0.
2014/08/07 08:58:29 docker daemon: 1.1.2 d84a070; execdriver: native; graphdriver:
[1233d0af] +job serveapi(unix:///var/run/docker.sock)
[1233d0af] +job initserver()
[1233d0af.initserver()] Creating server
2014/08/07 08:58:29 Listening for HTTP on unix (/var/run/docker.sock)
[1233d0af] +job init_networkdriver()
[1233d0af] -job init_networkdriver() = OK (0)
2014/08/07 08:58:29 WARNING: mountpoint not found
Anyone had any success overcoming this problem or should I throw in the towel and wait for the provider to update to RHEL 7?
I have the same issue.
(1) check cgconfig status
# /etc/init.d/cgconfig status
if it stopped, restart it
# /etc/init.d/cgconfig restart
check cgconfig is running
(2) check cgconfig is on
# chkconfig --list cgconfig
cgconfig 0:off 1:off 2:off 3:off 4:off 5:off 6:off
if cgconfig is off, turn it on
(3) if still does not work, may be some cgroups modules is missing. In the kernel .config file, make menuconfig, add those modules into kernel and recompile and reboot
after that, it should be OK
I ended up asking the same question at Google Groups and in the end finding a solution with some help. What worked for me was this:
umount cgroup
sudo service cgconfig start
The project of making Docker work was put on halt all the same. Later a problem of network connection for the containers. This took to much time to solve and had to give up.
So I spent the whole day trying to rig docker to work on my vps. I was running into this same error. Basically what it came down to was the fact that OpenVZ didn't support docker containers up until a couple months ago. Specifically this RHEL update:
https://openvz.org/Download/kernel/rhel6/042stab105.14
Assuming this is your problem, or some variation of it, the burden of solving it is on your host. They will need to follow these steps:
https://openvz.org/Docker_inside_CT
In my case
/etc/rc.d/rc.cgconfig start
was generating
Starting cgconfig service: Error: cannot mount cpu,cpuacct,memory to
/cgroup/cpu_and_mem: Device or resource busy /usr/sbin/cgconfigparser;
error loading /etc/cgconfig.conf: Cgroup mounting failed Failed to
parse /etc/cgconfig.conf
i had to use:
/etc/rc.d/rc.cgconfig restart
and it automagicly umouted and mounted groups
Stopping cgconfig service: Starting cgconfig service:
it seems like the cgconfig service not running,so check it!
# /etc/init.d/cgconfig status
# mkdir -p /cgroup/cpuacct /cgroup/memory /cgroup/devices /cgroup/freezer net_cls /cgroup/blkio
# cat /etc/cgconfig.conf |tail|grep "="|awk '{print "mount -t cgroup -o",$1,$1,$NF}'>cgroup_mount.sh
# sh ./cgroup_mount.sh
# /etc/init.d/cgconfig restart
# /etc/init.d/docker restart
This situation occurs when the kernel is booted with cgroup_disable=memory and /etc/cgconfig.conf contains memory = /cgroup/memory;
This causes only /cgroup/cpuset to be mounted instead of the full set.
Solution: either remove cgroup_disable=memory from your kernel boot options or comment out memory = /cgroup/memory; from cgconfig.conf.
The cgconfig service startup uses mount and umount which requires an extra privilege bump from docker.
See the --privileged=true flag here for more info.
I was able to overcome this issue by starting my container with:
docker run -it --privileged=true my-image.
Tested in Centos6, Centos6.5.