The /var/log/messages log is below
Aug 3 04:27:54 localhost lvm[3169]: Thin centos-docker--pool is now 100% full.
Aug 3 04:27:54 localhost lvm[3169]: Insufficient free space: 1 extents needed, but only 0 available
Aug 3 04:27:54 localhost lvm[3169]: Failed to extend thin centos-docker--pool.
And the systemctl status log is below:
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
Active: failed (Result: exit-code) since Mon 2015-08-03 04:22:52 UTC; 6min ago
Docs: http://docs.docker.com
Process: 7577 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
Main PID: 7577 (code=exited, status=1/FAILURE)
Aug 03 04:22:52 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Aug 03 04:22:52 localhost.localdomain docker[7577]: time="2015-08-03T04:22:52Z" level=info msg="+job serveapi(unix:///var/run/docker.sock)"
Aug 03 04:22:52 localhost.localdomain docker[7577]: time="2015-08-03T04:22:52Z" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
Aug 03 04:22:52 localhost.localdomain docker[7577]: time="2015-08-03T04:22:52Z" level=fatal msg="Shutting down daemon due to errors: error intializing graphdriver: Unable to take ownership of... data blocks"
Aug 03 04:22:52 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Aug 03 04:22:52 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Aug 03 04:22:52 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
And here is the lsblk result:
[root#localhost vagrant]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 9.9G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 9.4G 0 part
├─centos-root 253:0 0 8.4G 0 lvm /
├─centos-swap 253:1 0 1016M 0 lvm [SWAP]
├─centos-docker--pool_tmeta 253:2 0 12M 0 lvm
│ └─centos-docker--pool 253:4 0 16M 0 lvm
└─centos-docker--pool_tdata 253:3 0 16M 0 lvm
└─centos-docker--pool 253:4 0 16M 0 lvm
sr0 11:0 1 1024M 0 rom
sr1 11:1 1 1024M 0 rom
This problem maybe there is not enough lvm space to use. It was only to be allocated 16MB.
The solution is to resize the logic volume of centos-docker-pool
Related
After I install the docker, I have set the selinux status to Permissive, still can not run docker.
In my /etc/selinux/config, I have edit the SELINUX=disabled.
and setenforce 0, check with:
# getenforce
Permissive
I use systemctl start docker, but failed, get bellow error:
# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since 五 2018-06-29 09:05:47 CST; 14s ago
Docs: http://docs.docker.com
Process: 21615 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
Main PID: 21615 (code=exited, status=1/FAILURE)
6月 29 09:05:46 123.xyz systemd[1]: Starting Docker Application Container Engine...
6月 29 09:05:46 123.xyz dockerd-current[21615]: time="2018-06-29T09:05:46.451911058+08:00" level=warning msg="could not ch...found"
6月 29 09:05:46 123.xyz dockerd-current[21615]: time="2018-06-29T09:05:46.453472267+08:00" level=info msg="libcontainerd: ...21626"
6月 29 09:05:47 123.xyz dockerd-current[21615]: time="2018-06-29T09:05:47.463085812+08:00" level=warning msg="overlay2: the back...
6月 29 09:05:47 123.xyz dockerd-current[21615]: Error starting daemon: SELinux is not supported with the overlay2 graph dr...false)
6月 29 09:05:47 123.xyz systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
6月 29 09:05:47 123.xyz systemd[1]: Failed to start Docker Application Container Engine.
6月 29 09:05:47 123.xyz systemd[1]: Unit docker.service entered failed state.
6月 29 09:05:47 123.xyz systemd[1]: docker.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
Why there still says:
Error starting daemon: SELinux is not supported with the overlay2 graph dr...false)
My linux is CentOS 7.2
I find the solution.
in the /etc/sysconfig/docker:
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
if [ -z "${DOCKER_CERT_PATH}" ]; then
DOCKER_CERT_PATH=/etc/docker
fi
set the -selinux-enabled to --selinux-enabled=false.
I install docker on CentOS7(Linux version 3.10.0-327.el7.x86_64) with command yum install -y docker, but when I try to start docker with systemctl start docker, the docker failed to start, below is the error message
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2018-03-15 16:38:37 CST; 10s ago
Docs: http://docs.docker.com
Process: 5166 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
Main PID: 5166 (code=exited, status=1/FAILURE)
Mar 15 16:38:36 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Mar 15 16:38:36 localhost.localdomain dockerd-current[5166]: time="2018-03-15T16:38:36.570661801+08:00" level=info msg="libcontainerd... 5171"
Mar 15 16:38:37 localhost.localdomain dockerd-current[5166]: time="2018-03-15T16:38:37.585565695+08:00" level=warning msg="overlay2: the ba...
Mar 15 16:38:37 localhost.localdomain dockerd-current[5166]: Error starting daemon: SELinux is not supported with the overlay2 graph ...false)
Mar 15 16:38:37 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mar 15 16:38:37 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Mar 15 16:38:37 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
Mar 15 16:38:37 localhost.localdomain systemd[1]: docker.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
How to solve this issue?
I am running docker on a Redhat system with devicemapper and thinpool device just as recommended for production systems. Now when I want to reinstall docker I need two steps:
1) remove docker directory (in my case /area51/docker)
2) clear thinpool device
The docker documentation states that when using devicemapper with dm.metadev and dm.datadev options, the easiest way of cleaning devicemapper would be:
If setting up a new metadata pool it is required to be valid.
This can be achieved by zeroing the first 4k to indicate empty metadata, like this:
$ dd if=/dev/zero of=$metadata_dev bs=4096 count=1
Unfortunately, according to the documentation, the dm.metadatadev is deprecated, it says to use dm.thinpooldev instead.
My thinpool has been created along the lines of this docker instruction
So, my setup now looks like this:
cat /etc/docker/daemon.json
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.thinpooldev=/dev/mapper/thinpool_VG_38401-thinpool",
"dm.basesize=18G"
]
}
Under the devicemapper directory i see the following thinpool devices
ls -l /dev/mapper/thinpool_VG_38401-thinpool*
lrwxrwxrwx 1 root root 7 Dec 6 08:31 /dev/mapper/thinpool_VG_38401-thinpool -> ../dm-8
lrwxrwxrwx 1 root root 7 Dec 6 08:31 /dev/mapper/thinpool_VG_38401-thinpool_tdata -> ../dm-7
lrwxrwxrwx 1 root root 7 Dec 6 08:31 /dev/mapper/thinpool_VG_38401-thinpool_tmeta -> ../dm-6
So, after running docker successfully I tried to reinstall as described above and clear the thinpool by writing 4K zeroes into the tmeta device and restart docker:
dd if=/dev/zero of=/dev/mapper/thinpool_VG_38401-thinpool_tmeta bs=4096 count=1
systemctl start docker
And endet up with
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2017-12-06 10:28:46 UTC; 10s ago
Docs: https://docs.docker.com
Process: 1566 ExecStart=/usr/bin/dockerd -G uwsgi --data-root=/area51/docker -H unix:///var/run/docker.sock (code=exited, status=1/FAILURE)
Main PID: 1566 (code=exited, status=1/FAILURE)
Memory: 236.0K
CGroup: /system.slice/docker.service
Dec 06 10:28:45 yoda3 systemd[1]: Starting Docker Application Container Engine...
Dec 06 10:28:45 yoda3 dockerd[1566]: time="2017-12-06T10:28:45.816049000Z" level=info msg="libcontainerd: new containerd process, pid: 1577"
Dec 06 10:28:46 yoda3 dockerd[1566]: time="2017-12-06T10:28:46.816966000Z" level=warning msg="failed to rename /area51/docker/tmp for background deletion: renam...chronously"
Dec 06 10:28:46 yoda3 dockerd[1566]: Error starting daemon: error initializing graphdriver: devmapper: Unable to take ownership of thin-pool (thinpool_VG_38401-...data blocks
Dec 06 10:28:46 yoda3 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Dec 06 10:28:46 yoda3 systemd[1]: Failed to start Docker Application Container Engine.
Dec 06 10:28:46 yoda3 systemd[1]: Unit docker.service entered failed state.
Dec 06 10:28:46 yoda3 systemd[1]: docker.service failed.
I assumed I could get around the 'unable to take ownership of thin-pool' by doing a reboot. But after reboot and trying to start docker again I got the following error:
systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2017-12-06 10:30:37 UTC; 2min 29s ago
Docs: https://docs.docker.com
Process: 3180 ExecStart=/usr/bin/dockerd -G uwsgi --data-root=/area51/docker -H unix:///var/run/docker.sock (code=exited, status=1/FAILURE)
Main PID: 3180 (code=exited, status=1/FAILURE)
Memory: 37.9M
CGroup: /system.slice/docker.service
Dec 06 10:30:36 yoda3 systemd[1]: Starting Docker Application Container Engine...
Dec 06 10:30:36 yoda3 dockerd[3180]: time="2017-12-06T10:30:36.893777000Z" level=warning msg="libcontainerd: makeUpgradeProof could not open /var/run/docker/lib...containerd"
Dec 06 10:30:36 yoda3 dockerd[3180]: time="2017-12-06T10:30:36.901958000Z" level=info msg="libcontainerd: new containerd process, pid: 3224"
Dec 06 10:30:37 yoda3 dockerd[3180]: Error starting daemon: error initializing graphdriver: devicemapper: Non existing device thinpool_VG_38401-thinpool
Dec 06 10:30:37 yoda3 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Dec 06 10:30:37 yoda3 systemd[1]: Failed to start Docker Application Container Engine.
Dec 06 10:30:37 yoda3 systemd[1]: Unit docker.service entered failed state.
Dec 06 10:30:37 yoda3 systemd[1]: docker.service failed.
So, obviously writing zeroes into the thinpool_meta device is not the right thing to do, it seems to destroy my thinpool device.
Anyone here that can tell me the right steps to clear the thin-pool device? Preferably the solution should not require a reboot.
I'm trying to have docker container stored in a luks device, but using the below command it is not working.
I'm also trying to have a docker container to use (and maybe open) a luks file as a volume data, but I don't know how to bind it to the container
I have done the following :
pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created
vgcreate vgluks /dev/sdb
Volume group "vgluks" successfully created
lvcreate -l 90%FREE -n lvdocker-data vgluks
WARNING: crypto_LUKS signature detected on /dev/vgluks/lvdocker-data at offset 0. Wipe it? [y/n]: y
Wiping crypto_LUKS signature on /dev/vgluks/lvdocker-data.
Logical volume "lvdocker-data" created.
lvcreate -l 100%FREE -n lvdocker-metadata vgluks
Logical volume "lvdocker-metadata" created.
created a key
dd if=/dev/urandom of=/tmp/key bs=4K count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00126301 s, 3.2 MB/s
cryptsetup luksFormat --batch-mode --key-file=/tmp/key /dev/vgluks/lvdocker-data
cryptsetup luksFormat --batch-mode --key-file=/tmp/key /dev/vgluks/lvdocker-metadata
cryptsetup luksOpen --key-file=/tmp/key /dev/vgluks/lvdocker-data cryptfs-data
cryptsetup luksOpen --key-file=/tmp/key /dev/vgluks/lvdocker-metadata cryptfs-metadata
formated in ext4
mkfs.ext4 /dev/mapper/cryptfs-data
mkfs.ext4 /dev/mapper/cryptfs-metadata
my docker.service looks like this :
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/docker
ExecStart=/usr/bin/docker daemon -H fd:// $OPTIONS
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
[Install]
WantedBy=multi-user.target
and the /etc/sysconfig/docker is like :
OPTIONS="--storage-driver=devicemapper --storage-opt dm.datadev=/dev/vgluks/lvdocker-data --storage-opt dm.metadatadev=/dev/vgluks/lvdocker-metadata --insecure-registry myregistryhost:443 -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"
The output of systemctl is :
systemctl status -l docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2015-12-23 16:05:28 CET; 25min ago
Docs: https://docs.docker.com
Main PID: 6544 (code=exited, status=1/FAILURE)
Dec 23 16:05:28 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Dec 23 16:05:28 localhost.localdomain docker[6544]: time="2015-12-23T16:05:28.457356524+01:00" level=warning msg="/!\\ DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING /!\\"
Dec 23 16:05:28 localhost.localdomain docker[6544]: time="2015-12-23T16:05:28.478448525+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: Error running deviceCreate (CreatePool) dm_task_run failed"
Dec 23 16:05:28 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Dec 23 16:05:28 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Dec 23 16:05:28 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
Dec 23 16:05:28 localhost.localdomain systemd[1]: docker.service failed.
I have the following issue installing and provisioning my Kubernetes CoreOS-libvirt-based cluster.
When I'm logging on the master node, I see the following:
ssh core#192.168.10.1
Last login: Thu Dec 10 17:19:21 2015 from 192.168.10.254
CoreOS alpha (884.0.0)
Update Strategy: No Reboots
Failed Units: 1
kube-addons.service
Trying to debug it, I run and receive the following:
core#kubernetes-master ~ $ systemctl status kube-addons.service
● kube-addons.service - Kubernetes addons
Loaded: loaded (/etc/systemd/system/kube-addons.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2015-12-10 16:41:06 UTC; 41min ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 801 ExecStart=/opt/kubernetes/bin/kubectl create -f /opt/kubernetes/addons (code=exited, status=1/FAILURE)
Process: 797 ExecStartPre=/bin/sleep 10 (code=exited, status=0/SUCCESS)
Process: 748 ExecStartPre=/bin/bash -c while [[ "$(curl -s http://127.0.0.1:8080/healthz)" != "ok" ]]; do sleep 1; done (code=exited, status=0/SUCCESS)
Main PID: 801 (code=exited, status=1/FAILURE)
Dec 10 16:40:53 kubernetes-master systemd[1]: Starting Kubernetes addons...
Dec 10 16:41:06 kubernetes-master kubectl[801]: replicationcontroller "skydns" created
Dec 10 16:41:06 kubernetes-master kubectl[801]: error validating "/opt/kubernetes/addons/skydns-svc.yaml": error validating data: found invalid field portalIP for v1.ServiceSpec; if you choose to ignore these errors, turn validation off with --validate=false
Dec 10 16:41:06 kubernetes-master systemd[1]: kube-addons.service: Main process exited, code=exited, status=1/FAILURE
Dec 10 16:41:06 kubernetes-master systemd[1]: Failed to start Kubernetes addons.
Dec 10 16:41:06 kubernetes-master systemd[1]: kube-addons.service: Unit entered failed state.
Dec 10 16:41:06 kubernetes-master systemd[1]: kube-addons.service: Failed with result 'exit-code'.
My etcd version is:
etcd --version
etcd version 0.4.9
But I have a etcd2 also:
etcd2 --version
etcd Version: 2.2.2
Git SHA: b4bddf6
Go Version: go1.4.3
Go OS/Arch: linux/amd64
And at the current moment the second one is being runned:
ps aux | grep etcd
etcd 731 0.5 8.4 329788 42436 ? Ssl 16:40 0:16 /usr/bin/etcd2
root 874 0.4 7.4 59876 37804 ? Ssl 17:19 0:02 /opt/kubernetes/bin/kube-apiserver --address=0.0.0.0 --port=8080 --etcd-servers=http://127.0.0.1:2379 --kubelet-port=10250 --service-cluster-ip-range=10.11.0.0/16
core 953 0.0 0.1 6740 876 pts/0 S+ 17:27 0:00 grep --colour=auto etcd
What causes the issue and how can I solve it?
Thank you.
The relevant log line is:
/opt/kubernetes/addons/skydns-svc.yaml": error validating data: found invalid field portalIP for v1.ServiceSpec; if you choose to ignore these errors, turn validation off with --validate=false
You should figure out what's invalid about that IP or set the flag to ignore.