I start docker at private cloud host and get next error
sudo docker run hello-world
docker: Error response from daemon: OCI runtime create failed: /var/lib/docker/overlay2/83d55d497417883ea85b63ebe3138e5f5dbc2b3c8b2db663c1a007e7af1757f1/merged
is not an absolute path or is a symlink: unknown.
ERRO[0000] error waiting for container: context canceled
ll shows next
ll /var/lib/docker/overlay2
lrwxrwxrwx 1 root root 20 Jun 19 15:32 /var/lib/docker/overlay2 -> /dev/docker-overlay2/
Are there any way to fix it ?
I met this problem, I restart docker service, solve it
systemctl restart docker
problem was that someone configured symlink for "fat" directory with images to move it to another disc. Instead of properly configuring in /etc/docker/daemon.json
Related
I am facing an issue with mounting a host directory into docker container with both -v and --mount options.
Using mount:
docker run --mount type=bind,source=/home/myuser/docker_test/out_dir,target=/home/out_dir --user 12345:1000 -it docker-name:0.1 bash
docker: Error response from daemon: invalid mount config for type "bind": stat /home/myuser/docker_test/out_dir: permission denied.
But I am able to do stat on this directory.
stat /home/myuser/docker_test/out_dir
File: '/home/myuser/docker_test/out_dir'
Size: 4096 Blocks: 8 IO Block: 32768 directory
Device: 33h/51d Inode: 9275022755226025350 Links: 2
Access: (0770/drwxrwx---) Uid: (12345/ myuser) Gid: ( 1000/ hercules)
Access: 2022-12-01 02:12:54.430582000 -0500
Modify: 2022-12-01 02:12:38.239629000 -0500
Change: 2022-12-01 02:12:38.239629000 -0500
Birth: -
Using -v:
docker run -v /home/myuser/docker_test/out_dir:/home/out_dir --user 12345:1000 -it docker-name:0.1:0.1 bash
docker: Error response from daemon: error while creating mount source path '/home/myuser/docker_test/out_dir': mkdir /home/myuser/docker_test: permission denied.
ERRO[0000] error waiting for container: context canceled
I don't know why it's trying to do mkdir but /home/myuser/docker_test already exists and is writable for the current user.
Am I missing something here?
BTW - /home is a NFS mounted directory.
EDIT: mounting /tmp worked. So this means it is related to the NFS mounted directory /home.
EDIT 2
I am working on a network machine where I don’t have root (sudo) access.
The docker service is installed by root user.
/home/myuser/docker_test/out_dir has 700 (rwx------) permissions. If I change the permission to 755, it will work. But I can’t change the directory permissions.
My question is why stat is failing when the user starting the docker has the permissions to access the source directory?
Is the stat being called by the docker executable as some ‘other’ user?
Use:
sudo docker run -v /home/myuser/docker_test/out_dir:/home/out_dir --user 12345:1000 -it docker-name:0.1:0.1 bash
I'm trying to deploy a docker container to perform some testings with gitlab runners but when I'm doing the docker-compose up command I get the following output:
admin#runners-test:~/runner-test$ sudo docker-compose up -d
Starting gitlab-runner ... error
ERROR: for gitlab-runner Cannot start service gitlab-runner: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown
ERROR: for gitlab-runner Cannot start service gitlab-runner: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown
ERROR: Encountered errors while bringing up the project.
Abd this is the output of journalctl:
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.560275437Z" level=error msg="stream copy error: reading from a closed fifo"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.577270402Z" level=error msg="stream copy error: reading from a closed fifo"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.675282811Z" level=error msg="0cd3bbb779a947012c9059921f092b569eb088bb2fe0bf99a8ae3266ec43abbd cleanup: failed to delete container from containerd: no such container"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.675625496Z" level=error msg="Handler for POST /v1.25/containers/0cd3bbb779a947012c9059921f092b569eb088bb2fe0bf99a8ae3266ec43abbd/start returned error: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown"
And the docker-compose file:
version: '3'
services:
gitlab-runner:
container_name: gitlab-runner
image: 'gitlab/gitlab-runner:latest'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./config:/etc/gitlab-runner
restart:
unless-stopped
This error occurs with any container I try to lift, even doing a simple docker run hello-world...
I also take a look to this post on proxmox forum but I don't know how to add kernel command line parameters for a lxc container, I'v been trying editing the /etc/default/grub file but update-grub command doesn't work as I want inside the lxc, I get the following output:
admin#runners-test:/$ sudo update-grub
[sudo] password for admin:
/usr/sbin/grub-probe: error: failed to get canonical path of `/dev/mapper/pve-vm--1010118--disk--0'.
I'm really stuck with this, so any kind of help would be welcome :) Thanks!
I referenced many other links when finally discovering doing a (on CentOS) update did the trick. So:
yum update -y
reboot
After the reboot, the system came back online and I proceeded to run my docker container, in this case a docker registry:
[root#server ~]# docker run -d --name registry registry:2
c40941bf42c853709bcca05bad4e8914df1f4932a355607f37b55f7e0ed01e60
[root#server ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
c40941bf42c8 registry:2 "/entrypoint.sh /e..." 3 seconds ago Up 2 seconds 5000/tcp
The docker container booted up fine.
So, also here are reference pages I checked out, and I did not have to change anything in /boot/boot.txt as this one references, but is good info:
https://my-take-on.tech/2021/05/07/fix-docker-cgroup-errors-after-systemd-248-update/
and this one, which after I read it, it had me finally trying to do a yum update for fix:
docker-compose throws errors by starting mariadb for an private nextcloud installation
How to run crictl as non-root user.
My docker commands work with non-root user because my user is added to docker group.
id
uid=1002(kube) gid=100(users) groups=100(users),10(wheel),1001(dockerroot),1002(docker)
I am running dockerD daemon which uses containerd and runc as runtime.
I installed crictl binary and pointed it to connect to existing dockershim socket with config file as below.
cat /etc/crictl.yaml
runtime-endpoint: unix:///var/run/dockershim.sock
image-endpoint: unix:///var/run/dockershim.sock
timeout: 2
debug: false
pull-image-on-create: false
crictl works fine with sudo but without sudo it fails like this.
[user#hostname~]$ crictl ps
FATA[0002] connect: connect endpoint 'unix:///var/run/dockershim.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
I also tried to change group of dockershim.sock to 'docker' from 'root' just like docker.sock was to try, still same.
srwxr-xr-x 1 root docker 0 Jan 2 23:36 /var/run/dockershim.sock
srw-rw---- 1 root docker 0 Jan 2 23:33 /var/run/docker.sock
sudo usermod -aG docker $USER
or you can see docker postinstall
I'm using Window Linux Subsystem (Debian stretch). Followed the instruction on Docker website, I installed docker-ce, but it cannot start. Here is the info:
$ sudo service docker start
grep: /etc/fstab: No such file or directory
[ ok ] Starting Docker: docker.
$ sudo service docker status
[FAIL] Docker is not running ... failed!
What should I do with /etc/fstab not found?
to fix fstab
touch /etc/fstab
if you run dockerd, it will give you the failed message:
INFO[2022-01-27T17:55:14.100489400+07:00] Loading containers: start.
WARN[2022-01-27T17:55:14.191666800+07:00] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.2 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING`, error: exit status 4
INFO[2022-01-27T17:55:14.493716300+07:00] stopping event stream following graceful shutdown error="<nil>" module=libcontainerd namespace=moby
INFO[2022-01-27T17:55:14.494906600+07:00] stopping event stream following graceful shutdown error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2022-01-27T17:55:14.495048400+07:00] stopping healthcheck following graceful shutdown module=libcontainerd
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables --wait -t nat -N DOCKER: iptables v1.8.2 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING
(exit status 4)
that is Debian nat issue, fix it with:
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
now you can start the service again
you can follow this to make it start on startup https://askubuntu.com/a/1356147/138352
Edited:
if the issue with IP table still persisted try to set WSL version to 2, run the command from Windows shell:
wsl --set-version <distribution name> 2
the distribution list can be found with command wsl -l
I was getting the same error. Apparently on my install of WSL with Debian, I didn't have an etc/fstab file. Surprisingly, just creating the file via 'touch' worked:
sudo touch /etc/fstab
Perhaps a good signal https://learn.microsoft.com/en-us/windows/wsl/release-notes#build-17093
WSL now processes the /etc/fstab file during instance start [GH 2636].
For anybody stumbling across this years later like me, Docker doesn't work inside WSL.
But you can use Docker for Windows and WSL2 to run native containers inside your Linux Distro and the install and config is quite painless https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-containers
I am using Docker version 17.06.0-ce on Redhat with devicemapper storage. I am launching a container running a long-running service. The master process inside the container sometimes dies for whatever reason. I get the following error message.
/bin/bash: line 1: 40 Killed python -u scripts/server.py start go
I would like the container to exit and to be restarted by docker. However docker never exits. If I do it manually I get the following error:
Error response from daemon: driver "devicemapper" failed to remove root filesystem.
After googling, I tried a bunch of things:
docker rm -f <container>
rm -f <pth to mount>
umount <pth to mount>
All result in device is busy. The only remedy right now is to reboot the host system which is obviously not a long-term solution.
Any ideas?
I had the same problem and the solution was a real surprise.
So here is the error om docker rm:
$ docker rm 08d51aad0e74
Error response from daemon: driver "devicemapper" failed to remove root filesystem for 08d51aad0e74060f54bba36268386fe991eff74570e7ee29b7c4d74047d809aa: remove /var/lib/docker/devicemapper/mnt/670cdbd30a3627ae4801044d32a423284b540c5057002dd010186c69b6cc7eea: device or resource busy
Then I did the following (basically go through all processes and look for docker in mountinfo):
$ grep docker /proc/*/mountinfo | grep 958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac
/proc/20416/mountinfo:629 574 253:15 / /var/lib/docker/devicemapper/mnt/958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,relatime shared:288 - xfs /dev/mapper/docker-253:5-786536-958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota
This got be the PID of the offending process keeping it busy - 20416 (the item after /proc/)
So I did a ps -p and to my surprise find:
[devops#dp01app5030 SeGrid]$ ps -p 20416
PID TTY TIME CMD
20416 ? 00:00:19 ntpd
A true WTF moment. So I pair problem solved with Google and found this:
Then found this https://github.com/docker/for-linux/issues/124
Turns out I had to restart ntp daemon and that fixed the issue!!!