I am getting the below error message：
{"changed": false, "msg": "Error creating container: 500 Server Error:
Internal Server Error ("error creating overlay mount to
/var/lib/docker/overlay2/e17accf17aa46c9d12dbc4da10e399121b476698039cdcf54a5bb42443ffc260-init/merged:
invalid argument")"}
docker info shows below：
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: false
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: 1556cce7e5c5349fdffeb123c092a7681769866a (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 1556cce7e5c5349fdffeb123c092a7681769866a-dirty (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 4.18.7-200.fc28.x86_64
Operating System: Fedora 28 (Server Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 24
Total Memory: 94.34 GiB
Name: xxxxxx.com
ID: U63T:T7ZF:HWDJ:4NKZ:YUW2:KO6W:7RI5:Y6MY:VKSQ:DJSL:FGD5:QMZS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
Registries: docker.io (secure), registry.fedoraproject.org (secure), quay.io (secure), registry.access.redhat.com (secure), registry.centos.org (secure), docker.io (secure)
Here Backing filesystem shows unknown here. How to change it to xfs or somerelevant to work?
I was able to resolve this by doing a factory reset in Docker.
Docker for Mac -> Troubleshooting (it is the "bug" icon next to the Settings icon) -> Reset Factory Defaults
$ vim /etc/docker/daemon.json
{
"max-concurrent-uploads": 1
}
$ service docker restart
Posting it in case it helps someone.
I had the same issue with the docker. It was not able to push images to GCR. The above was able to solve my issue.
First, I think you can check the /var/lib/docker/overlay2 directory using xfs_info whether or not it can be supported the overlay2.
For instance,
$ xfs_info /var/lib/docker/overlay2 | grep ftype
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
If the output is ftype=1, then it can be supported the overlay2.
I hope it help you. :^)
I have just meet the same issue. And I fixed it by change the disk volume type used by docker into Ext4 or change the path into a Ext4 volume by this command in docker.service
ExecStart=/usr/bin/dockerd --graph /new-path/docker
It might happen because daemon.json doesn't exist. For create it you can do following steps: open "Docker Desktop" > "Preferences" > "Docker Engine" > edit the texteditor under "Configure the Docker daemon by typing a json Docker daemon configuration file" > "Apply & Restart".
Related
I'm having problems getting a listing of images from a specific registry that I've set up on a local server, or, maybe, I'm having issues publishing them to that registry in the first place, as this is my first adventure into docker registries, I may just be confused with the terms used.
There's an old question, here, that kind of looks like what I want to achieve, but it appears that docker has gained built-in support for this, in the meanwhile, so the methods mentioned here are no longer relevant.
I have 2 servers (for the purpose of this question):
rancher-server: This server has a rancher:v2.6.0 container running and a registry:2 container.
k8s-server: This is just a freshly installed server, with the docker and kubernetes packages installed, that I want the rancher server to administer.
On k8s-server, I'm trying to spin up a docker image rancher/rancher-agent:v2.6.0 with a few arguments, that should let it relinquish control to the rancher server.
The trick here is, that this is all required to work without internet access (currently there IS internet access, but it's a PoC for a task that requires to be air-gapped). For the purposes of this question, I really just want to be able to spin up docker containers on k8s-server, using the registry on rancher-server.
Currently, this is the state of rancher-server:
# docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b9a15ea00d5e registry:2 "/entrypoint.sh /e..." About an hour ago Up About an hour 0.0.0.0:5000->5000/tcp local-registry
1b6bc6b88a8e 08c9693b4357 "entrypoint.sh 08c..." 26 hours ago Up 2 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp goofy_minsky
# docker image ls --all (the list is big, this is just a sample):
REPOSITORY TAG IMAGE ID CREATED
rancher/rancher-agent v2.6.0 9c35a790aa16 2 weeks ago
rancher-server.example.com:5000/rancher/rancher-agent v2.6.0 9c35a790aa16 2 weeks ago
# docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 225
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 66aedde759f33c190954815fb765eedc1d782dd9 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-1160.41.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 3.701 GiB
Name: rancher-server
ID: SA2T:G2IA:CGER:6BC5:HIV2:4T6T:LF3Q:2YVS:SYU7:SQ5V:ACUS:BMEX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
rancher-server.example.com:5000
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
On the k8s-server, I try to list the contents of that registry:
# docker image ls --all rancher-server.example.com:5000
REPOSITORY TAG IMAGE ID CREATED SIZE
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 66aedde759f33c190954815fb765eedc1d782dd9 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-1160.41.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 3.701 GiB
Name: k8s-server
ID: QETJ:QSPQ:VS36:OOOA:ZPYL:CDHK:AJ5G:N4BD:ZQUH:UL6O:PHAB:5UOE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
rancher-server.example.com:5000
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
I had to jump through a few hoops to get there, in the first place, marking the registry as unsafe in /etc/docker/daemon.json on the k8s-server and disabling selinux on the rancher-server, for example.
I've tried to docker login rancher-server.example.com:5000 first, but that made no difference. It does look like, to me, that the k8s-server is configured correctly, but that the images on rancher-server haven't been tagged/pushed properly, but when I look back at the registry, I don't know how to do it differently, and, as far as I understand the registry, it looks fine to me?
I've changed the server names for anonymity and the output has been lightly edited for presentation.
EDIT:
I think I found a clue to what's happening here, it turns out that I can actually run the images from this registry remotely, just fine, it just so happens that I have no way to discover the names of the images, however, if I do a docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher-server.example.com:5000/rancher/rancher-agent:v2.6.0 --server https://rancher-server.example.com:5000 --token <token> --ca-checksum <ca-checksum> --etcd --controlplane it actually pulls and runs the container, so it looks like the registry itself is fine, but maybe the index isn't?
I am trying to run a container as daemon as follows :
$docker run --privileged --net=host -d d34f188005c4
34045723709f3553533a1c823800697e55d43a7e676499310751f7116d3a1068
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
As seen from above, docker run cmd returns a long string, but it fails to run as seen from the docker ps output. Further, I get following from the syslog while the docker run cmd as above executes:
kernel: [281942.173389] aufs au_opts_verify:1597:dockerd[21361]: dirperm1 breaks the protection by the permission bits on the lower branch
Kernel/docker version and other required info is as follows:
# docker info
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 1
Server Version: 17.03.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 8
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-116-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67 GiB
Name: ubuntu1604
ID: B35S:GX2J:VTYF:HKJ2:KMIP:SU35:YCSV:YHM2:R6L5:K7CW:4ZUV:EYGX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Can anyone please let me know how to fix this? Can we get any clue from the syslog above? If any other info is required, please let me know.
Adding additional info based on comments below :
$docker logs 34045723709f3553533a1c823800697e55d43a7e676499310751f7116d3a1068
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Access to docker on host failed!!`
$ps -aux | grep dockerd
root 21836 0.4 0.2 436168 42920 ? Ssl 10:39 0:00 /usr/bin/dockerd -H fd://
Using ubuntu 18.04
after upgrading docker to "Docker version 18.09.0-ce-tp5, build 9eb3d36"
Everything works fine before updating docker
Getting following error on docker-compose up:
ERROR: for mysql_1 Cannot start service mysql: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/242793c2e7ad05e93ccff53ae37e5d8b054a72f9c2076c1b480f33929dbb45aa/log.json: no such file or directory): exec: "docker-runc": executable file not found in $PATH: unknown
docker info gives
linux#linux-linux:/$ docker info
Containers: 14
Running: 14
Paused: 0
Stopped: 0
Images: 1091
Server Version: 18.09.0-ce-tp5
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1038
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: containerd runc
Default Runtime: containerd
Init Binary: docker-init
containerd version: ce243288e27971e324363de8f322d221635a8521 (expected: 468a545b9edcd5932818eb9de8e72413e616e86e)
runc version: 1555a78945e56737bc3d15565191d857ee6fa2b9 (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: fec3683
Security Options:
apparmor
Kernel Version: 4.15.0-32-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.719GiB
Name: linux-linux
ID: F475:F7JY:3HSG:XLGM:BAPD:CZYD:HRRO:SN3N:SFSX:7XOK:VDZ4:SNMV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
Using ubuntu 18.04 after upgrading docker to "Docker version 18.09.0-ce-tp5, build 9eb3d36"
The TP# releases are pre beta (I suspect it stands for technical preview). These will have issues, and there were lots of changes with the containerd integration. You can try upgrading to beta1, or go back to the stable 18.06.1 release. You can see the releases here:
https://github.com/docker/docker-ce/releases
Issue:
I faced the same issue.
Solution
first i restart docker,but when i tried to start my container i was having Error response from daemon: OCI runtime create failed: container with id exists:xxxxxxxxxx
but in ps -a it was showing nothing.
while i was able to run the new container using the same id.
than tried
reboot my server and it is working fine.
as you can see in this 2 rows:
containerd version: ce243288e27971e324363de8f322d221635a8521 (expected: 468a545b9edcd5932818eb9de8e72413e616e86e)
runc version: 1555a78945e56737bc3d15565191d857ee6fa2b9 (expected: 69663f0bd4b60df09991c08812a60108003fa340)
docker expects another internal container id as the tagged container has. i guess this causes the problem.
have you tried to rebuild your container?
I faced a similar problem.
Try committing your container.
Then run it again.
e.g.
docker commit RUNNING_CONTAINER IMAGE1
docker run -d IMAGE1
service docker restart solved the problem in my case
I have been trying to compile docker for the ARM-64 architecture. Docker doesn’t officially support ARM 64-bits (at least not through the package management tools); hence I have to build it from source. Building docker binary set needs docker itself as a dependency. I’ve already managed to compile both the docker daemon and the client via the following (hack) command:
./hack/make.sh dynbinary
However, I haven’t managed to run it successfully. Both binaries are compiled and work, but when I want to start up the daemon it complains about other dependencies:
Failed to connect to containerd. Please make sure containerd is installed in your PATH or you have specified the correct address. Got error: exec: "docker-containerd": executable file not found in $PATH
As I mentioned earlier, I cannot build all the binaries as they need docker itself running.
Looking forward to your help.
Two weeks ago, I was able to install Docker on a Pine64 running Armbian (Debian based). It was as easy as following the official documentation for armhf with one exception, change [arch=armhf] by [arch=arm64] when you add the new apt source.
After the install you have a real Arm64 docker running :
root#pine64:~# docker system info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 60
Server Version: 17.12.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 28
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 3.10.107-pine64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 979.6MiB
Name: pine64
ID: xxx
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: xxx
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
when I volume mapping in Docker 1.13.0, some file is corrupted.
In docker container, when "ls -l" the folder, it displayed like this:
"?????????? ? ? ? ? ? file_corrpted_and_cant_access.conf"
and can't edit it or delete it.
it just show "No such file or directory".
I think it can't link file inode and path.
how to fix it?
Additional information
after volume mapping, I soft link volume mapping folder inside docker container.
docker run --privileged -d -v /opt/volume_mapping_folder/:/inside_container/inside_folder --restart=always testcontainer
and insde docker container I soft link folder
docker -it testcontainer /bin/bash
ln -s /inside_container/inside_folder /opt/appFolder
touch /opt/appFolder/file_corrpted_and_cant_access.conf
Output of docker info:
Server Version: 1.13.0
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.6.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.671 GiB
ID: 2D2E:73MA:BJQ3:WQAJ:BR3W:TYF5:F3MQ:E7S3:KZGV:A64K:ASZK:UEXE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
This is is related xfs filesystem with overlay mode.
https://github.com/docker/docker/issues/27358
the solution is one of these.
xfs partitions need to be formatted with ftype set to 1
use ext4 filesystem instead of xfs
setting docker Storage Driver to 'devicemapper'
I hit a similar problem today using Docker 1.13.0 running a CentOS image on a RHEL 7.3 system. Files in my containers root filesystem were acting odd. For example, I would try to delete a file but it would not be removed from the filesystem and display the same '?' you posted when running 'ls'. I also had problems trying to change ownership of files. Anyway, my guess is Docker 1.13.0 changed the default Storage Driver to overlay when in the past (Docker 1.12.5) it was devicemapper. I changed the default back to devicemapper and the problems went away.
I am not a Docker or Linux Filesystem expert and I am not sure if the change to overlay as a default was intentional.