Docker command hang on container - docker

I have created one container by using msql-server:5.7. After creating, I am able to access it without a problem. But after a couple of minutes, all Docker commands in the container becomes unresponsive. I cannot inspect, stop or kill. Is there a way I can debug? I have some other containers running as well, but those are OK.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ca471079614b mysql/mysql-server:5.7 "/entrypoint.sh mysq…" 2 hours ago Up 2 hours (healthy) 33060/tcp db
$ docker info
Containers: 8
Running: 8
Paused: 0
Stopped: 0
Images: 94
Server Version: 17.12.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1047-aws
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.42GiB
Name: ip-172-31-12-60
ID: I7FQ:XUUN:UU5C:KVJI:JPDT:L2BV:B3EQ:5LHI:5XD5:PSWP:NI7Y:BDX7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
$ docker version
Client:
Version: 17.12.0-ce
API version: 1.35
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:11:19 2017
OS/Arch: linux/amd64
Server:
Engine:
Version: 17.12.0-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:09:53 2017
OS/Arch: linux/amd64
Experimental: false
$ uname -a
Linux ip-172-31-12-60 4.4.0-1047-aws #56-Ubuntu SMP Sat Jan 6 19:39:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Are you double sure this is not docker specific issue? Maybe unsupported kernel version? All logs are dumped to a log file inside the volume in the /var/lib/docker/volumes directory. Try running the container without the -d flag so all output goes to stdout directly to identify the problem.

after couple of day trying, it turns out I change my instance-type from m4.2xlarge to m4.4xlarge. and after trying 1 days. it seems the problem solved. I have no idea why but it is good so far

Related

Docker cant´t find, pull images : server mis behaving

Docker toolbox installed for windows 10 Home Edition. Docker toolbox for this OS is installed. But when I can start learning the program, it gives me these errors. I cannot create an image or run an image. At the end of each error message it tells me that I have a server misbehaving. At the moment of running an image gives me the following error:
Unable to find image 'busybox:latest'
locally
latest: Pulling from library / busybox
d9cbbca60e5f: Pulling fs layer C: \Program Files\ Docker Toolbox\ docker.exe: error pulling image configuration: Get https: //registry-1.docker.io/v2/library/busybox/blobs/sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: server misbehaving.
See 'C:\Program Files\Docker Toolbox\docker.exe run --help'.
And at the momento to pulling and image, i got this:
$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
d9cbbca60e5f: Pulling fs layer error pulling image configuration: Get https://registry-1.docker.io/v2/library/busybox/blobs/sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: server misbehaving
This is my docker info :
$ docker info Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.154-boot2docker
Operating System: Boot2Docker 19.03.5 (TCL 10.1)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 989.5MiB
Name: default
ID: ANYM:Q52D:BUAW:R6IO:HTFN:S4I6:JLX6:WVNK:QEGO:OHCZ:PCHP:NWJN
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode: false
Username: dsabillon94
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Docker version
$ docker version
Client:
Version: 19.03.1
API version: 1.40
Go version: go1.12.7
Git commit: 74b1e89e8a
Built: Wed Jul 31 15:18:18 2019
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea838
Built: Wed Nov 13 07:28:45 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
You have these options
Wipe everything and re-install
Switch to Beta version.
You can also set DNS Server to 8.8.8.8 and try again pulling an image.
I hope one of option will help you :)

Docker fails building image when using default seccomp profile

When I try to build a docker image I get the following error
OCI runtime create failed: container_linux.go:346: starting container
process caused "seccomp: config provided but seccomp not supported":
unknown
I'm running docker build -t cprates/lws_base:latest -f Dockerfile.base .
The Dockerfile is a basic dockerfile with nothing special to it
FROM golang:1.12.7-buster
WORKDIR /lws
COPY go.mod .
RUN go mod download \
&& go get -u golang.org/x/lint/golint
I suspect that this started happening after a system update, but not sure.
I can run a container by not using the default seccomp profile with --security-opt seccomp=unconfined. But docker build does not accept the flag --security-opt.
I already re-installed docker-ce following the instructions in the official website.
I'm using Fedora 30 which according to the docs is supported:
uname -srom
Linux 5.5.10-100.fc30.x86_64 x86_64 GNU/Linux
Docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:26:25 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:25:01 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc9+dev
GitCommit: 2186cfa3cd52b8e00b1de76db7859cacdf7b1f94
docker-init:
Version: 0.18.0
GitCommit: fec3683
Docker info
Client:
Debug Mode: false
Server:
Containers: 9
Running: 0
Paused: 0
Stopped: 9
Images: 5
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: 2186cfa3cd52b8e00b1de76db7859cacdf7b1f94
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 5.5.10-100.fc30.x86_64
Operating System: Fedora 30 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.53GiB
Name: taplop.taplopmain
ID: HT7E:UK3I:IYBL:FO46:PHZS:AV7O:GYCY:3QUS:7I6H:PIS4:LBJZ:VRLH
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Any clue on what could be the problem?
It turns out that containerd was picking up the wrong runC. Some time ago I installed runC from github and completely forgot.
If you're wondering if you have the same problem, run to following command to check how many runC binaries you have in your system
$ whereis runc
runc: /usr/bin/runc
This is the correct one for me, using Fedora 30.

Not able to ping Windows Host machine from my Docker Container

I am running docker container using Docker Desktop on Windows 2016 server. I am using Docker Desktop, instead of Docker EE on Windows 2016 server because I want to run Linux Container. Details of Docker is as follows: -
Version of Docker Desktop is
Version 2.0.0.3 (31259)
Channel: stable
Output of docker info is as follows: -
Containers: 3
Running: 1
Paused: 0
Stopped: 2
Images: 2
Server Version: 18.09.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.125-linuxkit
Operating System: Docker for Windows
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.934GiB
Name: linuxkit-00155d2dab04
ID: OQK7:Z24A:V5KV:CYEW:6QQE:KXOB:UXWY:KVRF:QEHF:SATF:CTLU:WTPM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 32
Goroutines: 53
System Time: 2020-02-26T06:36:51.0000459Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Output of docker version is as follows: -
Client: Docker Engine - Community
Version: 18.09.2
API version: 1.39
Go version: go1.10.8
Git commit: 6247962
Built: Sun Feb 10 04:12:31 2019
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:06 2019
OS/Arch: linux/amd64
Experimental: false
I am not able to ping my windows host machine from my Linux container.
But this is possible on my Windows 10 machine.
Any clue in this what I am missing???
Run ifconfig and check the IP of docker0 interface.
And ping using that IP.

Docker: Error grabbing logs: invalid character '\x00' looking for beginning of value

I upgraded my Docker version to the latest one and I got this error:
error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value
I can’t read logs using:
docker-compose logs -f myservice
nor with:
docker logs -f 6f454c73ff9c
Output of Docker version:
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:18 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:40:56 2017
OS/Arch: linux/amd64
Experimental: false
Output of Docker info:
Containers: 11
Running: 11
Paused: 0
Stopped: 0
Images: 8
Server Version: 17.09.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 76
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-97-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.695GiB
Name: egmintel-desktop
ID: VB22:IXWI:GY6D:QPM4:SPHX:HYUP:OQN7:ZM55:LLKE:P3UU:XK7F:26TH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
My docker-compose version:
docker-compose version 1.16.1, build 6d1ac21
I also noticed that one of my containers is not working properly (but I can't see the error since I can't read logs) and I need to restart the Docker service so my container starts working again.
Any ideas how to solve this? Thanks!
simply remove the ~/.docker/ directory
Looks like your events.log file got corrupted. Try deleting following file
"/var/run/docker/libcontainerd/containerd/events.log"
restart the docker through init.d
Using windows Docker Desktop
Troubleshoot -> Reset to factory defaults
This fixed the issue for me.
For windows you need delete json file with 0x00 bytes
located in %USERPROFILE%\.docker\contexts folder and it's subdirs.

Docker command with build / run / etc. "too many open files"

Im installed Docker on server machine (with Ubuntu 16.04), but trying run any commands, I get this error:
"Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: too many open files"
Please, who know how solve this issue?
Docker Info:
Containers: 18
Running: 0
Paused: 0
Stopped: 18
Images: 4
Server Version: 17.03.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 76
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-ini
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953 GiB
Name: web.renderdeal.com
ID: LXNZ:I6HH:ZVBB:KS3V:3WCT:ADWY:C2MZ:QJ37:VZUU:EZ6T:PYWO:66WQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: animarender
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
/Docker Info
docker version
Client:
Version: 17.03.0-ce
API version: 1.26
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 11:02:43 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.0-ce
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 11:02:43 2017
OS/Arch: linux/amd64
Experimental: false
/ docker version
Considering Docker status seems OK, check if the issue is on your side (as in this ticket)
Let's check on the current open files
lsof | grep "rclone" | wc -l
returns 6458
Now, Check on current ulimit
ulimit -n
returns 1024
Set that to 9000 and Check again
ulimit -n 9000 && ulimit -n
returns 9000
To make the ulimit more persistent you can either edit /etc/security/limits.conf
and add :
* soft nofile 9000 + reboot
or write it into your user's .bashrc

Resources