how to in increase number of tcp connection above 65536 in Docker - docker

Description
I have did a tcp connect test, start a tcp server in a host, and start other host run containers in containers connect to tcp server.
when total number of connections (sum all connectoin number of all containers) almost reach to 65535 (eg: 64512 just now), the other tcp connection throw exception "connection timeout".
I tried to use 3 hosts without docker to connect tcp server each 60K connections is ok, it said that the tcp server do 180K connections is ok.
Steps to reproduce the issue:
1. start tcp server in host A
2. start container C1 in host B
3. start 60K connection to tcp server in C1
4. start container C2 in host B
5. start 60K connection to tcp server in C2
Describe the results you received:
total tcp connection more less than 120K
Describe the results you expected:
each 60K connection in two container total 120K connection made.
Additional information you deem important (e.g. issue happens only occasionally):
host configurations:
in host
```
root#debian:~# ulimit -n
800000
net.ipv4.ip_local_port_range = 5000 65535
net.ipv4.tcp_mem = 8388608 12582912 16777216
net.ipv4.netfilter.ip_conntrack_max = 1000000
net.netfilter.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_count = 64515
net.nf_conntrack_max = 1000000
total used free shared buffers cached
Mem: 986M 808M 178M 8.5M 5.0M 132M
-/+ buffers/cache: 670M 316M
Swap: 1.0G 872K 1.0G
```
in container
```
ulimit -n
800000
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.netfilter.ip_conntrack_max = 1000000
net.netfilter.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_count = 6653
```
Output of docker version:
```
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:07:28 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:07:28 2017
OS/Arch: linux/amd64
Experimental: false
```
Output of docker info:
Containers: 6
Running: 2
Paused: 0
Stopped: 4
Images: 15
Server Version: 17.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 27
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 986.9 MiB
Name: debian
ID: GEUG:DUKW:FDXQ:DFUP:GOI7:S4YO:GIH7:RX55:SQ6Z:ZZFQ:MEXB:2EHP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Related

Docker - Error response from daemon: invalid header field value "oci runtime error: rootfs

I am getting error while running the below docker command.
docker run -it -v /tmp:/tmp/host registry.docker.com/base/jdk1.8 rm -rf /tmp/host/mesos
Error :-
docker: Error response from daemon: invalid header field value "oci
runtime error: rootfs
(\"/tst/app/docker/devicemapper/mnt/140eb7aadfb7e1530a3fc5955f57ae154008be913cf920ba908cf064d5e586cc/rootfs\")
does not exist\n".
Docker version result :-
docker version
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 1512168
Built: Wed Jan 11 09:49:56 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 1512168
Built: Wed Jan 11 09:49:56 2017
OS/Arch: linux/amd64
Docker Info Result :-
docker info
Containers: 90
Running: 0
Paused: 0
Stopped: 90
Images: 5
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-249:1-1838266-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop2
Metadata file: /dev/loop3
Data Space Used: 4.219 GB
Data Space Total: 107.4 GB
Data Space Available: 43.82 GB
Metadata Space Used: 8.79 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.139 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /tst/app/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /tst/app/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.117-RHEL6 (2016-12-13)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 4.1.12-103.10.1.el6uek.x86_64
Operating System: Oracle Linux Server 6.9
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.55 GiB
Name: host.com
ID: GL42:TV5D:F5NA:3RIU:JW3A:W62C:O4RF:DBIV:WBVZ:U2Q4:FGC7:XX3Q
Docker Root Dir: /tst/app/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8

HTTP port exposed but not responding properly

Running Docker 1.9.1 on OpenSuse 42.1 (can't upgrade at this time but also the same problem on Docker 1.12).
I have a Docker images that exposes a web application on port 8080. I can ssh into the container and access the application correctly from inside the container on http://localhost:8080/.
I run the container using
docker run -p 10000:8080 ...
and I would expect the web application being accessible on either
http://localhost:10000 or http://public-ip:10000
The port is open but the connection is being closed immediately.
sudo netstat -nap|grep LISTEN|grep -v LISTENING|grep 1000
root's password:
tcp 0 0 :::10000 :::* LISTEN 17375/docker-proxy
I can telnet to port 10000 but as said the connection is being closed immediately
docker --version
Docker version 1.9.1, build a34a1d5
docker version
Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built:
OS/Arch: linux/amd64
Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built:
OS/Arch: linux/amd64
docker info (1.9)
Containers: 23
Running: 1
Paused: 0
Stopped: 22
Images: 198
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-8:2-1074640197-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 13.22 GB
Data Space Total: 107.4 GB
Data Space Available: 94.16 GB
Metadata Space Used: 20.58 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.127 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.136 (2016-11-05)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: null bridge host overlay
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Security Options: seccomp
Kernel Version: 4.11.8-200.fc25.x86_64
Operating System: Fedora 25 (Server Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 6
Total Memory: 29.45 GiB
Name: dev2.zopyx.com
ID: DE7T:KEX4:UCRB:KOED:2DHE:3437:MYDS:LPQT:BBYO:SOQH:2EZL:VTYR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure)
docker info (1.12)
Containers: 23
Running: 1
Paused: 0
Stopped: 22
Images: 198
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-8:2-1074640197-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 13.22 GB
Data Space Total: 107.4 GB
Data Space Available: 94.16 GB
Metadata Space Used: 20.58 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.127 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.136 (2016-11-05)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host overlay null
Swarm: inactive
Runtimes: runc oci
Default Runtime: oci
Security Options: seccomp
Kernel Version: 4.11.8-200.fc25.x86_64
Operating System: Fedora 25 (Server Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 6
Total Memory: 29.45 GiB
Name: dev2.zopyx.com
ID: DE7T:KEX4:UCRB:KOED:2DHE:3437:MYDS:LPQT:BBYO:SOQH:2EZL:VTYR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure)
What could be the problem here?
Solution: the webservice internally was only binding to 127.0.0.1 instead to 0.0.0.0

Docker command with build / run / etc. "too many open files"

Im installed Docker on server machine (with Ubuntu 16.04), but trying run any commands, I get this error:
"Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: too many open files"
Please, who know how solve this issue?
Docker Info:
Containers: 18
Running: 0
Paused: 0
Stopped: 18
Images: 4
Server Version: 17.03.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 76
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-ini
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953 GiB
Name: web.renderdeal.com
ID: LXNZ:I6HH:ZVBB:KS3V:3WCT:ADWY:C2MZ:QJ37:VZUU:EZ6T:PYWO:66WQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: animarender
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
/Docker Info
docker version
Client:
Version: 17.03.0-ce
API version: 1.26
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 11:02:43 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.0-ce
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 11:02:43 2017
OS/Arch: linux/amd64
Experimental: false
/ docker version
Considering Docker status seems OK, check if the issue is on your side (as in this ticket)
Let's check on the current open files
lsof | grep "rclone" | wc -l
returns 6458
Now, Check on current ulimit
ulimit -n
returns 1024
Set that to 9000 and Check again
ulimit -n 9000 && ulimit -n
returns 9000
To make the ulimit more persistent you can either edit /etc/security/limits.conf
and add :
* soft nofile 9000 + reboot
or write it into your user's .bashrc

Docker container can't resolve DNS to reach another AWS Ec2 Machine

I am not able to ping another machine/host App2 by resolving the DNS from the container running on host App1. Though the /etc/resolv.conf is same as that of host. I am making use of AWS Route 53 private hosted DNS to allow intercommunication by resolving DNS and not IPs.
Some basic info for this :
ubuntu#app1:~$ docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 10
Server Version: 1.13.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 31
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 3.13.0-106-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.797 GiB
Name: app1
ID: 6GYC:GI6M:JNTM:MMSL:7LRD:BEUZ:RTRD:Q4AG:NEQU:XC5C:ALOK:N3LM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
############################################
ubuntu#app1:~$ docker version
Client:
Version: 1.13.1
API version: 1.26
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:42:29 2017
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:42:29 2017
OS/Arch: linux/amd64
Experimental: false
###########################################
ubuntu#app1:~$ docker exec -it conatiner1 sh
/data # ping app2
ping: bad address 'app2'
/data # ping app2.mydomain
PING app2.mydomain (10.xx.xx.xx): 56 data bytes
##############################################
resolv.conf on conatiner
/data # cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.xx.xx.xx
search mydomain
resolv.conf on host
ubuntu#app1:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.XX.XX.xx [ same as of container ]
search mydomain
From the docker host i am able to ping App2 wihtout giving full domain like app2.mydomain but same is not working from the container
When you call docker run, please add --net=host option to use host's network stack. It will do the trick.

docker-proxy - Error starting userland proxy while trying to bind on 443

I'm trying to install discourse with docker in an Ubuntu 16.04 LTS with Apache listening to port 80 and 443.
When I try to lunch the app I get the following error:
starting up existing container
+ /usr/bin/docker start app Error response from daemon: driver failed programming external connectivity on endpoint app
(dade361e77fbf29f4d9667febe57a06f168f916148e10cc1365093d8f97026bb):
Error starting userland proxy: listen tcp 0.0.0.0:443: listen: address
already in use Error: failed to start containers: app
For what I'v found docker-proxy is the one that is trying to bind on 443.
How can I solve this?
Some details...
docker version
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
docker info
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 4
Server Version: 1.11.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 25
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.4.0-28-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.39 GiB
Name: sd-12345
ID: 6OLH:SAG5:VWTW:BL7U:6QYH:4BBS:QHBN:37MY:DLXA:W64E:4EVZ:WBAK
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
perhaps, stop apache? – vitr Jul 22 '16 at 2:56
^^^ This comment from vitr should be the Accepted Answer:
Docker cannot proxy a service from within a container to the port on the host without first stopping any services that are already using that port.
In this case, Apache must be stopped with a command such as sudo service apache2 stop.
Then docker start app can then be run and docker should do its thing unhindered.
See the related question: docker run -> name is already in use by container
Edit /etc/docker/daemon.json and add:
{
"userland-proxy": false
}

Resources