Docker: Filesystem is not responding - docker

Question

I have to run 43 containers on the local machine.
Each startup I have warnings
WARNING: Connection pool is full, discarding connection: localhost
Warning don't block my work
But sometimes at containers startup, I got multiple errors below
Filesystem is not responding
Filesystem mounted at /var/lib/docker/overlay/{hash}/merged is not responding
A lot of them pushed as notifications in my manjaro KDE
These errors block my work, containers not working. Only reboot helps me.
I will provide any additional information you need
Filesystem is ext4
Storage driver: overlay2
docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 16 10 3.82GB 1.198GB (31%)
Containers 43 43 1.138MB 0B (0%)
Local Volumes 20 10 1.947GB 955.4MB (49%)
Build Cache 0 0 0B 0B
sudo docker network ls
NETWORK ID NAME DRIVER SCOPE
3e046a2f4038 bridge bridge local
8aadcba2673b dev_default bridge local
0f58cf29be50 host host local
dace3c787c02 none null local
sudo docker network inspect dev_default
[
{
"Name": "dev_default",
"Id": "8aadcba2673b48477a439252f9528493511d63d017acbe7981893e9abc8ef1ce",
"Created": "2020-02-14T13:40:06.643398755+03:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.20.0.0/16",
"Gateway": "172.20.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"016e0d8ec547831d3cb4ed971ccc9d4dd7c21d7bbd17d2295d9b6c2f37250ce1": {
"Name": "dev_emulator-22800014_1",
"EndpointID": "4d665c749b325820aac33d415baa8467e49b549c928d730b0fd16a01423f94c9",
"MacAddress": "02:42:ac:14:00:0a",
"IPv4Address": "172.20.0.10/16",
"IPv6Address": ""
},
"060a555c3a813188d49c06d22170eab6555b4acf6f532bf3c7dd7ea82f01b5fb": {
"Name": "dev_emulator-22800038_1",
"EndpointID": "85a43daca97c55f2ea2c83401468ba64315997f8c817d22fc81a2b54283236dc",
"MacAddress": "02:42:ac:14:00:10",
"IPv4Address": "172.20.0.16/16",
"IPv6Address": ""
},
"06aba1b6a003e42388fd4d4feaa66ab8f2fbab853bf32bf74ef50050cba30ae6": {
"Name": "dev_emulator-22800015_1",
"EndpointID": "2541a4e7d05c88ea2a6bccc3280b5eac8437646ed07fa13d6bd00a65aae0331d",
"MacAddress": "02:42:ac:14:00:12",
"IPv4Address": "172.20.0.18/16",
"IPv6Address": ""
},
"085d84226f8aa471ccbf784f7c34be362371d481d7a157ec87538c631c85868a": {
"Name": "dev_emulator-19400017_1",
"EndpointID": "0bef199879d04a55e0916d943cef1e883cf0032a32fc7c6ff6cc71bc2eb5d397",
"MacAddress": "02:42:ac:14:00:15",
"IPv4Address": "172.20.0.21/16",
"IPv6Address": ""
},
"09fbe0b08f81f1b30a056bcddbf02bafe50f5367447eea52393d6f85fe9e6e95": {
"Name": "dev_emulator-19400019_1",
"EndpointID": "0ab5142fa4f578b23600e0a1b00433009368a82efad15cb7b29273095b4ddb81",
"MacAddress": "02:42:ac:14:00:20",
"IPv4Address": "172.20.0.32/16",
"IPv6Address": ""
},
"160f645c05d1a7ca3c7189a24cbd1aa598ae64a0eced259580e5475544556e99": {
"Name": "dev_emulator-22800016_1",
"EndpointID": "3b376a61b5bd383b4ac92ad959bce1d51fa8f189bb8dc54c086ad0e7ad0c0ee3",
"MacAddress": "02:42:ac:14:00:02",
"IPv4Address": "172.20.0.2/16",
"IPv6Address": ""
},
"18d5f68d2f5d5fc2d39ecad9d16bdb75af91d74626ac77c559fd508c2e75e1e8": {
"Name": "dev_emulator-19400031_1",
"EndpointID": "771bb76578e24aba74f4780b5e3f02d7aa03bd1e6c6b0311ca8a4e66fcd00ea3",
"MacAddress": "02:42:ac:14:00:16",
"IPv4Address": "172.20.0.22/16",
"IPv6Address": ""
},
"23c427c2b75f27b79f2fe83eb9d361d506786fc2f702d5b854d443d14683b0da": {
"Name": "dev_tdiRelay_1",
"EndpointID": "00331149f91abef709e1eb494a937cbe534ef4bfa5da137edb648e748ff800e0",
"MacAddress": "02:42:ac:14:00:2b",
"IPv4Address": "172.20.0.43/16",
"IPv6Address": ""
},
"272c718eaa25a45952eceb860d875418b6ba6a68db026733052e67b7efa3ab8d": {
"Name": "dev_tdiParser_1",
"EndpointID": "02fecfde12e183c6ece57f1ab96b858fb4e3b0f1977a47a00f61ce9ece7960fa",
"MacAddress": "02:42:ac:14:00:2c",
"IPv4Address": "172.20.0.44/16",
"IPv6Address": ""
},
"29e9a0e3fa9eff5185fc6dda092f4358140c4ca852d97f31c31c77055b1ffe60": {
"Name": "dev_emulator-19400013_1",
"EndpointID": "4a501dcf83ba64273512a5381c562abbdc5dba0d0b6d01fe68a03499674e2f15",
"MacAddress": "02:42:ac:14:00:0f",
"IPv4Address": "172.20.0.15/16",
"IPv6Address": ""
},
"2cb4e8011fa4c338803f546dfb9181936030cae6d3d4eddddd7af0a04137f2b6": {
"Name": "dev_emulator-22800010_1",
"EndpointID": "0dc1d7639077e18c511b75638f9995e5a052914da6e14bca174fd6a8217ca2d4",
"MacAddress": "02:42:ac:14:00:17",
"IPv4Address": "172.20.0.23/16",
"IPv6Address": ""
},
"408a7b4f1f1c92be7022f514775ebc93fd6359237b74704da73ee51b158cbfe2": {
"Name": "dev_emulator-22800031_1",
"EndpointID": "380cfac2b1200c530d5c665b5290d8930448ed39c56710d29fe191af5e748d7d",
"MacAddress": "02:42:ac:14:00:0d",
"IPv4Address": "172.20.0.13/16",
"IPv6Address": ""
},
"43bda6ed5698b59d90fa1479547efa2c4bbcec187b03d4990632217bbb4cec9c": {
"Name": "dev_emulator-22800018_1",
"EndpointID": "c77dac271dedc5d05968a17fbf665358ba214d03a67dca082c37668410d0f1a2",
"MacAddress": "02:42:ac:14:00:18",
"IPv4Address": "172.20.0.24/16",
"IPv6Address": ""
},
"44a1e95f87f1eac91bfc06a16a32b8b478df871af9b9d1d9bf095e6f2b075e5a": {
"Name": "dev_maintempdb_1",
"EndpointID": "4021ea4e27729adeaa81b7d0bd6f7c67be17734c4d67b14d660880f7c0531f37",
"MacAddress": "02:42:ac:14:00:05",
"IPv4Address": "172.20.0.5/16",
"IPv6Address": ""
},
"554f1a422ece3eb0558a747ef6d63dba6b0e9d019c438d118ded9ff76ef176df": {
"Name": "dev_emulator-19400040_1",
"EndpointID": "f2a3bb793ab5be3e7c1276d38b3ad3a922ea0a901fa990e356a722875aa1556a",
"MacAddress": "02:42:ac:14:00:14",
"IPv4Address": "172.20.0.20/16",
"IPv6Address": ""
},
"5932a2cb3803e390ce98ef823610e6f3ca12213e52b9812c2d4b5519b473e709": {
"Name": "dev_emulator-22800023_1",
"EndpointID": "81b45bb2b1832a83dab9dfc803e44a724ffd5a6a1bc6e11fc4577b6489b090ed",
"MacAddress": "02:42:ac:14:00:04",
"IPv4Address": "172.20.0.4/16",
"IPv6Address": ""
},
"596314ba1bebbb5938735e7559a2cad8a3927a0b222c9a42f6a2abab2184779e": {
"Name": "dev_mainq_1",
"EndpointID": "9bc8505918c5bef6132172e90c5b50e4801a213e5fb7f18c54f8a841e0deee26",
"MacAddress": "02:42:ac:14:00:22",
"IPv4Address": "172.20.0.34/16",
"IPv6Address": ""
},
"61dea1acffa8eb3508bbf4c90d17fc01b32c7da578682555343d053ca65add04": {
"Name": "dev_outerdb_1",
"EndpointID": "4613444ab9c6e33dc6eb35ff0d5400a8d34b1ceeb30d8e1bb2f607ec9fb953db",
"MacAddress": "02:42:ac:14:00:23",
"IPv4Address": "172.20.0.35/16",
"IPv6Address": ""
},
"666aa5be6401cd4ac2fe34a74789e0a5e210c46fc43a569448f6f51cc255fa61": {
"Name": "dev_outerOctave_1",
"EndpointID": "fc3c5b47a330a68e7b0ff494f1fde3926b429158a1846197d31b11b0aae083cd",
"MacAddress": "02:42:ac:14:00:29",
"IPv4Address": "172.20.0.41/16",
"IPv6Address": ""
},
"6f3533a85093ddde99cac38201ab379bbba2948f08da50a4a173be982cea7c2a": {
"Name": "dev_emulator-22800040_1",
"EndpointID": "9615b87b9cea2f94ba46b0005d4b73c31cf358446cf3bf3ec94894a25a12241a",
"MacAddress": "02:42:ac:14:00:0e",
"IPv4Address": "172.20.0.14/16",
"IPv6Address": ""
},
"7100bd89110864bc254023ffb53bab12883897c42ba7cd5adb29a8b894150f20": {
"Name": "dev_emulator-22800033_1",
"EndpointID": "2bb75565256cb335c1f076e2e4ef4acc5623e0d0bdf4aa1b7c0514fa4eb04f51",
"MacAddress": "02:42:ac:14:00:09",
"IPv4Address": "172.20.0.9/16",
"IPv6Address": ""
},
"801f8d9ddeebf4c1e5f692797cdd65650cd09e05b23eab4e7886cc183c79b6c3": {
"Name": "dev_emulator-19400016_1",
"EndpointID": "e0b0f28260a5a03c79b7f41f1bf8db9640848c6d0aed35d6fdd53cba03d61366",
"MacAddress": "02:42:ac:14:00:0b",
"IPv4Address": "172.20.0.11/16",
"IPv6Address": ""
},
"809d22241a480f9192f29edc5d03e79db33043cc9bfa763639e67e8cfdd80f0e": {
"Name": "dev_emulator-19400015_1",
"EndpointID": "638e6b1cddd1fe78a11de8622ab42ca8ee967303330277e2187261b426aa2f8b",
"MacAddress": "02:42:ac:14:00:1e",
"IPv4Address": "172.20.0.30/16",
"IPv6Address": ""
},
"86995a099365f63f6112c25e3e784f6610ae278702c812b61370e186ca53e2ad": {
"Name": "dev_s3storage_1",
"EndpointID": "e4c8299d772eaa6a57a63bd351585ec5785c09f6df2e785730ae41d4ff3a25f2",
"MacAddress": "02:42:ac:14:00:19",
"IPv4Address": "172.20.0.25/16",
"IPv6Address": ""
},
"8b8e754c99ff6226e8ccd03a150173dfb06e6309c9774c263ae006d9f53f52aa": {
"Name": "dev_emulator-22800022_1",
"EndpointID": "dbd31d8e3150192e87b8917927d07729e0603f5288dcc9f296650f34f06c37ee",
"MacAddress": "02:42:ac:14:00:1c",
"IPv4Address": "172.20.0.28/16",
"IPv6Address": ""
},
"8cf73e2e2503e52bfcafbc1c9322b994127df20c359ca624be644210abd57dbc": {
"Name": "dev_tileserver_1",
"EndpointID": "77374f4cde143d455c0f062e673008c8b41fe4b2977392c9982bc0052217c6c3",
"MacAddress": "02:42:ac:14:00:03",
"IPv4Address": "172.20.0.3/16",
"IPv6Address": ""
},
"8e05504ce48d890fa9359938c533b4495814e67f36c8c5b24a736812a5eb0e2f": {
"Name": "dev_emulator-19400028_1",
"EndpointID": "9d26d053da0d0945d3ff707bccba3cbb12fb81f295678c3604ad60eb9aa9ef2c",
"MacAddress": "02:42:ac:14:00:1f",
"IPv4Address": "172.20.0.31/16",
"IPv6Address": ""
},
"968b41556187c66cb31e623b063d3acf594eb6abac0aa5444bc85d017a807ba6": {
"Name": "dev_outerWeb_1",
"EndpointID": "1959455650da340d1d59f8d6011b88657371d7e7e2db02db69462b6747835f97",
"MacAddress": "02:42:ac:14:00:28",
"IPv4Address": "172.20.0.40/16",
"IPv6Address": ""
},
"9a12736c89c8005b32ff9f11283dfbd666ad201fbd9a45bb1c5ad0f20b5f52eb": {
"Name": "dev_emulator-19400011_1",
"EndpointID": "c23c6ac83769cf06ccc856a47aa0859a9da3f942e7e792384842dde54564e463",
"MacAddress": "02:42:ac:14:00:25",
"IPv4Address": "172.20.0.37/16",
"IPv6Address": ""
},
"9d12ef78d15fec44b4fa42878c4268d1ec4ac21db4f694bc02d208acd1b561df": {
"Name": "dev_emulator-19400012_1",
"EndpointID": "e0f1c3f68bdfa4e7205b6ea8acad4ec791393139e54fc823b9cc7e079c2e419a",
"MacAddress": "02:42:ac:14:00:07",
"IPv4Address": "172.20.0.7/16",
"IPv6Address": ""
},
"b2958a80d70e47692c9ac69314b4fd69acc3e18760f644dceb8142d23d046b3a": {
"Name": "dev_emulator-22800007_1",
"EndpointID": "c285a1c6e276ff364bbb72d60ba94eea247aa524d6e9477de8fb92f60b4a8f83",
"MacAddress": "02:42:ac:14:00:0c",
"IPv4Address": "172.20.0.12/16",
"IPv6Address": ""
},
"b7f4d47981571d0970c1db4dc065a1c93186287b50938b1f1e5acc972678735b": {
"Name": "dev_emulator-19400014_1",
"EndpointID": "8625c39b48c0000b79c13b1d1f21ba07ce16b8813cf786a94ceedc8a84cda55d",
"MacAddress": "02:42:ac:14:00:06",
"IPv4Address": "172.20.0.6/16",
"IPv6Address": ""
},
"c08e59d978c478cd6bb8c30b1ff0dc51b10ac10fc616e6fc24d3600391d4b3da": {
"Name": "dev_emulator-19400020_1",
"EndpointID": "1b629f69d2364e39375cd4cb4e42bd5e34332e468b04169a7fcaa167d58f3f19",
"MacAddress": "02:42:ac:14:00:24",
"IPv4Address": "172.20.0.36/16",
"IPv6Address": ""
},
"c84a447df4e5b632e042e567033d243017b6f0ba58f97be1222110d50b1019b3": {
"Name": "dev_emulator-19400023_1",
"EndpointID": "642b31dd4818063828c9889237416aa7b297391fe557530e11959c971278d0dd",
"MacAddress": "02:42:ac:14:00:26",
"IPv4Address": "172.20.0.38/16",
"IPv6Address": ""
},
"ca15d4fee3f45b8d8a7552aaa519563d2b1b8ea13c51c0c0d1cd94765cf94512": {
"Name": "dev_emulator-19400009_1",
"EndpointID": "8a62973b92ad70a6aad08b717c1ff26fd781cd97e341d2d8ed753a2d2294d00e",
"MacAddress": "02:42:ac:14:00:11",
"IPv4Address": "172.20.0.17/16",
"IPv6Address": ""
},
"cb4fd70cdb8dac59bff2c0257224dc1d261df2770b50221132e5a23997afd1d6": {
"Name": "dev_maineternaldb_1",
"EndpointID": "d4cec9ce4110dd188bf7c958b89433d40d436b46f212fbdf35c0dcbcdf0831b9",
"MacAddress": "02:42:ac:14:00:08",
"IPv4Address": "172.20.0.8/16",
"IPv6Address": ""
},
"e4d60f497831a9c38ebf96a2d48aed8bfb26928a2ed76ba2b8e77139c4b832bc": {
"Name": "dev_emulator-22800004_1",
"EndpointID": "61a18f0b561336186ca3bf01a8b3439396f9c76e3ba76b28c7ec355441be2dd9",
"MacAddress": "02:42:ac:14:00:21",
"IPv4Address": "172.20.0.33/16",
"IPv6Address": ""
},
"e79451e6ad59896d21fcd88301ce099917ef96ea3d32a3060cb79978b6421fcf": {
"Name": "dev_emulator-19400036_1",
"EndpointID": "e3303c42d1b0ebac104762b989229773501b3a8e5b5c7059c765543e3031e8b6",
"MacAddress": "02:42:ac:14:00:1a",
"IPv4Address": "172.20.0.26/16",
"IPv6Address": ""
},
"ead3cffd714f12160dde8e35bae0222b521d6264338267edf9c00bbb11b0ebde": {
"Name": "dev_mainOctave_1",
"EndpointID": "e5f4c078e4b1ed13fd52eb5740f38c59760388a8bed6b01549841a649db6fb94",
"MacAddress": "02:42:ac:14:00:2a",
"IPv4Address": "172.20.0.42/16",
"IPv6Address": ""
},
"ec9aeb9218cef0b8176e3713258812dfcfba5baa691b9639e50fe965b5681cea": {
"Name": "dev_outerq_1",
"EndpointID": "69756ebf2b6a1bdceff4f37fe1dad22859983f417bdbe8e4ab82bd23668500fc",
"MacAddress": "02:42:ac:14:00:13",
"IPv4Address": "172.20.0.19/16",
"IPv6Address": ""
},
"f2b86f1ae769358a413c94d6bc7c129e50cdf2a6afe12d86be1e5b85fce6cad7": {
"Name": "dev_web2_1",
"EndpointID": "0efa83f1aaef42831b312582fbd7ac25c202f7650d237ea30c72de01eaffd1a8",
"MacAddress": "02:42:ac:14:00:27",
"IPv4Address": "172.20.0.39/16",
"IPv6Address": ""
},
"f3ed055ec5f6a6bf6ab73b6db24a495e531e2078694ad523723880e9ae7627e7": {
"Name": "dev_emulator-77700000_1",
"EndpointID": "a3ecfc5de45ba1bc0dd491f6dc4052034b077d0e3f717f182751b2a29e42e850",
"MacAddress": "02:42:ac:14:00:1d",
"IPv4Address": "172.20.0.29/16",
"IPv6Address": ""
},
"fca06c99c6f6140161a007bab98e885f468a40a21318f9b7be2fdec322645db9": {
"Name": "dev_maincouchdb_1",
"EndpointID": "45b118cabc4a5ca41a968f3d382045a9a1b62939f0a871cedb64a3f0068a0aec",
"MacAddress": "02:42:ac:14:00:1b",
"IPv4Address": "172.20.0.27/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
sudo docker network inspect bridge
[
{
"Name": "bridge",
"Id": "3e046a2f4038ab5cf2c2989c7c7adb2bb2f75cd75fef18d133cc7d6952b8fa72",
"Created": "2020-02-14T13:25:50.571033335+03:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
sudo docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 117
Server Version: 19.03.5-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d50db0a42053864a270f648048f9a8b4f24eced3.m
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.5.2-1-MANJARO
Operating System: Manjaro Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.33GiB
Name: gemini-f5
ID: EXYX:H5OE:U74F:HHTF:6P4M:B7HJ:YEXR:S34A:PPXJ:W56Z:BI3D:2DXI
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
dockerregistry.smart-dn.ru:5000
127.0.0.0/8
Live Restore Enabled: false

Answer 1

hi we have same problems sometimes when using nfs share volumes,sometimes on filesystem,all depend on how many volumes you have.
Simple answer is docker is not build to start up lots of containers.
Our production server when we make once per mont maintance (rebooting) starting 100 containers somtimes even 1 hour. producing not only filesystem (IO read/write) errors.
Try to find a way to block these
pushed as notifications in my manjaro KDE
containers will eventualy start up
we didnt find any solution
our stable workaround was rancher.com and another servers. Rancher helped in way of self distributing containers to server with low resource usage.

Answer 2

I suspect this has s.th. to do with Python's request library like described in docker-compose restart connection pool full. I suggest to try the solution and search for the setting DEFAULT_POOLSIZE and set it to a higher value.
On my mac this parameter can be found in /usr/local/homebrew/lib/python3.7/site-packages/requests/adapters.py.
This file may of course be overwritten as soon as you update that library however I did not found another way to set this parameter.