Can't use more than 32 Docker containers (networking issues) - docker

Question

I am currently setting up a multi-agent system in Docker, such that each agent is running in a separate container and communicates with the other agents within the Docker network.
Everything works fine, if I start less than 32 containers.
If I run more than that, the agents are not able to communicate anymore.
Further, the network on my host machine behaves strangely and I loose internet connection. The containers themselves start without any issues, though (only communication doesn't work). If I stop all containers, my host machine is connected to the internet again.
First, I create a custom Docker network:
docker network create --driver bridge agent_network --subnet 10.10.1.0/24
I made sure that the IP range is not used by other services on my machine.
Next, I specify the custom network in my docker-compose.yml file which also contains my agents as services:
version: "3.8"
networks:
default:
external: true
name: agent_network
services:
agent:
...
Finally, I run the system using the command:
docker-compose up --build --scale agent=32 agent
Such that 32 instances of the agent as defined in the docker-compose.yml file are started.
The agents start successfully and there is no error from Docker concerning network issues. Only problem is, the agents don't receive any messages.
Does anyone has a solution to this problem? Many answers I found online were considering using the default bridge of Docker and mainpulating the address space. However, this doen't work either in my case and it's recommended to not use the default bridge by the Docker docs.
Also editing the /etc/docker/daemon.json file to expand the address space did not work.
If it helps, here are some technical details of my system:
Host machine OS: Ubuntu 22.04.1 LTS
Host machine IP: 192.168.0.34
Docker: 20.10.22
docker-compose: 1.29.2
Update 1:
Below is the output of the command docker network inspect agent_network when the containers are running (43 containers were started):
[
{
"Name": "agent_network",
"Id": "7e5a4b6abe02d01144690f62dad120660d11319abd34b3dc62de37ce3f0c2a0e",
"Created": "2022-12-22T16:24:58.776012944+01:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "10.10.1.0/24"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"03fb10e0d4e0c1b28053ebb80294029f51018a91e84dadb266a8872be45d19ac": {
"Name": "docker_iwt_agent_17",
"EndpointID": "24c52fab24963827d02fa844f1c15a7b3d8296d9832b5b1f63cac1957a456a67",
"MacAddress": "02:42:0a:0a:01:05",
"IPv4Address": "10.10.1.5/24",
"IPv6Address": ""
},
"05c91dabd601118ea8269d77756b69236eddd0d97918633542b5640cbc65738c": {
"Name": "docker_iwt_agent_15",
"EndpointID": "efc0abf1397e2e55b0496aedfdb4619edaf38213eb8ee952186a5953bdf05d43",
"MacAddress": "02:42:0a:0a:01:28",
"IPv4Address": "10.10.1.40/24",
"IPv6Address": ""
},
"09cca2d89486b163314c402efa3e72a13a41e9c046837bec9d1308c63bd8288b": {
"Name": "docker_iwt_agent_1",
"EndpointID": "565d6010c9919eddfd98a937a532200d2477f87a8f056627f6b2abce8647a98e",
"MacAddress": "02:42:0a:0a:01:10",
"IPv4Address": "10.10.1.16/24",
"IPv6Address": ""
},
"0b3f43c0f6d250619c5051a76549f7f2f70014e020cefa4816eabb3adb5fb885": {
"Name": "docker_iwt_agent_8",
"EndpointID": "10b691061cd220b8dfc030d3641512a9fba3bb9c08b9c4d3514f7f661eaffde2",
"MacAddress": "02:42:0a:0a:01:03",
"IPv4Address": "10.10.1.3/24",
"IPv6Address": ""
},
"0b776566285f7be530ea91a4fd2295a459d4e27130e19f8e3181f1bb302ac493": {
"Name": "docker_iwt_agent_25",
"EndpointID": "cfc21fdee01fdfaba8d7e08f1f8c6e6a520c464b6fca219001119416b51ec906",
"MacAddress": "02:42:0a:0a:01:16",
"IPv4Address": "10.10.1.22/24",
"IPv6Address": ""
},
"25b4d77eaf2a249bd516131565621ee41edf85e9cd66ef2950657de88cf80d21": {
"Name": "docker_iwt_agent_11",
"EndpointID": "203333983f3ec9382da87212e48b3afe4ba890e86c9159b1b58bc94d6e9c3d89",
"MacAddress": "02:42:0a:0a:01:0c",
"IPv4Address": "10.10.1.12/24",
"IPv6Address": ""
},
"2654ba6e89d0153e59f2de477a787ab5b3064dd99284949d4db66d6bd659306d": {
"Name": "docker_iwt_agent_24",
"EndpointID": "89e5db4f584a584272a51cdf9301ff86b0931ae0cda786355829cbe8ac68ecb4",
"MacAddress": "02:42:0a:0a:01:1d",
"IPv4Address": "10.10.1.29/24",
"IPv6Address": ""
},
"26a6b5d80abadcf9b27f5606d9e53ef44e25f11bc85dc2c651e8c3034af9b122": {
"Name": "docker_iwt_agent_34",
"EndpointID": "6c8de526c126eb62fd4f01a21b01a3aba12963ae7e684102babcdf7c32fa7858",
"MacAddress": "02:42:0a:0a:01:2b",
"IPv4Address": "10.10.1.43/24",
"IPv6Address": ""
},
"2fda699d23ddf42546eefdc21b22b984f186eee10c9d087958d4fb4303b8a343": {
"Name": "docker_iwt_agent_41",
"EndpointID": "6410170b70bb49bf83d31c22e7eb3e0c8c8a79063f0705956e8a26683a4ab45b",
"MacAddress": "02:42:0a:0a:01:0b",
"IPv4Address": "10.10.1.11/24",
"IPv6Address": ""
},
"38f653bb69ea04d1424b64bacf258f66380c3a276bbd3df50cb45b6f73eae5d5": {
"Name": "docker_iwt_agent_20",
"EndpointID": "13bd9bba90ceeda0de8e8bdc9d2a075e0f1dba86ff94ff5ab0f7517733ca67c7",
"MacAddress": "02:42:0a:0a:01:14",
"IPv4Address": "10.10.1.20/24",
"IPv6Address": ""
},
"473e179b73f32b289435f3e3cbbb7f9a7034635d703fd66f7293893eabfb8802": {
"Name": "docker_iwt_agent_22",
"EndpointID": "22ce17c3dc719142bfa101d49583a2a0f71e1ec01a19c881621ebe66e8b4e211",
"MacAddress": "02:42:0a:0a:01:1f",
"IPv4Address": "10.10.1.31/24",
"IPv6Address": ""
},
"581f91b16bc22949de0e60a85669c909c8748e5438a35b01ad10b8c3d18e5bce": {
"Name": "docker_iwt_agent_6",
"EndpointID": "b0c5ea84bd9d5e9a642d1b7317e5f8792fe28204bf88bde503273e4df2227b40",
"MacAddress": "02:42:0a:0a:01:20",
"IPv4Address": "10.10.1.32/24",
"IPv6Address": ""
},
"58a966dd9478139d34ffa94355fe1daee168d7863cf211423e7f643f9f0b104f": {
"Name": "docker_iwt_agent_40",
"EndpointID": "f9e4efcc927fb4966d5b54c0210dbcf1f595222a1fb5f0c0607896dbb33d98e6",
"MacAddress": "02:42:0a:0a:01:21",
"IPv4Address": "10.10.1.33/24",
"IPv6Address": ""
},
"5f63e021e176fcc714b9c9b4d7579c38467cfe6ef23d97038d2c8ee84c252028": {
"Name": "docker_iwt_agent_35",
"EndpointID": "c3be1cd49404b5b60e63a37601e50d09928c7caedc5d4697fac862fa4fc9d50b",
"MacAddress": "02:42:0a:0a:01:0e",
"IPv4Address": "10.10.1.14/24",
"IPv6Address": ""
},
"6038fa2517624496e7fb5248841f5e1d28c22173ffeba9c1124ef4177e7761ed": {
"Name": "docker_iwt_agent_21",
"EndpointID": "b45812055779b2fc56e5a6c9765fef05b374089b626725f6e1f915661179d10c",
"MacAddress": "02:42:0a:0a:01:29",
"IPv4Address": "10.10.1.41/24",
"IPv6Address": ""
},
"62f24f4a5c3ad9dbd8b604c4eb6d5a0e8fd8ce539008a486e71c4d6716cfd976": {
"Name": "docker_iwt_agent_13",
"EndpointID": "7bc2201c2a9b602d708e609882b5c6bfbbae7df0db9a341f4633deae6d37ab42",
"MacAddress": "02:42:0a:0a:01:0f",
"IPv4Address": "10.10.1.15/24",
"IPv6Address": ""
},
"6a2a81660b72e3138ccddbfc70e8c907196217f7d5e5985e241eeb38a759477d": {
"Name": "docker_iwt_agent_16",
"EndpointID": "266df1ee74cca9e26dbe19350feff84530e37324001495c029058fffe63cceb3",
"MacAddress": "02:42:0a:0a:01:12",
"IPv4Address": "10.10.1.18/24",
"IPv6Address": ""
},
"6aea9c53022d5155d05f8013091d001187afa968fb04e0d1ab0d987371b1f1ce": {
"Name": "docker_iwt_agent_43",
"EndpointID": "90b1e7e4ff057207d6ad11b1d0e46793f9b849c953508c02cfbb2b9cf5b52c56",
"MacAddress": "02:42:0a:0a:01:19",
"IPv4Address": "10.10.1.25/24",
"IPv6Address": ""
},
"6e4847a8d50745a7b9bad71691367379c09e6cce8ee8117fffff0be19cb8c97f": {
"Name": "docker_iwt_agent_31",
"EndpointID": "8c5630ca5d6697140b316b0ab8e92c98eb707612eefa29002fae077525d6ac6a",
"MacAddress": "02:42:0a:0a:01:18",
"IPv4Address": "10.10.1.24/24",
"IPv6Address": ""
},
"80aa000edc914d7ea8d9bbc3288eb5e61e1c200643b043deff704c4d7b6ccdaf": {
"Name": "docker_iwt_agent_2",
"EndpointID": "a3e6fb766d3ed3b23b7447d0524a791e8b70e2e8c558f9f3be46d6e2ee35e050",
"MacAddress": "02:42:0a:0a:01:04",
"IPv4Address": "10.10.1.4/24",
"IPv6Address": ""
},
"85ef36d28f6decbb58f8140fc78f96dc33acf39142f00224baaa0bdede26b77a": {
"Name": "docker_iwt_agent_12",
"EndpointID": "35a6e014b1231f5f5b9eece616543f600d41a40036a50e373617e7801bf42593",
"MacAddress": "02:42:0a:0a:01:27",
"IPv4Address": "10.10.1.39/24",
"IPv6Address": ""
},
"876e22b8982b85eca3e4d3e4f1e621b206f74258d122c0f80586fa4e8e71e825": {
"Name": "docker_iwt_agent_14",
"EndpointID": "df91e7a3901cd70b644deb51ad25ca75106b2bfc109d9285978d5058c8d7de45",
"MacAddress": "02:42:0a:0a:01:09",
"IPv4Address": "10.10.1.9/24",
"IPv6Address": ""
},
"8bc187fd30125562970e9cd20ce3b79f55cb4a251a79683140b792ee2aa26437": {
"Name": "docker_iwt_agent_42",
"EndpointID": "e991d3aae99c93d4fb2475613f033ef020062c81696f07f3d032846baa646008",
"MacAddress": "02:42:0a:0a:01:2a",
"IPv4Address": "10.10.1.42/24",
"IPv6Address": ""
},
"934a0cc93ef080ee70a26cc168916e39bfd060d3edaa32a795599ba6b9520c32": {
"Name": "docker_iwt_agent_33",
"EndpointID": "f77ae0a71c0b92d2050546db3effc5f40781ff68b78d6efa5380114ae4205bce",
"MacAddress": "02:42:0a:0a:01:1c",
"IPv4Address": "10.10.1.28/24",
"IPv6Address": ""
},
"9c1194fe4c17087b9459674625269bb648539832f72a9bd9eb8ec295938bda6a": {
"Name": "docker_iwt_agent_5",
"EndpointID": "6e9a9dcf41b252f0895a1137928c82a8c1848718c4f1a05165b1f4f571f7db58",
"MacAddress": "02:42:0a:0a:01:06",
"IPv4Address": "10.10.1.6/24",
"IPv6Address": ""
},
"9cf86420156e6cb7a897e22ddba47be3e2a03e56fb508bc224c8c7086c534f4f": {
"Name": "docker_iwt_agent_10",
"EndpointID": "39b52cb28ff118b2b66ea6b10a40ae2a09f30d9a0a335352c8520dad257d4d5b",
"MacAddress": "02:42:0a:0a:01:17",
"IPv4Address": "10.10.1.23/24",
"IPv6Address": ""
},
"9d8a52bb6ccc127d59deb410510c8daf9dbe0b8390b95f9df88f77a5f4531227": {
"Name": "docker_iwt_agent_29",
"EndpointID": "74a37f1cda8d866d31307b9724de8b20c3586369b595004a646f57afcbdfc81b",
"MacAddress": "02:42:0a:0a:01:0a",
"IPv4Address": "10.10.1.10/24",
"IPv6Address": ""
},
"b0f08db9efc2f8fdede0c217131f4d9057b90b4bd9e46de4a0ca05cb303c6775": {
"Name": "docker_iwt_agent_19",
"EndpointID": "937a75187cf7769ea3dfdfcdb6784cca20074e3678c9d11595c2b06900b48991",
"MacAddress": "02:42:0a:0a:01:15",
"IPv4Address": "10.10.1.21/24",
"IPv6Address": ""
},
"b39034c21095449ca11005dc39dd6e6e01136626ab6505cd87838916b593125c": {
"Name": "docker_iwt_agent_30",
"EndpointID": "38176fb2110c43a65c9ab0e1980d2736dc8d51db7ae812b7b84f0d098cd7bc9a",
"MacAddress": "02:42:0a:0a:01:23",
"IPv4Address": "10.10.1.35/24",
"IPv6Address": ""
},
"b4a6e4b193df0db681e5d4e80c9e6d8c460419282a5925609a8bfc49e6da546b": {
"Name": "docker_iwt_agent_3",
"EndpointID": "2bd0b5e20bd4bf47e28a5bdb3151d065f7df7146d0a9c4c675076538de2a4445",
"MacAddress": "02:42:0a:0a:01:26",
"IPv4Address": "10.10.1.38/24",
"IPv6Address": ""
},
"baf7691f3aabaac9198bb152237ae40848cc3ec71bfd5c839d1bacded09839e5": {
"Name": "docker_iwt_agent_23",
"EndpointID": "e942e0d9f9e3d1849e3b43e316acc817eaacb9cdeab9b5c27fd2275aac10abef",
"MacAddress": "02:42:0a:0a:01:02",
"IPv4Address": "10.10.1.2/24",
"IPv6Address": ""
},
"c0810b6fb6f06a33971c132069fc5b15b0191762b5f9f2915da641e02212c481": {
"Name": "docker_iwt_agent_39",
"EndpointID": "b9da33c0858803fa3cf09c0a33410539046a8d0add3dbe5c8426db4599d9c8e6",
"MacAddress": "02:42:0a:0a:01:25",
"IPv4Address": "10.10.1.37/24",
"IPv6Address": ""
},
"c11c0e1d6c19b58fcb655bd421167fbc52ba046962f4c42397a6cb7d52a0b043": {
"Name": "docker_iwt_agent_9",
"EndpointID": "883f51cfdc72a9f5b5765038936853c3126fc9c20c597fe4db0378123569fa0f",
"MacAddress": "02:42:0a:0a:01:1a",
"IPv4Address": "10.10.1.26/24",
"IPv6Address": ""
},
"c32fc4209a75d4a17b8043007c7bc5f1ff19b8df6e0d5bf4dc779c0f9fb96d25": {
"Name": "docker_iwt_agent_37",
"EndpointID": "e0e03869ec48c5467eb7bd8c65ac115ca7948a3b20b440337482612a914d37b4",
"MacAddress": "02:42:0a:0a:01:22",
"IPv4Address": "10.10.1.34/24",
"IPv6Address": ""
},
"d5cf8cef442b85694dd73f74835dc6711bdff64e7e32ca43733cbe9e2b0a3017": {
"Name": "docker_iwt_agent_28",
"EndpointID": "899e7cf06e2965f5a899ac39ea599fd8e755f0d3c4e2f493fae6e0183ab03f0b",
"MacAddress": "02:42:0a:0a:01:1b",
"IPv4Address": "10.10.1.27/24",
"IPv6Address": ""
},
"d9f24bd6d4a6886fcef1368f64c21ebef3e2f99437681d5779f33168038935cc": {
"Name": "docker_iwt_agent_26",
"EndpointID": "e10cc1b2ce2e4fe15619742a5aab23d12c5dae19c2cbd5203171a3d54627c494",
"MacAddress": "02:42:0a:0a:01:08",
"IPv4Address": "10.10.1.8/24",
"IPv6Address": ""
},
"dd21d43db623ea47a8c2e2d353c6f50213fd478ca446e1bdca7a4b2b29648c57": {
"Name": "docker_iwt_agent_38",
"EndpointID": "32353ca1e5da0b6ab7f25568db8b0b4267956f0a0485bf0079049e5821ada648",
"MacAddress": "02:42:0a:0a:01:11",
"IPv4Address": "10.10.1.17/24",
"IPv6Address": ""
},
"e1a5a45f1842bb7149e15afd4e76aa973f40ec312b882da5629a881d887b4b38": {
"Name": "docker_iwt_agent_27",
"EndpointID": "7e233de0421931932330c4900bd273e93bc190f76bfd83a1521377cf9fe358f0",
"MacAddress": "02:42:0a:0a:01:2c",
"IPv4Address": "10.10.1.44/24",
"IPv6Address": ""
},
"e1b5f8a7a0f68579c8031051005a3ae7753d7fe2992d2c1e632cc7c1049346c3": {
"Name": "docker_iwt_agent_32",
"EndpointID": "a98a6db3f9bc0c3788eb91330f6f57f4f4373d508427e2b0e285397e8ac5b4ed",
"MacAddress": "02:42:0a:0a:01:0d",
"IPv4Address": "10.10.1.13/24",
"IPv6Address": ""
},
"e334302d1694a6458e014b90406a2945e91c43fedbb63566a0e3aa4d0068e60e": {
"Name": "docker_iwt_agent_7",
"EndpointID": "915f03e069de97948949df5bc2a69187ba54cca000be25921715a605a2611326",
"MacAddress": "02:42:0a:0a:01:07",
"IPv4Address": "10.10.1.7/24",
"IPv6Address": ""
},
"ecbac3f6726a40b04b8b874680f78d77cb27a0b462a0d082c053f62ce95f22b7": {
"Name": "docker_iwt_agent_18",
"EndpointID": "c50600484f96862bd41bcd7749f7e3cabef224778b622e7e3441715bab65c158",
"MacAddress": "02:42:0a:0a:01:1e",
"IPv4Address": "10.10.1.30/24",
"IPv6Address": ""
},
"f4f88c8d286072cfce68ef6f119c24f903be9e62e265fa9ef7c99b26f4bdea29": {
"Name": "docker_iwt_agent_36",
"EndpointID": "ef37ed88c4cac957f9d4794b865024e6be8c1ca8920c82a3bd84a149926d83cc",
"MacAddress": "02:42:0a:0a:01:24",
"IPv4Address": "10.10.1.36/24",
"IPv6Address": ""
},
"f9c42de6bb873e9ecc199d6c6799873d40ea45d4b436151c948c447089973ca1": {
"Name": "docker_iwt_agent_4",
"EndpointID": "9f757b85a87ea2f1fe05e9961feb978e16bc871a42042f7981cbdf12b44d1efb",
"MacAddress": "02:42:0a:0a:01:13",
"IPv4Address": "10.10.1.19/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Update 2:
Analogously to the proceedings above, I tried to set up the custom Docker network for the use of IPv6.
Hence, I create the network using following command:
docker network create --driver bridge --subnet=2a01:4f8:a:b:c::/80 --ipv6 agent_network
Additionally, I enabled IPv6 in the /etc/docker/daemon.json
file as follows:
{
"ipv6": true,
"fixed-cidr-v6": "2a01:4f8:a:b::/64"
}
However, the same issues as before persist, and even though the output of the command docker network inspect agent_network states that all containers are connected to the Docker network and have uniques IP addresses, there is no communication between the containers. As before, my host network shuts down as well as soon as I start the Docker containers. Still, everything is running perfectly if I start less than 32 containers (also using IPv6).