Docker containers in the same network can't communicate (ARP is possible but not upper layers messages) - docker

Question

I have been trying to split a simple API-rest into different services using docker. Unfortunately, I have not been able to make it work. I have read the docker docs several times and have followed multiple stack-over-flow and docker forum threads but non of the answers worked for me. I am new to Docker, so I might be missing something.
I detected that the communication host-container was ok but container-container wasn't, so in order to see what was going on I installed ping on get and post services (which run on a debian:bullseye-slim based image) and also wireshark in my host machine. What I have detected is that I can ping the host (172.22.0.1) and also the name resolution is okay (when I run ping post its IP is displayed) but for some reason when I send a ping request from post to get no reply is received.
My docker-compose.yaml file is the following:
version: '3.9'
services:
mydb:
image: mariadb:latest
environment:
MYSQL_DATABASE: 'cars'
MYSQL_ALLOW_EMPTY_PASSWORD: 'true'
ports:
- "3306:3306"
container_name: mydb
networks:
- mynw
post:
build: ./post-service
ports:
- "8081:8081"
container_name: post
networks:
- mynw
privileged: true
get:
build: ./get-service
ports:
- "8080:8080"
container_name: get
networks:
- mynw
privileged: true
nginx2:
build: ./nginx2
ports:
- "80:80"
container_name: nginx2
networks:
- mynw
networks:
mynw:
external: true
Initially, I was using the default network, but I read that this might cause internal DNS problems I changed it. I created the network by CLI without any special parameters (docker network create mynw). The JSON displayed when running docker network inspect mynw is the following:
[
{
"Name": "mynw",
"Id": "f925467f7efee99330f0eaaa82158006ac645cc92e7abda693f052c10da485bd",
"Created": "2022-10-14T18:42:14.145569533+02:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.22.0.0/16",
"Gateway": "172.22.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"4eb6e348d84b2433199e6581b4406eb74fb93c1fc2269691b81b34c13c723db5": {
"Name": "nginx2",
"EndpointID": "b19fab264c1489b616d919f09a5b80a1774561ea6f2538beb86157065c1e787b",
"MacAddress": "02:42:ac:16:00:03",
"IPv4Address": "172.22.0.3/16",
"IPv6Address": ""
},
"5f20802a59708bf4a592e137f52fca29dc857734983abc1c61548783e2e61896": {
"Name": "mydb",
"EndpointID": "3ef7b5d619b5b9ad9441dbc2efabd5a0e5a6bb2ea68bbd58fae8f7dfd2ac36ed",
"MacAddress": "02:42:ac:16:00:02",
"IPv4Address": "172.22.0.2/16",
"IPv6Address": ""
},
"dee816dd62aa08773134bb7a7a653544ab316275ec111817e11ba499552dea5b": {
"Name": "post",
"EndpointID": "cca2cbe801160fa6c35b3a34493d6cc9a10689cd33505ece36db9ca6dcf43900",
"MacAddress": "02:42:ac:16:00:04",
"IPv4Address": "172.22.0.4/16",
"IPv6Address": ""
},
"e23dcd0cecdb609e4df236fd8aed0999c12e1adc7b91b505fc88c53385a81292": {
"Name": "get",
"EndpointID": "83b73045887827ecbb1779cd27d5c4dac63ef3224ec42f067cfc39ba69b5484e",
"MacAddress": "02:42:ac:16:00:05",
"IPv4Address": "172.22.0.5/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Curiously, when sniffing the network using wireshark I see that the ARP messages between the containers are exchanged without problem (get service asks for post's MAC adress and this one replies with its MAC, and then this information is processed correctly to send the ICMP request).
I thought that maybe the network layer was dropping the replies for some reason and installed iptables to both services and added a ACCEPT rule for icmp messages to both INPUT and OUTPUT, but also didn't change anything. If someone knows what else could I do or what am I missing it would be very helpful.

Answer 1

Finally the solution was to delete everything and reinstall Docker and Docker Compose.