Docker containers on same host but different bridges can't connect - docker

Question

I have (2) Docker-Compose configs running on the SAME host but DIFFERENT Docker bridges. Each are on different subnets so their traffic must be routed. One Docker-Compose config is for a containerized website, while the other Docker-Compose config is for a Zabbix Agent to monitor the website Docker-Compose config.
Although the Docker host they both live on has routes in the Gateway Router to both subnets, the Linux Docker host itself is also configured as a router, so should route traffic between the subnets it hosts.
Why can't traffic pass between different bridges on the SAME Docker host?!?!?

Answer 1

Intro:
Before implementing containerized monitoring, I had no prior requirement to pass traffic between docker bridges on the same host. I'm a Linux & network engineer and this wasted an hour of my life trying to understand how things were breaking, so I imagine if you're not a network engineer, you'll waste a lot more time- or fail completely. Thus felt it was worthy of a moment to document.
Short Answer:
Docker was being "helpful" - again- by automagically inserting iptables rules in the DOCKER-ISOLATION-STAGE-1 & 2 Chains in the FORWARD table breaking connectivity. Delete these rules and connectivity between containers raised on different bridges assigned to different subnets on the same host can now be achieved.
Longer Answer w/ Proofs:
Diagnostics:
I re-cut the image for the Zabbix Agent including some diagnostic tools- traceroute, inetutils-ping & iproute2- and after logging into the container with docker exec -u root -it <container ID> bash, I found the Agent's container couldn't ping the containers on the other bridge despite ip route list proving there was a correct route out of the Agent's container.
A review of the Docker host's firewall rules revealed that passing traffic between the Docker Bridges is DISALLOWED by design:
iptables -nvx -L --line-numbers
<SNIP>
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num pkts bytes target prot opt in out source destination
1 530374 174564169 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
2 3559 5117334 DOCKER-ISOLATION-STAGE-2 all -- br-2dfcb90fe695 !br-2dfcb90fe695 0.0.0.0/0 0.0.0.0/0
3 1229457 499057258 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
2 16 960 DROP all -- * br-2dfcb90fe695 0.0.0.0/0 0.0.0.0/0
3 533917 179680543 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
<SNIP>
If the source & destination of traffic are NOT the same for our Docker bridge in DOCKER-ISOLATION-STAGE-1, then it matches the chain's rule 2 which passes it to Chain DOCKER-ISOLATION-STAGE-2 where it matches rule 2 here and drops the traffic.
We know this rule is having effect as we can see the packets incrementing for it; traffic is indeed being dropped. So Rule 2 in Chain DOCKER-ISOLATION-STAGE-2 if the offender.
Solution:
Print the rules so we can determine the rule numbers of the iptables busting our connectivity:
sudo iptables -nvx -L --line-numbers
Then delete the problematic rules by their respective numbers. Note the final number "2" at the end of each iptables command is the rule number you want to delete. We'll delete both the target and the referring rule:
sudo iptables -D DOCKER-ISOLATION-STAGE-1 2
sudo iptables -D DOCKER-ISOLATION-STAGE-2 2
WARNING: Although restarting containers will NOT cause the deleted iptables rules to be recreated, doing a docker-compose down followed by an up WILL.
Hope this saves others wasted cycles figuring out broken container network connectivity...