Docker multiple networks, unable to connect outside world - docker

Question

When deploying docker-compose with multiple networks, only the first interface have an access to the outside world
version: "3.9"
services:
speedtest:
build:
context: .
dockerfile: speedtest.Dockerfile
tty: true
networks:
- eth0
- eth1
networks:
eth0:
eth1:
Running inside the container ping for example ping -I eth0 google.com works fine
However running ping -I eth1 google.com will get the result
PING google.com (142.250.200.238) from 172.21.0.2 eth1: 56(84) bytes of data.
From c4d3b238f9a1 (172.21.0.2) icmp_seq=1 Destination Host Unreachable
From c4d3b238f9a1 (172.21.0.2) icmp_seq=2 Destination Host Unreachable
Any idea how to have egress to the internet on both networks?
Tried multiple combinations for creating the network, with external, bridge with custom config etc...
Update
After larsks answer, using ip route add for eth1 and running tcpdump -i any packets are coming in correctly:
11:26:12.098918 eth1 Out IP 8077ec32b69d > dns.google: ICMP echo request, id 3, seq 1, length 64
11:26:12.184195 eth1 In IP dns.google > 8077ec32b69d: ICMP echo reply, id 3, seq 1, length 64
But still 100% packet loss...

Answer 1

The problem here is that while there are two interfaces inside the container, there is only a single default route. Given a container with two interfaces, like this:
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
70: eth0#if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:10:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.16.2/20 brd 192.168.31.255 scope global eth0
valid_lft forever preferred_lft forever
72: eth1#if73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:30:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.48.2/20 brd 192.168.63.255 scope global eth1
valid_lft forever preferred_lft forever
The routing table looks like this:
/ # ip route
default via 192.168.16.1 dev eth0
192.168.16.0/20 dev eth0 proto kernel scope link src 192.168.16.2
192.168.48.0/20 dev eth1 proto kernel scope link src 192.168.48.2
When you run ping google.com or ping -I eth0 google.com, in both
cases your ICMP request egresses through eth0, goes to the
appropriate default gateway, and eventually works it way to
google.com.
But when you run ping -I eth1 google.com, there's no way to reach
the default gateway from that address; the gateway is only reachable
via eth0. Since the kernel can't find a useful route, it attempts to
connect directly. If we run tcpdump on the host interface that is
the other end of with eth1, we see:
23:47:58.035853 ARP, Request who-has 142.251.35.174 tell 192.168.48.2, length 28
23:47:59.083553 ARP, Request who-has 142.251.35.174 tell 192.168.48.2, length 28
[...]
That's the kernel saying, "I've been told to connect to this address
using this specific interface, but there's no route, so I'm going to
assume the address is on the same network and just ARP for it".
Of course that fails.
We can make this work by adding an appropriate route. You need to run
a privileged container to do this (or at least have
CAP_NET_ADMIN):
ip route add default via 192.168.48.1 metric 101
(The gateway address is the .1 address of the network associated with eth1.)
We need the metric setting to differentiate this from the existing
default route; without that the command would fail with RTNETLINK answers: File exists.
After running that command, we have:
/ # ip route
default via 192.168.16.1 dev eth0
default via 192.168.48.1 dev eth1 metric 101
192.168.16.0/20 dev eth0 proto kernel scope link src 192.168.16.2
192.168.48.0/20 dev eth1 proto kernel scope link src 192.168.48.2
And we can successfully ping google.com via eth1:
/ # ping -c2 -I eth1 google.com
PING google.com (142.251.35.174) from 192.168.48.2 eth1: 56(84) bytes of data.
64 bytes from lga25s78-in-f14.1e100.net (142.251.35.174): icmp_seq=1 ttl=116 time=8.87 ms
64 bytes from lga25s78-in-f14.1e100.net (142.251.35.174): icmp_seq=2 ttl=116 time=8.13 ms
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 8.127/8.497/8.868/0.370 ms
Having gone through all that, I'll add that I don't see many
situations in which it would be necessary: typically you use
additional networks in order to isolate things like database servers,
etc, while using the "primary" interface (the one with which the
default route is associated) for outbound requests.
I tested all this using the following docker-compose.yaml:
version: "3"
services:
sleeper:
image: alpine
cap_add:
- NET_ADMIN
command:
- sleep
- inf
networks:
- eth0
- eth1
networks:
eth0:
eth1: