Attaching a second network to a Docker NGINX container causes it to stop responding to any of them - docker

Question

I've been trying to setup what might be a rather complicated docker setup, and have run into a very weird issue. What I currently have is a collection of containers, all running different web services, and an Nginx container that routes them to be publicly accessible over HTTPS. This has worked fine, but meant I can only setup services that use HTTPS, and was run over one of my 5 static IPs my ISP has given me, by routing it through my UniFi network. When I went to add GitLab, I realized I needed to connect it to a separate public address, so that I could access port 22 for SSH based Git clones. Since I already had the switch port connected to my modem on a vlan (topology weirdness, it works fine,) I simply tagged the server port to allow that VLan through, and started using a macvlan network. As soon as I added the macvlan to my nginx container, it stopped working all together. After spending several hours making sure my static ips were actually setup correctly, I found out that if I attach more than one network to my Nginx server, it stops responding to anything at all. If I stick just the macvlan on it, it can respond just fine, even over my static ip. But if there is more than one, everything stops working. Pings, TCP requests, everything. If I use docker network disconnect to remove the network from the running instance, it starts working immediately again. I've tried this with just netcat on an alpine instance, and can confirm that all inbound traffic stops immediately when a second network is attached, and resumes as soon as it's removed. I'm including a sample docker-compose that shows this effect just by adding or removing the networks.
docker version:
Client: Docker Engine - Community
Version: 20.10.13
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 10 14:07:51 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.13
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 906f57f
Built: Thu Mar 10 14:05:44 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.10
GitCommit: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.8.0-docker)
compose: Docker Compose (Docker Inc., v2.2.3)
scan: Docker Scan (Docker Inc., v0.12.0)
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 9
Server Version: 20.10.13
Storage Driver: zfs
Zpool: Storage
Zpool Health: ONLINE
Parent Dataset: Storage/docker
Space Used By Parent: 87704957952
Space Available: 8778335683049
Parent Quota: no
Compression: off
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
runc version: v1.0.3-0-gf46b6ba
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-104-generic
Operating System: Ubuntu 20.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 39.18GiB
Name: server2
ID: <Redacted>
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
EDIT: forgot to add the docker compose file. Here it is:
services:
nginx:
image: nginx:1.21.6-alpine
networks:
public_interface:
ipv4_address: 123.456.789.102 //Replaced with nonsense for privacy reasons
private_interface:
ipv4_address: 192.168.5.2
web_interface:
networks:
web_interface:
public_interface:
driver: macvlan
driver_opts:
parent: enp10s0.100
ipam:
config:
- subnet: 123.456.789.101/29 //Replaced with nonsense for privacy reasons
gateway: 123.456.789.108 //Replaced with nonsense for privacy reasons
private_interface:
driver: macvlan
driver_opts:
parent: enp10s0.305
ipam:
config:
- subnet: 192.168.5.0/24
gateway: 192.168.5.1

Answer 1

Ok, time to answer this so I don't become the next #979. Turns out I was right about the routing, and my issue lay not actually in docker, but in
how the network router in the kernel works. I confirmed this by running an application without docker (just a simple python HTTP server), and testing, finding the exact same issue.
The solution, it turns out, is to use a combination of routing tables, iptables, and packet marks. The first depends on your network backend. I'm using Netplan, 'cause Ubuntu, which means I have to tell Netplan to setup routing tables:
network:
version: 2
ethernets:
eth0:
dhcp4: true
dhcp6: false
gateway4: 192.168.1.1
eth1:
dhcp4: false
dhcp6: false
addresses:
- 123.456.789.20/24 #Server address + subnet
routes:
- to: 0.0.0.0/0
via: 123.456.789.1 #Gateway address
metric: 500
table: 100
routing-policy:
- from: 123.456.789.20 #Server address
table: 100
If you're not using Docker, this patches everything nicely, and things "just work". If you are, you'll need to also add a packet mark, and tell iptables to keep said mark when transferring the packet to the docker container. First, mark incoming packets:
ip rule add fwmark 0x1 table 100
Followed by telling iptables to keep the marks:
iptables -t mangle -A PREROUTING -i eth1 -m conntrack --ctstate NEW --ctdir ORIGINAL -j CONNMARK --set-mark 0x1
iptables -t mangle -A PREROUTING -m conntrack ! --ctstate NEW --ctdir REPLY -m connmark ! --mark 0x0 -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -m conntrack ! --ctstate NEW --ctdir REPLY -m connmark ! --mark 0x0 -j CONNMARK --restore-mark
Hopefully that helps future docker users. It was certainly an experience.
I also wrote all of this up on my blog, along with a bit more detail of where things started, why I was in this pickle, and how I figured it out: https://wiki.faeranne.com/en/blogs/nexus-labs/docker-netplan-woes