Containers on different swarm nodes can't reach eachother - docker

Question

in short, I've got the problem that containers in a swarm can't reach containers that sit on another node. The worker node is in my home network, so not directly accessible externally.
Setup:
Manager node that is a publicly available server, let's give it the IP A.A.A.A
Worker node that is at home, behind a router, with the internal IP B.B.B.B and the router with public IP C.C.C.C
The worker can without a problem join the swarm, the manager can without problems allocate containers to that worker, so some sort of communication is established and working.
What is not working is, that containers on the manager can't reach containers on the worker and vice versa (but can reach containers on the same node)
docker node ls shows the worker node as Ready and Active. docker node inspect <NODE NAME> show the IP C.C.C.C under Status
minimal working example:
docker-compose
version: "3.8"
services:
manager1:
image: jwilder/whoami
hostname: manager1
deploy:
placement:
constraints:
- node.role == manager
manager2:
image: jwilder/whoami
hostname: manager2
deploy:
placement:
constraints:
- node.role == manager
worker1:
image: jwilder/whoami
hostname: worker1
deploy:
placement:
constraints:
- node.role == worker
worker2:
image: jwilder/whoami
hostname: worker2
deploy:
placement:
constraints:
- node.role == worker
deploying with docker stack deploy -c docker-compose.yml testing
docker network inspect testing_default -v on manager shows
"Peers": [
{
"Name": "f0de4150d01e",
"IP": "A.A.A.A"
}
],
"Services": {
"testing_manager1": {
"VIP": "10.0.25.5",
"Ports": [],
"LocalLBIndex": 21646,
"Tasks": [
{
"Name": "testing_manager1.1.w6b2wufu96vk1jmtez9dtewr0",
"EndpointID": "213b7182882e267f249edc52be57f6c56d83efafeba471639f2cbb9398854fe0",
"EndpointIP": "10.0.25.6",
"Info": {
"Host IP": "A.A.A.A"
}
}
]
},
"testing_manager2": {
"VIP": "10.0.25.8",
"Ports": [],
"LocalLBIndex": 21645,
"Tasks": [
{
"Name": "testing_manager2.1.5w51imw8toh81oyeruu48z2pr",
"EndpointID": "41eeb9eaf97cd3f744873ccea9577332e24c799f61171c59447e084de9c829a4",
"EndpointIP": "10.0.25.9",
"Info": {
"Host IP": "A.A.A.A"
}
}
]
}
}
docker network inspect testing_default -v on worker shows
"Peers": [
{
"Name": "75fba815742b",
"IP": "B.B.B.B"
},
{
"Name": "f0de4150d01e",
"IP": "A.A.A.A"
}
],
"Services": {
"testing_worker1": {
"VIP": "10.0.25.10",
"Ports": [],
"LocalLBIndex": 293,
"Tasks": [
{
"Name": "testing_worker1.1.ol4x1h560613l7e7yqv94sj68",
"EndpointID": "3a9dc067b4a0e7e5d26fabdcb887b823f49bfad21fc0ec159edd8dd4f976b702",
"EndpointIP": "10.0.25.11",
"Info": {
"Host IP": "B.B.B.B"
}
}
]
},
"testing_worker2": {
"VIP": "10.0.25.2",
"Ports": [],
"LocalLBIndex": 292,
"Tasks": [
{
"Name": "testing_worker2.1.m2d5fwn83uxg9b7udakq1o41x",
"EndpointID": "8317415fe2b0fa77d1195d33e91fa3354fcfd00af0bab5161c69038eb8fe38bb",
"EndpointIP": "10.0.25.3",
"Info": {
"Host IP": "B.B.B.B"
}
}
]
}
}
So the worker sees the manager as a peer, but does not see the other services. What confuses me, is that the Host IP for worker services is B.B.B.B, which is the internal IP of the worker node (so a 192.168.x.x IP) instead of the external IP of my home network.
Attaching to one of the containers with docker exec -it <CONTAINER ID> /bin/sh and executing wget -qO- <ANOTHER CONTAINERS IP>:8000 returns fine for containers on the same node, but Host unreachable for containers on the other node. (Testing with the defined host names returns "bad address" for the ones on the other node)
Looking at the docs, it reads at https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts that there need to be some ports open.
I was under the impression that creating the swarm comes with a virtual network between the nodes (which kinda seems to be the case, as the services can get created without a problem, so there is a connection). But as it did not work like that, I tested it with just plain port forwarding, which resulted in the manager "sometimes" seeing the other services when inspecting the network, but the containers still can't reach eachother.
Am I supposed to spin up a VPN for the nodes to be inside the same network, or what am I missing?