Docker swarm network debugging - docker

Question

I have created a service in docker using docker swarm and stack, but somehow my application running in a service replica is not able to connect to other replicas.
Here is more detail. I have three nodes: one manage and two workers. At the same node as manager I run a registry, which is the source for the image which I deploy to the workers using docker stack.
Here is how my configuration file for docker stack looks like:
version: "3.2"
services:
compute:
image: openmpi-docker.registry:443/openmpi
ports:
- "2222:22"
deploy:
replicas: 4
restart_policy:
condition: none
placement:
constraints: [node.role != manager]
networks:
- openmpi-network
networks:
openmpi-network:
driver: overlay
attachable: true
As you can guess the image contains an openmpi distribution. It starts sshd and opens port 22.
I can connect to one of the the services using following command:
ssh -p 2222 -i ./user-ssh/docker_id <worker node>
Then I try to run a simple test. I figure out what are the IP addresses of other replicas and try to connect them over ssh:
for i in $(seq 3 6) ; do ssh 10.0.1.$i hostname ; done
This works as expected I see 4 different hostnames.
So my next test involves mpi.
mpirun -H 10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.6 hostname
This command is supposed to do exactly the same, with the exception that it should use mpirun instead of ssh. But somehow this fails.
So I want to see where exactly mpirun hangs using strace, but it is impossible to do so with SYS_PTRACE capability. And there is seemingly no way to set it if you use swarm. So I try to create another container, not in swarm, which shares the same overlay network:
docker run --network openmpi_openmpi-network --rm -it openmpi-docker.registry:443/openmpi bash
Here openmpi_openmpi-network is the name of the network created by "docker stack". And originally I planned to run it with "--privileged", but it turned out that I do not need that.
Then I switch from the root to a normal user and run the very same mpi command with the very same IP addresses, and the command works. So I do not run in the very same problem.
In other cases the solution to similar problem is just to turn off the firewall, but I'm pretty sure I don't have it inside containers. I created the image myself from debian:9 image and the only server I installed is sshd. And still this wouldn't explain the difference between starting the container in swarm and starting the container using docker run.
This makes me wonder. First how can I debug such kind of problem? Second what is so different between starting the service in swarm and starting it manually in regard to networking?
I would be grateful if you could help me answering these two questions.
Update
I tried to run mpirun with "--mca oob_base_verbose 100" and got following result. The logs are relatively long, but the key difference is in behaviour of the node which initiates the communication.
I a working log at some mpirun of the initiator node prints following:
...
[3365816c2c1e:00033] [[39982,0],2] oob:tcp:init adding 10.0.1.4 to our list of V4 connections
[3365816c2c1e:00033] [[39982,0],2] TCP STARTUP
[3365816c2c1e:00033] [[39982,0],2] attempting to bind to IPv4 port 0
[3365816c2c1e:00033] [[39982,0],2] assigned IPv4 port 57161
[3365816c2c1e:00033] mca:oob:select: Adding component to end
[3365816c2c1e:00033] mca:oob:select: checking available component usock
[3365816c2c1e:00033] mca:oob:select: Querying component [usock]
[3365816c2c1e:00033] oob:usock: component_available called
[3365816c2c1e:00033] [[39982,0],2] USOCK STARTUP
[3365816c2c1e:00033] SUNPATH: /tmp/openmpi-sessions-1000#3365816c2c1e_0/39982/0/usock
[3365816c2c1e:00033] mca:oob:select: Inserting component
[3365816c2c1e:00033] mca:oob:select: Found 2 active transports
<Log output from other nodes>
[3365816c2c1e:00033] [[39982,0],2]: set_addr to uri 2620260352.0;usock;tcp://10.0.1.7,172.18.0.5:48695
[3365816c2c1e:00033] [[39982,0],2]:set_addr checking if peer [[39982,0],0] is reachable via component usock
[3365816c2c1e:00033] [[39982,0],2]: peer [[39982,0],0] is NOT reachable via component usock
[3365816c2c1e:00033] [[39982,0],2]:set_addr checking if peer [[39982,0],0] is reachable via component tcp
[3365816c2c1e:00033] [[39982,0],2] oob:tcp: ignoring address usock
[3365816c2c1e:00033] [[39982,0],2] oob:tcp: working peer [[39982,0],0] address tcp://10.0.1.7,172.18.0.5:48695
[3365816c2c1e:00033] [[39982,0],2] PASSING ADDR 10.0.1.7 TO MODULE
[3365816c2c1e:00033] [[39982,0],2]:tcp set addr for peer [[39982,0],0]
[3365816c2c1e:00033] [[39982,0],2] PASSING ADDR 172.18.0.5 TO MODULE
[3365816c2c1e:00033] [[39982,0],2]:tcp set addr for peer [[39982,0],0]
...
But in non working case the initiator node stops at "Found 2 active transports" message and never prints anything else.
Update 2
I also figured out that hostname works if I add "--mca oob_tcp_if_include 10.0.1.0/24", so that the full command is:
mpirun -H 10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.6 --mca oob_tcp_if_include 10.0.1.0/24 hostname
Still I do not really understand why and how to avoid the need of specifying the subnet.