Docker swarm on 'dind' images and networking problems - docker

Question

I am trying to create a local swarm environment based on 'dind' images. Below are the steps for environment recreation:
docker network create --attachable --subnet 10.0.0.0/16 tools_network
docker run -d --privileged --name swarm-manager-1 --hostname swarm-manager-1 --network tools_network --ip 10.0.0.3 -p 42421:2375 docker:17.03.1-dind
docker --host localhost:42421 swarm init --advertise-addr 10.0.0.3
docker run -d --privileged --name swarm-worker-1 --hostname swarm-worker-1 --network tools_network --ip 10.0.0.4 -p 42423:2375 docker:17.03.1-dind
docker --host localhost:42423 swarm join --token <swarm-token> 10.0.0.3:2377
After that I add an nginx-based proxy:
docker run -d --name swarm-proxy --network tools_network -v $(pwd)/temp-proxy:/etc/nginx:ro -p 80:80 nginx:stable-alpine
with the following contents of nginx.conf in the 'temp-proxy' folder:
events {
worker_connections 1024;
}
http {
upstream swarm {
server 10.0.0.3;
server 10.0.0.4;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
location / {
proxy_pass http://swarm;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
The service I use for testing purposes is launched by:
docker --host localhost:42421 service create --name test-web --publish 80:80 yeasy/simple-web
What I expect at this stage, based on the mesh networking documentation is that curl localhost will return the result from the deployed service. However, I get a 502 Bad Gateway response, with the following log messages from the proxy:
[error] 7#7: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://10.0.0.4:80/", host: "localhost"
[error] 7#7: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://10.0.0.3:80/", host: "localhost"
10.0.0.1 - - "GET / HTTP/1.1" 502 173 "-" "curl/7.47.0"
Same result (connection refused) for curling from a docker container deployed to the tools_network.
Running netstat -l from inside one of the swarm nodes shows that they don't listen on port 80:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.11:38943 0.0.0.0:* LISTEN
tcp 0 0 :::2375 :::* LISTEN
tcp 0 0 :::2377 :::* LISTEN
tcp 0 0 :::7946 :::* LISTEN
udp 0 0 127.0.0.11:35094 0.0.0.0:*
udp 0 0 0.0.0.0:4789 0.0.0.0:*
udp 0 0 :::7946 :::*
The questions are:
Is there something wrong with my configuration? What is it?
If not, what steps should I take to get to the problem's root?

Answer 1

After a lot of investigations I found the problem looking at this.
Docker inside dind images couldn't initialize properly due to absent modules and I actually saw the errors in the container logs (docker logs swarm-manager-1) but didn't pay attention to them.
So the solution for me was launching the swarm nodes like this:
docker run -d --privileged --name swarm-manager-1 --hostname swarm-manager-1 --network tools_network --ip 10.0.0.3 -p 42421:2375 -v /lib/modules:/lib/modules:ro docker:17.03.1-dind
where the /lib/modules mapping is the piece that was absent.