I have a swarm with 2 nodes. One is an ubuntu VM on azure and the other one is my VM on my local machine.
When the containers try to make requests to each other with I get this dial tcp 10.0.0.88:9999: connect: no route to host
I've enabled in the 2 nodes all the swarm communication ports needed: tcp 2377 udp/tcp 7946 and udp 4789.
Communication works if I run everything local.
Any ideas?
Thanks
An overlay network doesn't create connectivity between two nodes, it requires connectivity, and then uses that to connect containers running on each node. From the prerequisites, each node needs to be able to reach the overlay ports on every other node in the cluster. See the documentation for more details:
https://docs.docker.com/network/overlay/
Related
Hello docker swarm gurus!
I need your help today please.
I am facing some issue trying to deploy a swarm between a manager on a machine A and a worker on a machine B reached via ssh. The challenge here is that the machine B can only be reached via ssh through a vpn, so every communication with machine A has to happen via the VPN link using ssh port forwarding
I was able to build the swarm properly but when ever a service spawns a replica on the machine B, the container cannot connect to other services in the swam as if the dns or the swarm routing mesh was not properly forwarded on the machine B.
to set up my swarm from A where the swarm manager is running, I forwarded the port 2377, 7946 and 4789
machine-A$ ssh -R 2377:localhost:2377 -R 7946:localhost:7946 -R 4789:localhost:4789 machine-B
from there I was able to join the swarm
machine-B$ swarm join --token xxxxx localhost:2377
I am able to start services on both machines A and B and they do show in the swarm as running..... but..... the issue is when I search accessing to some service from another container running. only services spawned on node A can ping each other. any service hosted on B does not respond to ping. and in any container running on machine B, they cannot ping any service running either on B or A....
I checked /etc/resolv.conf in container on B, and it reads:
search mydomain.local
nameserver 127.0.0.11
options ndots:0
127.0.0.11 answers to the ping, but a ping to any other services returns
ping: <service_name>: Name or service not known
Any idea of what I am missing?
Is there any specific more to do to configure the swarm routing mesh properly?
thanks for any suggestion
Sam.
My friend and I are trying to connect our Docker daemon using Docker Swarm. We both are using Windows OS and we are NOT on the same network. According to Docker docs each docker host must have the following ports open;
TCP port 2377 for cluster management communications
TCP and UDP port 7946 for communication among nodes
UDP port 4789 for overlay network traffic
We both have added new rules for the given ports in inbound and outbound rules in the firewall. Though we keep getting the same two errors while trying to join using token created by the manager node using docker swarm join --token command;
1. error response from daemon: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.65.3:2377: connect: connection refused"
2. Timeout error
Also, if either of us runs docker swarm init it shows 192.168.65.3 IP address that isn't part of any network we're connected to. What does it mean?
Docker overlay tutorial also states that in order to connect to the manager node, the worker node should add the IP address of the manager.
docker swarm join --token \ --advertise-addr IP-ADDRESS-OF-WORKER-1
IP-ADDRESS-OF-MANAGER:2377
Does it mean that in our case we have to use public IP address of the manager node after enabling port forwarding?
Potential network issues aside, here is your problem:
We both are using Windows OS
I have seen this issue in other threads when attempting to use Windows nodes in a multi-node swarm. Here are some important pieces of information from the Docker overlay networks documentation:
Before you can create an overlay network, you need to either initialize your Docker daemon as a swarm manager using docker swarm init or join it to an existing swarm using docker swarm join. Either of these creates the default ingress overlay network which is used by swarm services by default.
Overlay network encryption is not supported on Windows. If a Windows node attempts to connect to an encrypted overlay network, no error is detected but the node cannot communicate.
By default, Docker encrypts all swarm service management traffic. As far as I know, disabling this encryption is not possible. Do not confuse this with the --opt encrypted option, as that involves encrypting application data, not swarm management traffic.
For a single-node swarm, using Windows is just fine. For a multi-node swarm, which would be deployed using Docker stack, I highly recommend using Linux for all worker and manager nodes.
A while ago I was using Linux as a manager node and Windows as a worker node. I noticed that joining the swarm would only work if the Linux machine was the swarm manager; If the Windows machine was the manager, joining the swarm would not work. After the Windows machine joined the swarm, container-to-container communication over a user-defined overlay network would not work no matter what. Replacing the Windows machine with a Linux machine fixed all issues.
I tried to run docker swarm over two different networks.
First network is 10.10.100.x/24
Second network is 10.10.150.x/24
Both networks can see each other. There are no firewall rules between them to block any traffic.
Specifically I tested 7946 TCP and UDP and 4789 UDP. I can confirm, that I can connect from the first network to the second network on both ports and both protocols. And also from the second network to the first network without any issue.
Docker swarm is up and running and used engine is 20.10.11
I can see that all nodes have status=Ready and availability=Active.
Ingress network is default:
and I can see all peers listed there as well.
But when I deploy any service to any node with port -p 20000:80, then I can see this node only from the network where it was deployed.
If service lands on the first network, it is accessible only through nodes from the first network, not from the second.
If service lands on the second network, it is accessible only through nodes from the second network, not from the first.
Any thoughts how to fix this?
Thanks
update 1:
Tried to run swarm with additional parameter docker swarm init --default-addr-pool 172.100.0.0/16. Result remains the same.
update 2:
Based on the advice from #BMitch
I verified with sudo tcpdump -nn -s0 -v port 4789 or 7946, that port 7946 works (UDP and TCP).
I also verified with the previous tcpdump command and nc -z -v -u 10.10.150.200 4789 (run from the first network), that port 4789 works as well.
Same issue for me, routing and overlay work great but ingress load balancer only works through the same site endpoints that runs the container.
Oddly I discovered ingress load balancer works cross sites when using nc -l as server socket, making the whole even more obscure to me.
REM: Underlay network is wireguard VPN (L3 point-to-point)
In the end - problem was in the NAT. Our second network was behind NAT, which caused this issue. Once we removed NAT, everything worked.
TLDR; I cannot ping my docker containers from my other network clients. Only when a container actively pings the gateway I am able to reach the containers afterwards.
On my homenetwork (192.168.0.0/24) I run a gateway 192.168.0.1 which hosts a DNS server and also routes the internet traffic. My docker host (192.168.0.100) has a macvlan network, created with
docker network create -d macvlan --subnet=192.168.0.0/24 --gateway=192.168.0.100 -o parent=eth0 dockernet
My containers now do get static IPs, like 192.168.0.200. The containers can actively ping other physical hosts on the network, so that works fine.
But if I spin up a new container, it cannot be pinged from my physical network. Not from the docker host (which is expected as this seems to be a limitation of the macvlan network), nor from the gateway or any other client.
Once the container actively pings the gateway, it gets also reachable for other clients.
So I guess some routing needs to be done and there I need your help.
Clients run on debian buster and I use an unmanaged switch to connect the clients.
The missing information above was that I am running docker on raspbian.
So this question is actually a duplicate of Docker MACVLAN only works Outbound
runsudo rpi-update on the host to make it work
I am trying to create a simple docker swarm of two systems in my local network. I gave command docker swarm join --token SWMTKN-1-ns78a9s9d9alnma7qnhwdna9o0hdf8ei8f xx.xx.xx.xx:2377 to make other system join the swarm. But I am getting error Error response from daemon: manager stopped: can't initialize raft node: rpc error: code = Unavailable desc = grpc: the connection is unavailable.
My systems are behind proxy and I have configured docker with proxy as well. I am able to download docker images and I can ping the other system as well.
Swarm nodes must have direct access to each other, and can't communitate through NAT or a proxy for intra-Swarm communcations.
Plus, you'll want to be sure they can talk to each other on the proper Swarm ports:
TCP port 2377 for cluster management & raft sync communications
TCP and UDP port 7946 for "control plane" gossip discovery communication
UDP port 4789 for "data plane" VXLAN overlay network traffic