How to connect docker container and swarm in different hosts - docker

Question

I have many processing tasks and require the use of many hosts to distribute these tasks to. For my use case, I am using zmq's malamute because of familiarity and my favor with zmq, but I would be glad to hear suggestions for other libraries that would make it easy to distribute tasks to workers.
I am trying to connect one container, a worker, to a swarm manager, which is a malamute broker, during development. I am developing what will be a global mode docker service. During development, I need to interactively give commands to write my code, so I am using a container with a shared overlay.
#### manager node, host 1
docker service create --replicas 0 --name malamute zeromqorg/malamute
#docker service update --publish-add 9999 malamute
docker service update --publish-add published=9999,target=9999 malamute
docker network create --attachable --driver overlay primarynet
docker service update --network-add primarynet malamute
### docker inspect primarynet
[
{
"Name": "primarynet",
"Id": "b7vq0p0purgiykebtykdn7syh",
"Created": "2021-11-08T13:34:08.600557322-05:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.2.0/24",
"Gateway": "10.0.2.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"f50adb23eebcac36e4c1c5816d140da469df1e1ee72ae2c7bada832ba69e1535": {
"Name": "malamute.1.e29sipw9i0rijvldeqbmh4r9r",
"EndpointID": "2fe7789b4c50eeca7d19007e447aa32a3bb8d6a33435051d84ba04e34f206446",
"MacAddress": "02:42:0a:00:02:0f",
"IPv4Address": "10.0.2.15/24",
"IPv6Address": ""
},
"lb-primarynet": {
"Name": "primarynet-endpoint",
"EndpointID": "9b3926bcfea39d77a48ac4fcc1c5df37c1dd5e7add6b790bc9cbc549a5bea589",
"MacAddress": "02:42:0a:00:02:04",
"IPv4Address": "10.0.2.4/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4098"
},
"Labels": {},
"Peers": [
{
"Name": "582d161489c7",
"IP": "192.168.1.106"
},
{
"Name": "c71e09e3cd2c",
"IP": "192.168.1.107"
}
]
}
]
### docker inspect ingress
docker inspect ingress
[
{
"Name": "ingress",
"Id": "od7bn815iuxyq4v9jzwe17n4p",
"Created": "2021-11-08T10:59:00.304709057-05:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": true,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"f50adb23eebcac36e4c1c5816d140da469df1e1ee72ae2c7bada832ba69e1535": {
"Name": "malamute.1.e29sipw9i0rijvldeqbmh4r9r",
"EndpointID": "bb87dbf390d23c8992c4f2b27597af80bb8d3b96b3bd62aa58618cca82cc0426",
"MacAddress": "02:42:0a:00:00:0a",
"IPv4Address": "10.0.0.10/24",
"IPv6Address": ""
},
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "2c1ef914e0aa506756b779ece181c7d8efb8697b71cb5dce0db1d9426137660f",
"MacAddress": "02:42:0a:00:00:02",
"IPv4Address": "10.0.0.2/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
},
"Labels": {},
"Peers": [
{
"Name": "582d161489c7",
"IP": "192.168.1.106"
},
{
"Name": "c71e09e3cd2c",
"IP": "192.168.1.107"
},
{
"Name": "c67bb64801c0",
"IP": "192.168.1.143"
}
]
}
]
This is where the error occurs
#### worker node, host 2
docker swarm join --token <token> 192.168.1.106:2377
docker run --net primarynet -ti zeromqorg/malamute /bin/bash
### within the container
export PYTHONPATH=/home/zmq/malamute/bindings/python/:/home/zmq/czmq/bindings/python
python3
>>> from malamute import MalamuteClient
>>> client=MalamuteClient()
>>> client.connect(b'tcp://manager.addr.ip.addr:9999', 100, b'service')
### failure happens on the connect, but it shsould go through.
### the error output:
I: 21-11-10 07:12:59 My address is 'service'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/zmq/malamute/bindings/python/malamute/__init__.py", line 51, in connect
"Could not connect to malamute server at {!r}", endpoint,
File "/home/zmq/malamute/bindings/python/malamute/__init__.py", line 41, in _check_error
fmt.format(*args, **kw) + ': ' + str(reason)
malamute.MalamuteError: Could not connect to malamute server at <addr>: Server is not reachable
# the malamute server is confirmed to be reachable with ping from the worker
zmq#8e2256f6a806:~/malamute$ ping 10.0.2.15
PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
64 bytes from 10.0.2.15: icmp_seq=1 ttl=64 time=0.646 ms
64 bytes from 10.0.2.15: icmp_seq=2 ttl=64 time=0.399 ms
64 bytes from 10.0.2.15: icmp_seq=3 ttl=64 time=0.398 ms
64 bytes from 10.0.2.15: icmp_seq=4 ttl=64 time=0.401 ms
^C
--- 10.0.2.15 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3079ms
rtt min/avg/max/mdev = 0.398/0.461/0.646/0.106 ms
The hostnames of the worker and manager host also confirm with ping, similar output as above; ping manager... ping worker..., just as above. But, if I were enter the docker id of the service running on swarm manager as the argument to ping in the worker, then it doesn't work.
After reading the guides, I don't know why the connect error occurs. Basically, once I'm done, the connect call needs to go through and then the rest of the script should execute. I will then script what I need executed and create a docker service called worker.