Elasticsearch cluster doesn't work on Docker Swarm - docker

Question

The docker-compose YAML below file brings up a 3-node Elasticsearch cluster when used with the docker compose command. That is OK for debug but I want to move to deployment, so I want to deploy on a swarm where the containers can run on different systems.
So
docker compose up
works, but
docker stack deploy -c docker-compose.yml p3es
creates the same containers (although on different systems) and the overlay network, but the elasticsearch instances are not able to talk to each other via port 9300. So a master never gets assigned and although elasticsearch responds to HTTP requests they just error out.
In the logs the following exception/stack trace appears on each container:
p3es_es01.1.sv26uqp4i4s3#carbon | "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [es03][10.0.12.9:9300][internal:cluster/coordination/join]",
p3es_es01.1.sv26uqp4i4s3#carbon | "Caused by: org.elasticsearch.transport.ConnectTransportException: [es01][10.0.0.53:9300] connect_exception",
(etc)
The cause of the exception turns out to be:
p3es_es01.1.sv26uqp4i4s3#carbon | "Caused by: java.io.IOException: connection timed out: 10.0.0.53/10.0.0.53:9300",
So here are some things I have tried:
I invoke a shell on one of the containers. I can ping each of the other containers. I can also do a curl -XGET on each of the containers and get a response from port 9200.
If I do a curl -XGET on port 9300 on one of the containers I get a "Not an HTTP Port" message. But at least it was able to resolve the address.
Docker stack likes to put prefixes on names for objects. So if you name a network xyz the network actually gets named project_xyz. So I changed the environment variables that tell elasticsearch who is part of the cluster to include the project name prefix. No luck.
I've run out of ideas. Any suggestions?
version: '3.9'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
environment:
- HOSTNAME=es01
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
networks:
- es9300
volumes:
- nfs-es01:/usr/share/elasticsearch/data
ports:
- 9200:9200
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
environment:
- HOSTNAME=es02
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
networks:
- es9300
volumes:
- nfs-es02:/usr/share/elasticsearch/data
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
environment:
- HOSTNAME=es03
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
networks:
- es9300
volumes:
- nfs-es03:/usr/share/elasticsearch/data
volumes:
nfs-es01:
driver_opts:
type: nfs
o: addr=10.2.0.1,rw,nfsvers=4,local_lock=all
device: :/sbn/process3/elasticsearch01
nfs-es02:
driver_opts:
type: nfs
o: addr=10.2.0.1,rw,nfsvers=4,local_lock=all
device: :/sbn/process3/elasticsearch02
nfs-es03:
driver_opts:
type: nfs
o: addr=10.2.0.1,rw,nfsvers=4,local_lock=all
device: :/sbn/process3/elasticsearch03
networks:
es9300:
driver: overlay
attachable: true

Answer 1

As it turns out, Elasticsearch discovery gets confused when Docker provides it with multiple overlay networks. So the directive:
ports:
- 9200:9200
causes each es0* service to be on the ingress overlay network in addition to the overlay network specified (in this case es9300). For some reason when Elasticsearch is running in the containers, it gets the wrong IP address when resolving the service/DNS es01.
I haven't determined why that is, but removing the ports directive to publish port 9200 resolves the issue.
Hopefully this posting will help someone encountering the same issue.