Docker Swarm - Elastichsearch instances cannot resolve each other on the same overlay - docker

Question

I am trying to create an elasticsearch stack using docker swarm (Don't really need swarm functionality, but the .yml has been written)
The problem I seem to be getting is that when i start up the stack, the two masters can't resolve each other (Despite confirming they're on the same network)
and it may be the fault of discovery.seed_hosts providing the masters with an empty list or it could be the docker network not working properly.
"message":"publish_address {10.0.120.16:9300}, bound_addresses {0.0.0.0:9300}"
"message":"bound or publishing to a non-loopback address, enforcing bootstrap checks",
"message":"failed to resolve host [es-master2]"
"message":"master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [es-master1, es-master2] to bootstrap a cluster: have discovered [{es-master1}{SmILpuyqTGCkzzG5KmMIRg}{ru7wfhJTRkSXitkT5Ubhgw}{10.0.115.6}{10.0.115.6:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{es-master1}{SmILpuyqTGCkzzG5KmMIRg}{ru7wfhJTRkSXitkT5Ubhgw}{10.0.115.6}{10.0.115.6:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
I'm not sure if the discovering from hosts [] means that something went wrong with the discovery.seed_hosts setting. I suspect it because without the setting it'll be replaced with [127.0.0.1:9300,... etc]
Here's the parts of my docker compose that's relevant. It's part of a bigger file, but right now i just need the two masters to talk first.
.
.
.
networks:
es-internal:
driver: overlay
es-connection:
driver: overlay
external:
driver: overlay
name: external
.
.
.
es-master1:
image: docker.elastic.co/elasticsearch/elasticsearch:8.2.3
hostname: es-master1
environment:
- node.name=es-master1
- cluster.name=${CLUSTER_NAME}
- node_role="master,ingest"
- cluster.initial_master_nodes=es-master1,es-master2
- discovery.seed_hosts=es-master2
# - bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-master1.key
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-master1.crt
- xpack.security.http.ssl.certificate_authorities/usr/share/elasticsearch/config/certs/ca.crt
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-master1.key
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-master1.crt
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
networks:
- es-internal
volumes:
- ./data/es-master1:/usr/share/elasticsearch/data
secrets:
- source: ca-crt
target: /usr/share/elasticsearch/config/certs/ca.crt
- source: es-master1-crt
target: /usr/share/elasticsearch/config/certs/es-master1.crt
- source: es-master1-key
target: /usr/share/elasticsearch/config/certs/es-master1.key
configs:
- source: jvm-coordination
target: /usr/share/elasticsearch/config/jvm.options.d/jvm-coordination
deploy:
endpoint_mode: dnsrr
mode: "replicated"
replicas: 1
resources:
limits:
memory: 1G
depends_on:
- es-coordination
healthcheck:
test: curl -fs http://localhost:9200/_cat/health || exit 1
interval: 30s
timeout: 5s
retries: 3
start_period: 45s
es-master2:
image: docker.elastic.co/elasticsearch/elasticsearch:8.2.3
hostname: es-master2
environment:
- node.name=es-master2
- cluster.name=${CLUSTER_NAME}
- node_role="master,ingest"
- cluster.initial_master_nodes=es-master1,es-master2
- discovery.seed_hosts=es-master1
# - bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-master2.key
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-master2.crt
- xpack.security.http.ssl.certificate_authorities/usr/share/elasticsearch/config/certs/ca.crt
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-master2.key
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-master2.crt
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
networks:
- es-internal
volumes:
- ./data/es-master2:/usr/share/elasticsearch/data
secrets:
- source: ca-crt
target: /usr/share/elasticsearch/config/certs/ca.crt
- source: es-master2-crt
target: /usr/share/elasticsearch/config/certs/es-master2.crt
- source: es-master2-key
target: /usr/share/elasticsearch/config/certs/es-master2.key
configs:
- source: jvm-coordination
target: /usr/share/elasticsearch/config/jvm.options.d/jvm-coordination
deploy:
endpoint_mode: dnsrr
mode: "replicated"
replicas: 1
resources:
limits:
memory: 1G
depends_on:
- es-coordination
healthcheck:
test: curl -fs http://localhost:9200/_cat/health || exit 1
interval: 30s
timeout: 5s
retries: 3
start_period: 45s
.
.
.
I've tried:
inspecting the network it creates to see if they are in the same network
All sorts of es config options
New swarm service meant to test out pinging between containers on the same overlay network. (They could resolve the hostnames given to them with the ping command)
Expose ports (I know it's useless now)
If anyone has some input it would be great!