Elasticsearch in Docker No route to host - docker

Question

I have a problem with Elasticsearch 5 in Docker.
Stack compose file:
version: "3.4"
services:
elastic01: &elasticbase
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.7
networks:
- default
restart: always
environment:
- node.name=elastic01
- cluster.name=elastic
- network.host=0.0.0.0
- xpack.security.enabled=false
- xpack.monitoring.enabled=false
- xpack.watcher.enabled=false
- bootstrap.memory_lock=false ## Docker swarm does not support that
- discovery.zen.minimum_master_nodes=2
- discovery.zen.ping.unicast.hosts=elastic02,elastic03
volumes:
- /var/docker/elastic:/usr/share/elasticsearch/data
deploy:
placement:
constraints: [node.hostname == node1]
elastic02:
<<: *elasticbase
depends_on:
- elastic01
environment:
- node.name=elastic02
- cluster.name=elastic
- network.host=0.0.0.0
- xpack.security.enabled=false
- xpack.monitoring.enabled=false
- xpack.watcher.enabled=false
- bootstrap.memory_lock=false ## Docker swarm does not support that
- discovery.zen.minimum_master_nodes=2
- discovery.zen.ping.unicast.hosts=elastic01,elastic03
volumes:
- /var/docker/elastic:/usr/share/elasticsearch/data
deploy:
placement:
constraints: [node.hostname == node2]
elastic03:
<<: *elasticbase
depends_on:
- elastic01
volumes:
- /var/docker/elastic:/usr/share/elasticsearch/data
environment:
- node.name=elastic03
- cluster.name=elastic
- network.host=0.0.0.0
- xpack.security.enabled=false
- bootstrap.memory_lock=false ## Docker swarm does not support that
- discovery.zen.minimum_master_nodes=2
- discovery.zen.ping.unicast.hosts=elastic01,elastic02
deploy:
placement:
constraints: [node.hostname == node3]
networks:
default:
driver: overlay
attachable: true
When I run stack file, it works like a charm. _cluster/health shows that nodes are up and running a the status is "Green" but after while, periodically, system goes down with exception Elastic exception
Feb 10 09:39:39 : [2018-02-10T08:39:39,159][WARN ][o.e.d.z.UnicastZenPing ] [elastic01] failed to send ping to [{elastic03}{2WS6GPu8Qka9YLE_PWfVKg}{AD_Nw1m9T-CZHUFhgXQjtQ}{10.0.9.5}{10.0.9.5:9300}{ml.max_open_jobs=10, ml.enabled=true}]
Feb 10 09:39:39 : org.elasticsearch.transport.ReceiveTimeoutTransportException: [elastic03][10.0.9.5:9300][internal:discovery/zen/unicast] request_id [5167] timed out after [3750ms]
Feb 10 09:39:39 : at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.7.jar:5.6.7]
Feb 10 09:39:39 : at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.7.jar:5.6.7]
Feb 10 09:39:39 : at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
Feb 10 09:39:39 : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
Feb 10 09:39:39 : at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Feb 10 09:39:40 : [2018-02-10T08:39:40,159][WARN ][o.e.d.z.UnicastZenPing ] [elastic01] failed to send ping to [{elastic03}{2WS6GPu8Qka9YLE_PWfVKg}{AD_Nw1m9T-CZHUFhgXQjtQ}{10.0.9.5}{10.0.9.5:9300}{ml.max_open_jobs=10, ml.enabled=true}]
Feb 10 09:39:40 : org.elasticsearch.transport.ReceiveTimeoutTransportException: [elastic03][10.0.9.5:9300][internal:discovery/zen/unicast] request_id [5172] timed out after [3750ms]
Feb 10 09:39:40 : at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.7.jar:5.6.7]
Feb 10 09:39:40 : at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.7.jar:5.6.7]
Feb 10 09:39:40 : at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
Feb 10 09:39:40 : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
Feb 10 09:39:40 : at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Feb 10 09:39:41 : [2018-02-10T08:39:41,159][WARN ][o.e.d.z.UnicastZenPing ] [elastic01] failed to send ping to [{elastic03}{2WS6GPu8Qka9YLE_PWfVKg}{AD_Nw1m9T-CZHUFhgXQjtQ}{10.0.9.5}{10.0.9.5:9300}{ml.max_open_jobs=10, ml.enabled=true}]
Feb 10 09:39:41 : org.elasticsearch.transport.ReceiveTimeoutTransportException: [elastic03][10.0.9.5:9300][internal:discovery/zen/unicast] request_id [5175] timed out after [3751ms]
Feb 10 09:39:41 : at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.7.jar:5.6.7]
Feb 10 09:39:41 : at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.7.jar:5.6.7]
Feb 10 09:39:41 : at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
Feb 10 09:39:41 : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
Feb 10 09:39:41 : at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
And sometimes:
Feb 10 09:44:10 [2018-02-10T08:44:10,810][WARN ][o.e.t.n.Netty4Transport ] [elastic01] exception caught on transport layer [[id: 0x3675891a, L:/10.0.9.210:53316 - R:10.0.9.5/10.0.9.5:9300]], closing connection
Feb 10 09:44:10 java.io.IOException: No route to host
Feb 10 09:44:10 at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
Feb 10 09:44:10 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
Feb 10 09:44:10 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:?]
Feb 10 09:44:10 at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:?]
Feb 10 09:44:10 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:?]
Feb 10 09:44:10 at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100) ~[netty-buffer-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
Feb 10 09:44:10 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Strange thing is that all the time it happens I am able to ping the container and resolve the name from the container where it happens. No packet loss, no timeouts. The only bad thing is the transport layer in the Elastic. All other services are running in the same cluster without issues (MongoDB, Redis, Internal microservices)
Does anybody have a clue?

Answer 1

I found the issue.
Elasticsearch must be binded to a one single interface, not to 0.0.0.0. Once I binded it to eth0, it started to work. Also it looks, there cannot be named volume - it throws another error during the time. It must be mounted to a local drive directly.
This works:
services:
elastic01:
environment:
network.host=_eth0_