ElasticSearch in Docker dies silently and restarts, but why? - docker

Question

I am monitoring a docker that runs ElasticSearch 6.2.3
Every day it dies, and I have not been able to figure out why...
I think it is memory, but where can I find proof of that ?
Running top on the host (Linux) I get
Virtual Mem = 53 Gig
Res = 26.7
right now ! that is...
docker docker inspect gives me this
"Memory": 34225520640,
"CpusetMems": "",
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 34225520640,
"MemorySwappiness": null,
"Name": "memlock",
The JVM params are
/usr/bin/java
-Xms1g
-Xmx1g
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=/usr/share/elasticsearch/tmp
-XX:+HeapDumpOnOutOfMemoryError
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-Xloggc:logs/gc.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=32
-XX:GCLogFileSize=64m
-Des.cgroups.hierarchy.override=/
-Xms26112m
-Xmx26112m
-Des.path.home=/usr/share/elasticsearch
-Des.path.conf=/usr/share/elasticsearch/config
-cp
/usr/share/elasticsearch/lib/*
org.elasticsearch.bootstrap.Elasticsearch
and here is the log
[2019-01-30T07:14:01,278] [app-mesos-orders_api-2019.01.30/6l7Ga1I5T3qhLKYmWjQpRA] update_mapping [doc]
[2019-01-30T07:25:53,489] initializing ...
[2019-01-30T07:25:53,581] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/vg_tobias-lv_tobias)]], net usable_space [126.8gb], net total_space [199.8gb], types [xfs]
[2019-01-30T07:25:53,581] heap size [25.4gb], compressed ordinary object pointers [true]
[2019-01-30T07:26:12,390], node ID [-sJqW_h1TKy9c_Ka08In0A]
[2019-01-30T07:26:12,391] version[6.2.3], pid[1], build[c59ff00/2018-03-13T10:06:29.741383Z], OS[Linux/3.10.0-862.11.6.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2019-01-30T07:26:12,391] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/usr/share/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Xms26112m, -Xmx26112m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2019-01-30T07:26:13,008] loaded module [aggs-matrix-stats]
[2019-01-30T07:26:13,008] loaded module [analysis-common]
[2019-01-30T07:26:13,008] loaded module [ingest-common]
[2019-01-30T07:26:13,008] loaded module [lang-expression]
[2019-01-30T07:26:13,008] loaded module [lang-mustache]
[2019-01-30T07:26:13,009] loaded module [lang-painless]
[2019-01-30T07:26:13,009] loaded module [mapper-extras]
[2019-01-30T07:26:13,009] loaded module [parent-join]
[2019-01-30T07:26:13,009] loaded module [percolator]
[2019-01-30T07:26:13,009] loaded module [rank-eval]
[2019-01-30T07:26:13,009] loaded module [reindex]
[2019-01-30T07:26:13,009] loaded module [repository-url]
[2019-01-30T07:26:13,009] loaded module [transport-netty4]
[2019-01-30T07:26:13,009] loaded module [tribe]
[2019-01-30T07:26:13,009] no plugins loaded
[2019-01-30T07:26:19,947] using discovery type [zen]
[2019-01-30T07:26:20,444] initialized
[2019-01-30T07:26:20,444] starting ...
[2019-01-30T07:26:20,600] publish_address {172.16.44.8:9300}, bound_addresses {172.17.0.14:9300}
[2019-01-30T07:26:21,507] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-01-30T07:26:24,855] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {elasticsearch-1}{-sJqW_h1TKy9c_Ka08In0A}{iGTgehBjQ3yPRm9nlTLbYw}{172.16.44.8}{172.16.44.8:9300}{rack=rack1}
[2019-01-30T07:26:24,861] new_master {elasticsearch-1}{-sJqW_h1TKy9c_Ka08In0A}{iGTgehBjQ3yPRm9nlTLbYw}{172.16.44.8}{172.16.44.8:9300}{rack=rack1}, reason: apply cluster state (from master [master {elasticsearch-1}{-sJqW_h1TKy9c_Ka08In0A}{iGTgehBjQ3yPRm9nlTLbYw}{172.16.44.8}{172.16.44.8:9300}{rack=rack1} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-01-30T07:26:24,881] publish_address {172.16.44.8:9200}, bound_addresses {172.17.0.14:9200}
[2019-01-30T07:26:24,881] started
[2019-01-30T07:26:37,033] recovered [1535] indices into cluster_state
I am thinking that the Docker container runs out of memory and dies silently
How can I prove this ? and what can I do to solve this ?