Elasticsearch query slow response via kibana console - docker

Question

Server background : 3 node elasticsearch cluster + kibana + logstash running on docker environment. host server runs rhel7.7(2cpu, 8GB RAM + 200GB fileshare).
Versions :
elasticsearch 7.5.1
kibana 7.5.1
logstash 7.5.1
filebeat 7.5.1 (runs on separate server)
## Cluster health status
{
"cluster_name" : "es-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 116,
"active_shards" : 232,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
## Node status
172.20.1.3 60 91 13 0.98 1.30 1.45 dilm - elasticsearch2
172.20.1.4 57 91 13 0.98 1.30 1.45 dilm - elasticsearch3
172.20.1.2 61 91 14 0.98 1.30 1.45 dilm * elasticsearch
## Host server TOP output
top - 11:37:10 up 11 days, 22:30, 3 users, load average: 0.74, 1.29, 1.47
Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.4 us, 0.8 sy, 0.0 ni, 94.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 7999840 total, 712736 free, 5842300 used, 1444804 buff/cache
KiB Swap: 3071996 total, 2794496 free, 277500 used. 1669472 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
48491 vtha837 20 0 4003724 713564 23880 S 5.0 8.9 1:57.53 java
59023 vtha837 20 0 6796456 1.5g 172556 S 2.7 20.3 28:18.07 java
59006 vtha837 20 0 6827100 1.5g 176840 S 1.7 19.9 30:03.59 java
760 vtha837 20 0 6877220 1.5g 180752 S 0.7 19.9 24:37.88 java
59610 vtha837 20 0 1663436 258152 7336 S 0.3 3.2 16:51.84 node
## Kibana environment variables I used for kibana docker image
environment:
SERVER_NAME: "kibana"
SERVER_PORT: 9548
ELASTICSEARCH_PASSWORD: ${ES_PASSWORD}
ELASTICSEARCH_HOSTS: "http://elasticsearch:9550"
KIBANA_DEFAULTAPPID: "dashboard/Default"
LOGGING_QUIET: "true"
XPACK_SECURITY_ENCRYPTIONKEY: ${KIBANA_XPACK_SEC_KEY}
XPACK_SECURITY_SESSIONTIMEOUT: 600000
Issue :
A. When I run elasticsearch queries via kibana console at least took 20000 ms return output to the console. But if I run the same query directly(elasticsearch) via curl or postman or chrome it took only less than 200 ms to get the output
B. even this happening when I loading kibana dashboad(not all the time), get following error message and not loading some graphs. but I can't see any exceptions or errors from console logs
Error in visualization
[esaggs] > Request to Elasticsearch failed: {"error":{}}
If I refresh the page, I can see all the graphs.
Chrome performance profile directly hitting elasticsearch query URL: http://testnode.mycompany.com.nz:9550/_cat/indices
Chrome performance profile via kibana dev console elasticsearch query Query : GET /_cat/indices
What I not understand is If I run same docker compose file in my laptop(windoes 10, 16GB, i7 2cpu, docker desktop running) i'm not facing any slowness either kibana dev console query or directly query elasticseach.
Anyone having this issue and appreciate let me know how to fix this?
Thanks in advance.

Answer 1

Issue is the docker service discovery. Due to some reason docker service discovery not happened. as soon as I change the elasticsearch host to IP get the real performance.
In docker-compose Previous kibana configuration
ELASTICSEARCH_HOSTS: "http://elasticsearch:9550"
New configuration
ELASTICSEARCH_HOSTS: "http://172.20.1.2:9550"
more details refer to the elasticsearch discuss page