Below is my docker.compose.yml.
After executing it, it shows:
kibana | {"type":"log","#timestamp":"2018-04-24T18:27:43Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://localhost:9200/"}
kibana | {"type":"log","#timestamp":"2018-04-24T18:27:43Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
I believe the problem might be related to the platinum release of elasticsearch I am excuting, related to the fact I might not be setting the right parameters to use it with Xpath.
Am I forgetting to set up anything to make platinum work?
I tried even going to kitematic and linking manually Kibana to Elasticsearch container but the same problem continues.
Nothing that I tried worked. How can I fix this?
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-platinum:6.2.4
container_name: elasticsearch
environment:
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=MagicWord
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch-platinum:6.2.4
container_name: elasticsearch2
environment:
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=MagicWord
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata2:/usr/share/elasticsearch/data
networks:
- esnet
kibana:
image: docker.elastic.co/kibana/kibana:6.2.4
container_name: kibana
environment:
- ELASTICSEARCH_URL="http://localhost:9200"
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=MagicWord
- "xpack.monitoring.ui.container.elasticsearch.enabled=true"
ports:
- 5601:5601
networks:
- esnet
depends_on:
- elasticsearch
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
Anyone can help?
Related
I am attempting to run an elasticsearch cluster with Kibana and Logstash using docker-compose.
The problem I'm running into is that Logstash keeps looking for the elastic search DB hostname as http://elasticsearch:9200. Here's an example of the logstash output.
logstash | [2021-08-23T15:30:03,534][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash | [2021-08-23T15:30:03,540][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
I'm also attaching my docker-compose.yml file.
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
networks:
- elastic
kib01:
image: docker.elastic.co/kibana/kibana:7.14.0
container_name: kib01
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://es01:9200
ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
networks:
- elastic
logstash:
image: logstash:7.14.0
environment:
ELASTICSEARCH_HOST: localhost
container_name: logstash
hostname: localhost
ports:
- 9600:9600
- 8089:8089
volumes:
- ./logstash/logstash.yml
- ./logstash/pipelines.yml
- ./logstash/data
command: --config.reload.automatic
environment:
LS_JAVA_OPTS: "-Xmx1g -Xms1g"
links:
- es01:es01
depends_on:
- es01
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
driver: bridge
For some reason, putting the host into the docker compose yaml file doesn't seem to work. Where should I go to point logstash to locahost rather than 'elasticsearch'?
Thanks
I don't think Logstash has any such variable ELASTICSEARCH_HOST... Plus, localhost would refer to the Logstash container itself, not something else. And don't set hostname: localhost for a container...
You have no container/service named elasticsearch, you have es01-3, thus why its unable to connect, (notice that Kibana uses the correct addresses) and you'd modify that address in the Logstash config/pipeline files
Using the docker-compose.yml file by here
(online changed version 7.9.1. to 7.9.2)
as seen below full content of docker-compose.yml;
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
ports:
- 9201:9201
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
ports:
- 9202:9202
networks:
- elastic
kib01:
image: docker.elastic.co/kibana/kibana:7.9.2
container_name: kib01
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://es01:9200
ELASTICSEARCH_HOSTS: http://es01:9200
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
driver: bridge
when run docker-compose up here the logs in command window;
It seems that both containers are started at same time so kibana is unable to identify elasticsearch container.
in Kibana service: Add below parameters in docker-compose.yml file
depends_on:
es01
environment:
'ELASTICSEARCH_HOSTS=http://es01:9200'
'XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH.ENABLED=false'
Instead of dict. try to use lists in env. variable
I am trying to run ELK stack on Docker using docker-compose. I am not seeing any errors But I am only able to access elasticsearch but not Kibana. No page is being loaded when I try to access localhost:5601. Here is the docker-compose file
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
container_name: elasticsearch2
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata2:/usr/share/elasticsearch/data
networks:
- esnet
kibana:
image: docker.elastic.co/kibana/kibana:6.4.0
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://elasticsearch:9200
networks:
- esnet
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
driver: bridge
As I see, you have used a user-defined network named esnet, in this way you have to publish the required ports. See this question, it might help you.
I was able to get it working with single node of Elastic search. Here is the docker-compose file that is working
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
networks:
- esnet
kibana:
image: docker.elastic.co/kibana/kibana:6.8.0
container_name: kib01
ports:
- 5601:5601
environment:
SERVER_NAME: kibana
SERVER_HOST: kibana
ELASTICSEARCH_URL: http://elasticsearch:9200
XPACK_MONITORING_ENABLED: "true"
networks:
- esnet
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
driver: bridge
I have this docker-compose.yml config:
version: '2.2'
services:
kibana:
image: docker.elastic.co/kibana/kibana:7.3.1
environment:
SERVER_NAME: kibana.interos.io
ELASTICSEARCH_HOSTS: http://elasticsearch.example.org
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.1
container_name: es01
environment:
- node.name=es01
- discovery.seed_hosts=es02
- cluster.initial_master_nodes=es01,es02
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx2512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.1
container_name: es02
environment:
- node.name=es02
- discovery.seed_hosts=es01
- cluster.initial_master_nodes=es01,es02
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx2512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata01:
driver: local
esdata02:
driver: local
networks:
esnet:
how to can I tell Kibana to use the same docker network as the elasticsearch containers? I assume I have to add
networks:
- esnet
to the kibana container, anything else I need to do? What is the best practice here?
Your assumption is right. The kibana service needs to look like this:
kibana:
image: docker.elastic.co/kibana/kibana:7.3.1
networks:
- esnet
environment:
SERVER_NAME: kibana.interos.io
ELASTICSEARCH_HOSTS: http://elasticsearch.example.org
Liferay is not able to recognize my Elasticsearch cluster when starting. Here is my docker-compose configuration:
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
container_name: es01
environment:
- node.name=es01
- discovery.seed_hosts=es02
- cluster.initial_master_nodes=es01,es02
- cluster.name=liferay-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- "9299:9200"
- "9399:9300"
expose:
- "9299"
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
container_name: es02
environment:
- node.name=es02
- discovery.seed_hosts=es01
- cluster.initial_master_nodes=es01,es02
- cluster.name=liferay-cluster2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- "9298:9200"
- "9398:9300"
expose:
- "9298"
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata01:
driver: local
esdata02:
driver: local
networks:
esnet:
com.liferay.portal.search.elasticsearch6.configuration.ElasticsearchConfiguration.config file content
transportAddresses="127.0.0.1:9299"
logExceptionsOnly="false"
operationMode="REMOTE"
indexNamePrefix="myprefix-"
clusterName="liferay-cluster"
When starting docker-compose, I'm able to access my two ES clusters on: http://127.0.0.1:9299/ and http://127.0.0.1:9298/
However, when liferay starts it is unable to access ES nodes:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{vUNCF_HNRtu_tYUjkqhXvg}{127.0.0.1}{127.0.0.1:9299}]]
Anyone tried this configuration ? Any help would be appreciated. Thanks :-)
I've found a solution. It could help if someone is trying to do the same.
As, I said in my comment to #ibexit, I'm running two dockerized ES clusters and two separate Liferay portals (not in containers) on the same machine (development mode).
I changed th transport address in Liferay OSGi config file, since it must match the transport tcp port where ES is running:
transportAddresses="127.0.0.1:9301"
logExceptionsOnly="false"
operationMode="REMOTE"
indexNamePrefix="myprefix-"
clusterName="liferay-cluster"
I also added the property network.publish_host=127.0.0.1 in my ES clusters (without this property Liferay was not able to detect ES nodes)
Here is my docker-compose.yml:
Using ES 6.1.4
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:6.1.4
container_name: es01
environment:
- node.name=es01
- cluster.name=liferay-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- transport.tcp.port=9301
- network.publish_host=127.0.0.1
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- "9201:9200"
- "9301:9301"
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:6.1.4
container_name: es02
environment:
- node.name=es02
- cluster.name=liferay-cluster2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- transport.tcp.port=9302
- network.publish_host=127.0.0.1
ulimits:
memlock:
soft: -1
hard: -1
ports:
- "9202:9200"
- "9302:9302"
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata01:
driver: local
esdata02:
driver: local
networks:
esnet:
network.publish_host did the trick !
Per default, elasticsearch binds the transport and http ports to localhost (local) only. So your ports exposed by docker are not working. You need to bind to a specific ip or using 0.0.0.0 for all or site as explained here: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#network-interface-values
Please have in mind, that enabling this will startup the node in production mode followed by several bootstrap checks. Please see the docs if you need more informations on this topics or search SO.
My working local setup with docker compose and two elasticsearch nodes in docker containers and Liferay running on host. I am using elasticsearch 6.8.2 images modified according to Liferay docs available from docker hub with url: https://hub.docker.com/repository/docker/ktorek/liferay7-elasticsearch.
I am using gradle workspace. So I've configured: configs/local/osgi/configs/com.liferay.portal.search.elasticsearch6.configuration.ElasticsearchConfiguration.config:
operationMode=REMOTE
clusterName=docker-cluster
transportAddresses=127.0.0.1:9300,127.0.0.1:9301
I've killed a lot of time with transportAddresses configuration as it's documented to use square brackets and square quotes transportAddresses=["192.168.1.1:9300","192.168.1.2:9300"] but it does not work. Configuration listed above contains actual working configuration syntax.
My docker-compose.yml:
version: '3.7'
services:
es01:
container_name: "es01"
image: ktorek/liferay7-elasticsearch:latest
environment:
- node.name=es01
- node.data=true
- cluster.name=docker-cluster
- xpack.security.enabled=false
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=es02"
ports:
- "9300:9300"
- "9200:9200"
networks:
- mynetwork
volumes:
- es01-data:/usr/share/elasticsearch/data
ulimits:
memlock:
soft: -1
hard: -1
es02:
container_name: "es02"
image: ktorek/liferay7-elasticsearch:latest
environment:
- node.name=es02
- node.data=true
- cluster.name=docker-cluster
- xpack.security.enabled=false
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=es01"
ports:
- "9301:9300"
- "9201:9200"
networks:
- mynetwork
volumes:
- es02-data:/usr/share/elasticsearch/data
ulimits:
memlock:
soft: -1
hard: -1
networks:
mynetwork:
name: mynetwork
driver: bridge
ipam:
config:
- subnet: 172.30.29.0/24