I am trying to run ELK stack on Docker using docker-compose. I am not seeing any errors But I am only able to access elasticsearch but not Kibana. No page is being loaded when I try to access localhost:5601. Here is the docker-compose file
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
container_name: elasticsearch2
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata2:/usr/share/elasticsearch/data
networks:
- esnet
kibana:
image: docker.elastic.co/kibana/kibana:6.4.0
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://elasticsearch:9200
networks:
- esnet
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
driver: bridge
As I see, you have used a user-defined network named esnet, in this way you have to publish the required ports. See this question, it might help you.
I was able to get it working with single node of Elastic search. Here is the docker-compose file that is working
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
networks:
- esnet
kibana:
image: docker.elastic.co/kibana/kibana:6.8.0
container_name: kib01
ports:
- 5601:5601
environment:
SERVER_NAME: kibana
SERVER_HOST: kibana
ELASTICSEARCH_URL: http://elasticsearch:9200
XPACK_MONITORING_ENABLED: "true"
networks:
- esnet
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
driver: bridge
Related
I am attempting to run an elasticsearch cluster with Kibana and Logstash using docker-compose.
The problem I'm running into is that Logstash keeps looking for the elastic search DB hostname as http://elasticsearch:9200. Here's an example of the logstash output.
logstash | [2021-08-23T15:30:03,534][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash | [2021-08-23T15:30:03,540][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
I'm also attaching my docker-compose.yml file.
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
networks:
- elastic
kib01:
image: docker.elastic.co/kibana/kibana:7.14.0
container_name: kib01
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://es01:9200
ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
networks:
- elastic
logstash:
image: logstash:7.14.0
environment:
ELASTICSEARCH_HOST: localhost
container_name: logstash
hostname: localhost
ports:
- 9600:9600
- 8089:8089
volumes:
- ./logstash/logstash.yml
- ./logstash/pipelines.yml
- ./logstash/data
command: --config.reload.automatic
environment:
LS_JAVA_OPTS: "-Xmx1g -Xms1g"
links:
- es01:es01
depends_on:
- es01
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
driver: bridge
For some reason, putting the host into the docker compose yaml file doesn't seem to work. Where should I go to point logstash to locahost rather than 'elasticsearch'?
Thanks
I don't think Logstash has any such variable ELASTICSEARCH_HOST... Plus, localhost would refer to the Logstash container itself, not something else. And don't set hostname: localhost for a container...
You have no container/service named elasticsearch, you have es01-3, thus why its unable to connect, (notice that Kibana uses the correct addresses) and you'd modify that address in the Logstash config/pipeline files
Using the docker-compose.yml file by here
(online changed version 7.9.1. to 7.9.2)
as seen below full content of docker-compose.yml;
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
ports:
- 9201:9201
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
ports:
- 9202:9202
networks:
- elastic
kib01:
image: docker.elastic.co/kibana/kibana:7.9.2
container_name: kib01
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://es01:9200
ELASTICSEARCH_HOSTS: http://es01:9200
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
driver: bridge
when run docker-compose up here the logs in command window;
It seems that both containers are started at same time so kibana is unable to identify elasticsearch container.
in Kibana service: Add below parameters in docker-compose.yml file
depends_on:
es01
environment:
'ELASTICSEARCH_HOSTS=http://es01:9200'
'XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH.ENABLED=false'
Instead of dict. try to use lists in env. variable
I'm novice with elasticsearch and Docker. I found a great utility to filter data to final users "Search UI": https://github.com/elastic/search-ui
I see that all services I need to manage my "ES" network are available with Docker, but Search UI not. My question is, how can I add Search UI to my docker compose file?
Here is my actual docker-compose file:
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9201:9200
- 9301:9300
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
ports:
- 9202:9200
- 9302:9300
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
ports:
- 9203:9200
- 9303:9300
networks:
- elastic
cerebro:
image: lmenezes/cerebro:0.8.5
container_name: cerebro-01
ports:
- "9000:9000"
external_links:
- es01:es01
- es02:es02
- es03:es03
networks:
- elastic
kibana:
image: docker.elastic.co/kibana/kibana:7.6.1
container_name: kibana-01
ports:
- "5601:5601"
environment:
ELASTICSEARCH_HOSTS: http://es01:9200
networks:
- elastic
appsearch:
image: docker.elastic.co/app-search/app-search:7.6.1
container_name: appsearch-01
environment:
- "elasticsearch.host=http://es01:9200"
- "allow_es_settings_modification=true"
- "JAVA_OPTS=-Xms2g -Xmx2g"
ports:
- 3002:3002
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
driver: bridge
Any suggestions?
Thanks!
I have this docker-compose.yml config:
version: '2.2'
services:
kibana:
image: docker.elastic.co/kibana/kibana:7.3.1
environment:
SERVER_NAME: kibana.interos.io
ELASTICSEARCH_HOSTS: http://elasticsearch.example.org
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.1
container_name: es01
environment:
- node.name=es01
- discovery.seed_hosts=es02
- cluster.initial_master_nodes=es01,es02
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx2512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.1
container_name: es02
environment:
- node.name=es02
- discovery.seed_hosts=es01
- cluster.initial_master_nodes=es01,es02
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx2512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata01:
driver: local
esdata02:
driver: local
networks:
esnet:
how to can I tell Kibana to use the same docker network as the elasticsearch containers? I assume I have to add
networks:
- esnet
to the kibana container, anything else I need to do? What is the best practice here?
Your assumption is right. The kibana service needs to look like this:
kibana:
image: docker.elastic.co/kibana/kibana:7.3.1
networks:
- esnet
environment:
SERVER_NAME: kibana.interos.io
ELASTICSEARCH_HOSTS: http://elasticsearch.example.org
Below is my docker.compose.yml.
After executing it, it shows:
kibana | {"type":"log","#timestamp":"2018-04-24T18:27:43Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://localhost:9200/"}
kibana | {"type":"log","#timestamp":"2018-04-24T18:27:43Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
I believe the problem might be related to the platinum release of elasticsearch I am excuting, related to the fact I might not be setting the right parameters to use it with Xpath.
Am I forgetting to set up anything to make platinum work?
I tried even going to kitematic and linking manually Kibana to Elasticsearch container but the same problem continues.
Nothing that I tried worked. How can I fix this?
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-platinum:6.2.4
container_name: elasticsearch
environment:
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=MagicWord
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch-platinum:6.2.4
container_name: elasticsearch2
environment:
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=MagicWord
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata2:/usr/share/elasticsearch/data
networks:
- esnet
kibana:
image: docker.elastic.co/kibana/kibana:6.2.4
container_name: kibana
environment:
- ELASTICSEARCH_URL="http://localhost:9200"
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=MagicWord
- "xpack.monitoring.ui.container.elasticsearch.enabled=true"
ports:
- 5601:5601
networks:
- esnet
depends_on:
- elasticsearch
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
Anyone can help?