I'm trying to use Elasticsearch with docker.
And you can see the guide here -> https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
my docker-compose.yml below
version: '2.2'
services:
elasticsearch1:
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
container_name: elasticsearch1
environment:
- node.name=master-node
- cluster.name=es-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es-data01:/usr/share/elasticsearch/data
ports:
- 127.0.0.1:9200:9200
- 127.0.0.1:9300:9300
networks:
- elastic
stdin_open: true
tty: true
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
container_name: elasticsearch2
environment:
- node.name=data-node1
- cluster.name=es-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch1"
ports:
- 127.0.0.1:9301:9300
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es-data02:/usr/share/elasticsearch/data
networks:
- elastic
stdin_open: true
tty: true
volumes:
es-data01:
driver: local
es-data02:
driver: local
networks:
elastic:
# driver: bridge
the problem is
I cannot connect by curl -XGET localhost:9200
docker container exits automatically after few seconds
can you help me?
ps : when I try docker run it works. what is the difference between them?
docker run -d -p 9200:9200 -p 9300:9300 --name elasticsearch -it --rm -v els:/usr/share/elasticsearch/data -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.7.0
Please check the container logs by using docker logs <your stopped container-id>, here you can get the container id using docker ps -a command.
Also please follow this SO answer and set the memory requirements
which would help you run the Elasticsearch in docker. if it doesn't help then provide the logs which you can get as explained earlier.
Based on comments adding the updated docker-compose
version: '2.2'
services:
elasticsearch1:
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
container_name: elasticsearch1
environment:
- node.name=master-node
- node.master=true
- cluster.name=es-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "cluster.initial_master_nodes=master-node"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es-data01:/usr/share/elasticsearch/data
ports:
- 127.0.0.1:9200:9200
- 127.0.0.1:9300:9300
networks:
- elastic
stdin_open: true
tty: true
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
container_name: elasticsearch2
environment:
- node.name=data-node1
- node.master=false
- cluster.name=es-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "cluster.initial_master_nodes=master-node"
ports:
- 127.0.0.1:9301:9300
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es-data02:/usr/share/elasticsearch/data
networks:
- elastic
stdin_open: true
tty: true
volumes:
es-data01:
driver: local
es-data02:
driver: local
networks:
elastic:
# driver: bridge
As you are following this article, https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
worth checking the second section with limits and memory resources as the containers in docker-compose is exiting due to low resources.
Exited with code 137 error is because of resource limitation (usually RAM) on the host machine. You can resolve this problem by adding this line to the environment variables of your docker-compose file:
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
You can read more about heap size settings, on official Elasticsearch documentation, in this link.
Related
hi i want to connect to Elasticsearch inside my app which is defined as "cog-app" service in docker-compose.yml along with ditsro elasticsearch and kibana
i am not able to connect to elasticsearch when i run docker file, can you please tell me how i can connect elasticsearch service to app service
i have defined elasticsearch in cog-app service, and im getting connection failure with elasticsearch
version: "3"
services:
cog-app:
image: app:2.0
build:
context: .
dockerfile: ./Dockerfile
stdin_open: true
tty: true
ports:
- "7111:7111"
environment:
- LANG=C.UTF-8
- LC_ALL=C.UTF-8
- CONTAINER_NAME=app
volumes:
- /home/developer/app:/app
odfe-node1:
image: amazon/opendistro-for-elasticsearch:1.13.2
container_name: odfe-node1
environment:
- cluster.name=odfe-cluster
- node.name=odfe-node1
- discovery.seed_hosts=odfe-node1,odfe-node2
- cluster.initial_master_nodes=odfe-node1,odfe-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "ES_JAVA_OPTS=-Xms2g -Xmx2g" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- odfe-data1:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
odfe-node2:
image: amazon/opendistro-for-elasticsearch:1.13.2
container_name: odfe-node2
environment:
- cluster.name=odfe-cluster
- node.name=odfe-node2
- discovery.seed_hosts=odfe-node1,odfe-node2
- cluster.initial_master_nodes=odfe-node1,odfe-node2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- odfe-data2:/usr/share/elasticsearch/data
networks:
- odfe-net
kibana:
image: amazon/opendistro-for-elasticsearch-kibana:1.13.2
container_name: odfe-kibana
ports:
- 5601:5601
expose:
- "5601"
environment:
ELASTICSEARCH_URL: https://odfe-node1:9200
ELASTICSEARCH_HOSTS: https://odfe-node1:9200
networks:
- odfe-net
volumes:
odfe-data1:
odfe-data2:
networks:
odfe-net:
please tell me how two services can communicate with each other
As the elasticsearch service is running in another container, localhost is not valid. You should use odfe-node1:9200 as the url
I ran the following docker compose script and I am expecting two nodes to be up, however this only one. There seems to be some obvious error.
Taken from the documentation
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.12
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.12
container_name: elasticsearch2
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata2:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
http://127.0.0.1:9200/_cat/health
1598033352 18:09:12 docker-cluster green 1 1 0 0 0 0 0 0 - 100.0%
docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------
elasticsearch /usr/local/bin/docker-entr ... Up 0.0.0.0:9200->9200/tcp, 9300/tcp
elasticsearch2 /usr/local/bin/docker-entr ... Up 9200/tcp, 9300/tcp
Ok, I found out the issue. It seems both the containers are somehow not aware of each other and trying to create the clusters on their own.
I introduced delay with depends, it works fine now. Thanks
It is up. But you need to map the port to some local unused port. Like you have mapped for first one 9200:9200. Add the same for other one too like 8200:9200. Then try hitting second one from local at 8200 port. It should work.
I've installed Elastic Search and Kibana using docker image in ubuntu machine and the commands used to run are:
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.5.1
docker run --link 36c0ea06f9e3:elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:7.5.1
and it gets successfully run in my local machine.
Now the major concern for me when you try to create a cluster node with one master and slave node, user need to edit the /etc/elasticsearc/elasticsearch.yml.
But in this case (installed using docker image in ubuntu).
No such file or folder is created. Do let me know how can I create a cluster and store data in such case.
You already have part of the solution in your first docker run statement. The docker image of elasticsearch is configured using environment variables. For example your run statement sets the node to single-node mode using the -e "discovery.type=single-node" flag.
A full list of configuration options you can find here.
You may also share the config directory with the host using a host-shared volume:
docker run -p 9200:9200 -p 9300:9300 -v <your-dir>/elasticsearch.yml/:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:7.5.1
This allows you to edit the config at your_dir/elasticsearch.yml directly from your hosts filesystem. Just make sure the config file exists in the specified directory before attempting to start the container.
Not sure if this is useful but here you go
https://blog.patricktriest.com/text-search-docker-elasticsearch/
from is official guide
you can use envirmental varible to pass option to /etc/elasticsearc/elasticsearch.yml
this is sample docker-compose.yml:
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
driver: bridge
I have an elasticsearch image that is being used as a base image for multiple containers. I am wondering if there is any way to pre-configure an ingest pipeline such that the process of creating the image and building a container also creates the pipeline for me? It'd be great if the base image comes with the pipeline that i want it to have, otherwise I'd have to docker exec into each container that uses this image and send a curl request in each one to create the pipeline.
Right now I'm thinking that I have to add a curl to the elasticsearch server (after it starts) in docker-entrypoint.sh, but i'm not sure if there's any other way
I can advice you to use docker-compose. I personally find it very convenient. With one file you can configure a whole stack.
Here is an example to help you start:
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- node.name=node-test1
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- node-test1data:/usr/share/elasticsearch/data
ports:
- 9200:9200
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: elasticsearch2
environment:
- cluster.name=docker-cluster
- node.name=node-test2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- node-test2data:/usr/share/elasticsearch/data
kibana:
image: docker.elastic.co/kibana/kibana:6.3.2
container_name: kibana
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://elasticsearch:9200
depends_on:
- elasticsearch
logstash:
image: docker.elastic.co/logstash/logstash:6.3.2
container_name: logstash
ports:
- "5000:5000"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
Elasticsearch's official docker image documentation provides this docker-compose.yml example:
version: '2'
services:
elasticsearch1:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
container_name: elasticsearch1
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch1"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
volumes:
- esdata2:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
However, it doesn't explain how to customize the password. It does direct us to a X-Pack documentation page, but I refuse to believe I have to go through all that trouble just to change a password. Is there any simpler, canonical way of configuring a custom password for elasticsearch on a Docker Compose file?
Starting from 6.0 elasticsearch docker images has the ability to configure the password using the following environment variable - ELASTIC_PASSWORD.
For example:
docker run -e ELASTIC_PASSWORD=MagicWord docker.elastic.co/elasticsearch/elasticsearch-platinum:6.1.3
See:
https://www.elastic.co/guide/en/elasticsearch/reference/6.1/docker.html