I'm using Docker-Compose to run Elasticsearch.
I want to set path.repo for snapshot by array.
version: '2.1'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.3
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- node.name=es01
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- **'path.repo=["/usr/share/elasticsearch/book_backup/search_example", "/usr/share/elasticsearch/book_backup/agg_example"]'** ==> not recognize.
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- /Users/kkk/apps/elastic-cp/book_backup/search_example:/usr/share/elasticsearch/book_backup/search_example
- /Users/kkk/apps/elastic-cp/book_backup/agg_example:/usr/share/elasticsearch/book_backup/agg_example
ports:
- "9200:9200"
- "9300:9300"
networks:
- docker_net
networks:
docker_net:
driver: bridge
So I set it like this but it doesn't recognize the array.
What is the solution?
An array cannot be an environment variable, an environment variable can only be a pair of key value chains.
You can use several environnement variables or use others methods to pass this arguments.
I don't know what means '**' at environment section, but if we want pass array as environments variables in docker compose, use:
example:
- Vault:Kv:paths:0=path/data/infrastructure/mongo
- Vault:Kv:paths:1=path/data/infrastructure/redis
- Vault:Kv:paths:2=path/data/infrastructure/tefBus
Related
I am looking for a working version of a docker-compose file that starts up kibana and elasticsearch together on docker for mac at version 7.3.2. I've followed the most recent instructions on kibana and elasticsearch's 7.3.2 documentation and my docker-compose.yml file below is the union of what I gathered from both docs. (The kibana doc was the most vague with respect to the docker compose config). I've also tried following other stack overflow articles (written for older versions) but they don't seem to work with the latest versions. I now suspect I'm missing something version specific. 7.3.1 didn't work with the same config either.
I should note that the elasticsearch portion of the file works fine; I can hit http://localhost:9200 and I get a json response. However Kibana's url (http://localhost:5601) returns Kibana server is not ready yet with this error:
kibana | {"type":"log","#timestamp":"2019-09-12T21:45:04Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: http://elasticsearch:9200/"}
This is my best attempt so far:
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
container_name: es01
environment:
- node.name=es01
- discovery.seed_hosts=es02
- cluster.initial_master_nodes=es01,es02
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
container_name: es02
environment:
- node.name=es02
- discovery.seed_hosts=es01
- cluster.initial_master_nodes=es01,es02
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.3.2
ports:
- 5601:5601
networks:
- esnet
environment:
SERVER_NAME: kibana.example.org
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
esdata01:
driver: local
esdata02:
driver: local
networks:
esnet:
Docker Compose automatically creates a private Docker network for you, and within that, the names of the service: blocks are valid hostnames.
When you set
ELASTICSEARCH_URL: http://elasticsearch:9200
None of your containers are named elasticsearch so the hostname lookup fails, but if you pick either node es01 or es02 it will work
ELASTICSEARCH_URL: http://es01:9200
(Note that you don’t explicitly need a networks: definition for this to work, Compose will create a network named default for you. You also don’t need to explicitly set container_name: unless you’re planning on trying to manage the same containers with non-Compose tooling.)
Use ELASTICSEARCH_HOSTS: http://es01:9200 instead of ELASTICSEARCH_URL to update the environment from the docker-compose.yml file. Here is the Elasticsearch documentation about environment variable configuration https://www.elastic.co/guide/en/kibana/current/docker.html#environment-variable-config.
You need to add network configuration also in kibana service like
networks:
- esnet
and in ELASTICSEARCH_HOST: http://es01:9200
note es01 is your container name
Liferay is not able to recognize my Elasticsearch cluster when starting. Here is my docker-compose configuration:
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
container_name: es01
environment:
- node.name=es01
- discovery.seed_hosts=es02
- cluster.initial_master_nodes=es01,es02
- cluster.name=liferay-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- "9299:9200"
- "9399:9300"
expose:
- "9299"
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
container_name: es02
environment:
- node.name=es02
- discovery.seed_hosts=es01
- cluster.initial_master_nodes=es01,es02
- cluster.name=liferay-cluster2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- "9298:9200"
- "9398:9300"
expose:
- "9298"
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata01:
driver: local
esdata02:
driver: local
networks:
esnet:
com.liferay.portal.search.elasticsearch6.configuration.ElasticsearchConfiguration.config file content
transportAddresses="127.0.0.1:9299"
logExceptionsOnly="false"
operationMode="REMOTE"
indexNamePrefix="myprefix-"
clusterName="liferay-cluster"
When starting docker-compose, I'm able to access my two ES clusters on: http://127.0.0.1:9299/ and http://127.0.0.1:9298/
However, when liferay starts it is unable to access ES nodes:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{vUNCF_HNRtu_tYUjkqhXvg}{127.0.0.1}{127.0.0.1:9299}]]
Anyone tried this configuration ? Any help would be appreciated. Thanks :-)
I've found a solution. It could help if someone is trying to do the same.
As, I said in my comment to #ibexit, I'm running two dockerized ES clusters and two separate Liferay portals (not in containers) on the same machine (development mode).
I changed th transport address in Liferay OSGi config file, since it must match the transport tcp port where ES is running:
transportAddresses="127.0.0.1:9301"
logExceptionsOnly="false"
operationMode="REMOTE"
indexNamePrefix="myprefix-"
clusterName="liferay-cluster"
I also added the property network.publish_host=127.0.0.1 in my ES clusters (without this property Liferay was not able to detect ES nodes)
Here is my docker-compose.yml:
Using ES 6.1.4
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:6.1.4
container_name: es01
environment:
- node.name=es01
- cluster.name=liferay-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- transport.tcp.port=9301
- network.publish_host=127.0.0.1
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- "9201:9200"
- "9301:9301"
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:6.1.4
container_name: es02
environment:
- node.name=es02
- cluster.name=liferay-cluster2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- transport.tcp.port=9302
- network.publish_host=127.0.0.1
ulimits:
memlock:
soft: -1
hard: -1
ports:
- "9202:9200"
- "9302:9302"
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata01:
driver: local
esdata02:
driver: local
networks:
esnet:
network.publish_host did the trick !
Per default, elasticsearch binds the transport and http ports to localhost (local) only. So your ports exposed by docker are not working. You need to bind to a specific ip or using 0.0.0.0 for all or site as explained here: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#network-interface-values
Please have in mind, that enabling this will startup the node in production mode followed by several bootstrap checks. Please see the docs if you need more informations on this topics or search SO.
My working local setup with docker compose and two elasticsearch nodes in docker containers and Liferay running on host. I am using elasticsearch 6.8.2 images modified according to Liferay docs available from docker hub with url: https://hub.docker.com/repository/docker/ktorek/liferay7-elasticsearch.
I am using gradle workspace. So I've configured: configs/local/osgi/configs/com.liferay.portal.search.elasticsearch6.configuration.ElasticsearchConfiguration.config:
operationMode=REMOTE
clusterName=docker-cluster
transportAddresses=127.0.0.1:9300,127.0.0.1:9301
I've killed a lot of time with transportAddresses configuration as it's documented to use square brackets and square quotes transportAddresses=["192.168.1.1:9300","192.168.1.2:9300"] but it does not work. Configuration listed above contains actual working configuration syntax.
My docker-compose.yml:
version: '3.7'
services:
es01:
container_name: "es01"
image: ktorek/liferay7-elasticsearch:latest
environment:
- node.name=es01
- node.data=true
- cluster.name=docker-cluster
- xpack.security.enabled=false
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=es02"
ports:
- "9300:9300"
- "9200:9200"
networks:
- mynetwork
volumes:
- es01-data:/usr/share/elasticsearch/data
ulimits:
memlock:
soft: -1
hard: -1
es02:
container_name: "es02"
image: ktorek/liferay7-elasticsearch:latest
environment:
- node.name=es02
- node.data=true
- cluster.name=docker-cluster
- xpack.security.enabled=false
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=es01"
ports:
- "9301:9300"
- "9201:9200"
networks:
- mynetwork
volumes:
- es02-data:/usr/share/elasticsearch/data
ulimits:
memlock:
soft: -1
hard: -1
networks:
mynetwork:
name: mynetwork
driver: bridge
ipam:
config:
- subnet: 172.30.29.0/24
I have an elasticsearch image that is being used as a base image for multiple containers. I am wondering if there is any way to pre-configure an ingest pipeline such that the process of creating the image and building a container also creates the pipeline for me? It'd be great if the base image comes with the pipeline that i want it to have, otherwise I'd have to docker exec into each container that uses this image and send a curl request in each one to create the pipeline.
Right now I'm thinking that I have to add a curl to the elasticsearch server (after it starts) in docker-entrypoint.sh, but i'm not sure if there's any other way
I can advice you to use docker-compose. I personally find it very convenient. With one file you can configure a whole stack.
Here is an example to help you start:
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- node.name=node-test1
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- node-test1data:/usr/share/elasticsearch/data
ports:
- 9200:9200
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: elasticsearch2
environment:
- cluster.name=docker-cluster
- node.name=node-test2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- node-test2data:/usr/share/elasticsearch/data
kibana:
image: docker.elastic.co/kibana/kibana:6.3.2
container_name: kibana
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://elasticsearch:9200
depends_on:
- elasticsearch
logstash:
image: docker.elastic.co/logstash/logstash:6.3.2
container_name: logstash
ports:
- "5000:5000"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
I am trying to implement elastic search in my rails web app. I am using docker. I used this link for reference. My docker-compose.yml file is:
mysql:
image: mysql:5.6.34
ports:
- "3002:3002"
volumes_from:
- dbdata
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=dev
dbdata:
image: tianon/true
volumes:
- /var/lib/mysql
web:
build: .
environment:
RAILS_ENV: development
ports:
- '3000:3000'
volumes_from:
- appdata
links:
- "mysql"
- elasticsearch
appdata:
image: tianon/true
volumes:
- ".:/workspace"
elasticsearch:
image: elasticsearch
ports:
- "9200:9200"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 1g
cap_add:
- IPC_LOCK
volumes:
- /usr/share/elasticsearch/data
When I am trying to run Student.__elasticsearch__.create_index! force:true as indicated in the above given link. I am getting following error:
You need to set ENV for ELASTICSEARCH_URL to correct value
ELASTICSEARCH_URL="http://<ip-of-your-docker-container>:9200"
As you have linked network, you can provide as bellow
ELASTICSEARCH_URL="http://elasticsearch:9200"
Links allow you to define extra aliases by which a service is reachable from another service. Any service can reach any other service at that service’s name
If no ENV is set, your rails app will use http://localhost:9200
Elasticsearch's official docker image documentation provides this docker-compose.yml example:
version: '2'
services:
elasticsearch1:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
container_name: elasticsearch1
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
elasticsearch2:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=elasticsearch1"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
volumes:
- esdata2:/usr/share/elasticsearch/data
networks:
- esnet
volumes:
esdata1:
driver: local
esdata2:
driver: local
networks:
esnet:
However, it doesn't explain how to customize the password. It does direct us to a X-Pack documentation page, but I refuse to believe I have to go through all that trouble just to change a password. Is there any simpler, canonical way of configuring a custom password for elasticsearch on a Docker Compose file?
Starting from 6.0 elasticsearch docker images has the ability to configure the password using the following environment variable - ELASTIC_PASSWORD.
For example:
docker run -e ELASTIC_PASSWORD=MagicWord docker.elastic.co/elasticsearch/elasticsearch-platinum:6.1.3
See:
https://www.elastic.co/guide/en/elasticsearch/reference/6.1/docker.html