I am trying to implement elastic search in my rails web app. I am using docker. I used this link for reference. My docker-compose.yml file is:
mysql:
image: mysql:5.6.34
ports:
- "3002:3002"
volumes_from:
- dbdata
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=dev
dbdata:
image: tianon/true
volumes:
- /var/lib/mysql
web:
build: .
environment:
RAILS_ENV: development
ports:
- '3000:3000'
volumes_from:
- appdata
links:
- "mysql"
- elasticsearch
appdata:
image: tianon/true
volumes:
- ".:/workspace"
elasticsearch:
image: elasticsearch
ports:
- "9200:9200"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 1g
cap_add:
- IPC_LOCK
volumes:
- /usr/share/elasticsearch/data
When I am trying to run Student.__elasticsearch__.create_index! force:true as indicated in the above given link. I am getting following error:
You need to set ENV for ELASTICSEARCH_URL to correct value
ELASTICSEARCH_URL="http://<ip-of-your-docker-container>:9200"
As you have linked network, you can provide as bellow
ELASTICSEARCH_URL="http://elasticsearch:9200"
Links allow you to define extra aliases by which a service is reachable from another service. Any service can reach any other service at that service’s name
If no ENV is set, your rails app will use http://localhost:9200
Related
hi i want to connect to Elasticsearch inside my app which is defined as "cog-app" service in docker-compose.yml along with ditsro elasticsearch and kibana
i am not able to connect to elasticsearch when i run docker file, can you please tell me how i can connect elasticsearch service to app service
i have defined elasticsearch in cog-app service, and im getting connection failure with elasticsearch
version: "3"
services:
cog-app:
image: app:2.0
build:
context: .
dockerfile: ./Dockerfile
stdin_open: true
tty: true
ports:
- "7111:7111"
environment:
- LANG=C.UTF-8
- LC_ALL=C.UTF-8
- CONTAINER_NAME=app
volumes:
- /home/developer/app:/app
odfe-node1:
image: amazon/opendistro-for-elasticsearch:1.13.2
container_name: odfe-node1
environment:
- cluster.name=odfe-cluster
- node.name=odfe-node1
- discovery.seed_hosts=odfe-node1,odfe-node2
- cluster.initial_master_nodes=odfe-node1,odfe-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "ES_JAVA_OPTS=-Xms2g -Xmx2g" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- odfe-data1:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
odfe-node2:
image: amazon/opendistro-for-elasticsearch:1.13.2
container_name: odfe-node2
environment:
- cluster.name=odfe-cluster
- node.name=odfe-node2
- discovery.seed_hosts=odfe-node1,odfe-node2
- cluster.initial_master_nodes=odfe-node1,odfe-node2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- odfe-data2:/usr/share/elasticsearch/data
networks:
- odfe-net
kibana:
image: amazon/opendistro-for-elasticsearch-kibana:1.13.2
container_name: odfe-kibana
ports:
- 5601:5601
expose:
- "5601"
environment:
ELASTICSEARCH_URL: https://odfe-node1:9200
ELASTICSEARCH_HOSTS: https://odfe-node1:9200
networks:
- odfe-net
volumes:
odfe-data1:
odfe-data2:
networks:
odfe-net:
please tell me how two services can communicate with each other
As the elasticsearch service is running in another container, localhost is not valid. You should use odfe-node1:9200 as the url
I am running ElasticSearch instance on my Ubuntu server using docker container.
during a large insert-or-update request I get the following exception.
OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 413 from: POST /_bulk?pretty=true&error_trace=true
It sounds like I need to increase the http.max_content_length content from the default 100mb.
I bring up my docker instance using the following docker-compose
version: '3.4'
services:
# Nginx proxy
nginx-proxy:
image: jwilder/nginx-proxy
container_name: nginx-proxy
ports:
- 80:80
- 443:443
restart: always
networks:
- nginx-proxy
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- /etc/nginx/vhost.d:/etc/nginx/vhost.d:ro
- /etc/certificates:/etc/nginx/certs
# ElasticSearch instance
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
container_name: es01
environment:
- VIRTUAL_HOST=my.elastic-domain.com
- VIRTUAL_PORT=9200
- ELASTIC_PASSWORD=mypassword
- xpack.security.enabled=true
- discovery.type=single-node
- http.max_content_length=3000mb
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add:
- IPC_LOCK
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
expose:
- 9200
networks:
- nginx-proxy
depends_on:
- nginx-proxy
volumes:
data01:
driver: local
networks:
nginx-proxy:
default:
external:
name: nginx-proxy
As you can see, I tried to increase the value by setting an environment variable http.max_content_length=3000mb
Also, in Nginx proxy, I set the client_max_body_size 0; to ensure the proxy allow unlimited size.
How to set the http.max_content_length value for ElasticSearch container using docker container?
you can modify the elasticsearch.yml file and add the
http.max_content_length=3000mb
This file is in your elastic docker conntainer /config/elasticsearch.yml
I have a small app with a python backend where I'm streaming and classifying tweets in real-time.
I use elasticsearch to collect classified tweets and Kibana to make visualizations based on es data.
In my frontend, I just use kibana visualizations.
For the moment, I'm trying to run my application in a multi-node swarm as a services stack but I'm having problems with my compose file.
I tried to start with elastisearch and to use this info https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html but didn't help, and I didn'd succed to deploy my docker-compose file even with just elasticsearch serivce.
This is my yml file:
version: '3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- 'ES_JAVA_OPTS=-Xms512m -Xmx512m'
ulimits:
memlock:
soft: -1
hard: -1
ports:
- '9200:9200'
kibana:
image: docker.elastic.co/kibana/kibana:7.6.2
ports:
- '5601:5601'
Below is the docker-compose file which works for a single node in a development environment, which have disabled security and has discovery.type=single-node param to make sure elasticsearch production bootstrap checks are not kicked in.
version: '2.2'
services:
#Elasticsearch Docker Images: https://www.docker.elastic.co/
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
container_name: elasticsearch
environment:
- xpack.security.enabled=false
- discovery.type=single-node
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add:
- IPC_LOCK
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
volumes:
elasticsearch-data:
driver: local
networks:
elastic:
external: true
I am looking for a working version of a docker-compose file that starts up kibana and elasticsearch together on docker for mac at version 7.3.2. I've followed the most recent instructions on kibana and elasticsearch's 7.3.2 documentation and my docker-compose.yml file below is the union of what I gathered from both docs. (The kibana doc was the most vague with respect to the docker compose config). I've also tried following other stack overflow articles (written for older versions) but they don't seem to work with the latest versions. I now suspect I'm missing something version specific. 7.3.1 didn't work with the same config either.
I should note that the elasticsearch portion of the file works fine; I can hit http://localhost:9200 and I get a json response. However Kibana's url (http://localhost:5601) returns Kibana server is not ready yet with this error:
kibana | {"type":"log","#timestamp":"2019-09-12T21:45:04Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: http://elasticsearch:9200/"}
This is my best attempt so far:
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
container_name: es01
environment:
- node.name=es01
- discovery.seed_hosts=es02
- cluster.initial_master_nodes=es01,es02
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
container_name: es02
environment:
- node.name=es02
- discovery.seed_hosts=es01
- cluster.initial_master_nodes=es01,es02
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.3.2
ports:
- 5601:5601
networks:
- esnet
environment:
SERVER_NAME: kibana.example.org
ELASTICSEARCH_URL: http://elasticsearch:9200
volumes:
esdata01:
driver: local
esdata02:
driver: local
networks:
esnet:
Docker Compose automatically creates a private Docker network for you, and within that, the names of the service: blocks are valid hostnames.
When you set
ELASTICSEARCH_URL: http://elasticsearch:9200
None of your containers are named elasticsearch so the hostname lookup fails, but if you pick either node es01 or es02 it will work
ELASTICSEARCH_URL: http://es01:9200
(Note that you don’t explicitly need a networks: definition for this to work, Compose will create a network named default for you. You also don’t need to explicitly set container_name: unless you’re planning on trying to manage the same containers with non-Compose tooling.)
Use ELASTICSEARCH_HOSTS: http://es01:9200 instead of ELASTICSEARCH_URL to update the environment from the docker-compose.yml file. Here is the Elasticsearch documentation about environment variable configuration https://www.elastic.co/guide/en/kibana/current/docker.html#environment-variable-config.
You need to add network configuration also in kibana service like
networks:
- esnet
and in ELASTICSEARCH_HOST: http://es01:9200
note es01 is your container name
I don't know if this a good question or not. I've never worked on elastic search before.
I'm getting "[WARN ][o.e.d.c.ParseField ] [vLJycm6] Deprecated field [disable_coord] used, replaced by [disable_coord has been removed]" in the docker output log when I start running the elastic search container. I'm using this container for graylog3.
Do I need to be concerned about this "warning" from ElasticSearch?
This is code is part of docker compose file
mongodb:
container_name: mongodb
image: mongo:latest
restart: on-failure
networks:
- dev
volumes:
- mongodbdata_dev:/data/db
ports:
- '27017:27017'
elasticsearch:
container_name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
volumes:
- elasticsearchdata_dev:/usr/share/elasticsearch/data
networks:
- dev
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
graylog:
container_name: graylog
image: graylog/graylog:3.0
volumes:
- graylogdata_dev:/usr/share/graylog/data
networks:
- dev
environment:
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
- GRAYLOG_ROOT_PASSWORD_SHA2=somesha
- GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
links:
- mongodb:mongo
- elasticsearch
depends_on:
- mongodb
- elasticsearch
ports:
- 9000:9000
- 1514:1514
- 1514:1514/udp
- 12201:12201
- 12201:12201/udp
Please let me know if you need any other information.
Apparently Graylog still has some option for elasticsearch 5 and since you're using 6+, it's complaining.But no harm done until Elasticsearch 5 support is removed from graylog.So ignore for now