I'm trying to configure docker-compose with kibana and elasticsearch and I would like to know do I need logstash as well?
No. If you don't need the Logstash functionality, you don't need it.
Simple example with Elasticsearch and Kibana would be:
---
version: '2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:$ELASTIC_VERSION
volumes:
- /usr/share/elasticsearch/data
ports:
- 9200:9200
kibana:
image: docker.elastic.co/kibana/kibana:$ELASTIC_VERSION
links:
- elasticsearch
ports:
- 5601:5601
Kibana credentials (if you are using version 5):
login: elastic
password: changeme
Related
I ran a docker-compose file to setup elasticsearch and Kibana on Ubuntu 18.04LTS. Kibana container is up and running just fine but elasticsearch goes down after about 10secs. I have restarted the containers and docker service several times and still got the same result. Been on this all day and hoping that I get some help.
Docker-Compose file.
version: "3.0"
services:
elasticsearch:
container_name: es-container
image: docker.elastic.co/elasticsearch/elasticsearch:7.16.3
environment:
- xpack.security.enabled=true
- xpack.security.audit.enabled=true
- "discovery.type=single-node"
- ELASTIC_PASSWORD=secretpassword
networks:
- es-net
ports:
- 9200:9200
kibana:
container_name: kb-container
image: docker.elastic.co/kibana/kibana:7.16.3
environment:
- ELASTICSEARCH_HOSTS=http://es-container:9200
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=secretpassword
networks:
- es-net
depends_on:
- elasticsearch
ports:
- 5601:5601
networks:
es-net:
driver: bridge
Also checked the logs on the es-container and it displayed;
Created elasticsearch keystore in
/usr/share/elasticsearch/config/elasticsearch.keystore
Audit logging can be only enabled with paid ES subscription and you don't provide any license info to your container.
How to setup login credentials for kibana gui with docker elk stack containers.
What arguments and environmental variables must be passed in docker-compose.yaml file to get this working.
For setting kibana user credentials for docker elk stack, we have to set xpack.security.enabled: true either in elasticsearch.yml or pass this as a environment variable in docker-compose.yml file.
Pass username & password as environment variable in docker-compose.yml like below:
version: '3.3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
ports:
- "9200:9200"
- "9300:9300"
configs:
- source: elastic_config
target: /usr/share/elasticsearch/config/elasticsearch.yml
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
ELASTIC_USERNAME: "elastic"
ELASTIC_PASSWORD: "MyPw123"
http.cors.enabled: "true"
http.cors.allow-origin: "*"
xpack.security.enabled: "true"
networks:
- elk
deploy:
mode: replicated
replicas: 1
logstash:
image: docker.elastic.co/logstash/logstash:6.6.1
ports:
- "5044:5044"
- "9600:9600"
configs:
- source: logstash_config
target: /usr/share/logstash/config/logstash.yml:rw
- source: logstash_pipeline
target: /usr/share/logstash/pipeline/logstash.conf
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
xpack.monitoring.elasticsearch.url: "elasticsearch:9200"
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "MyPw123"
networks:
- elk
deploy:
mode: replicated
replicas: 1
kibana:
image: docker.elastic.co/kibana/kibana:6.6.1
ports:
- "5601:5601"
configs:
- source: kibana_config
target: /usr/share/kibana/config/kibana.yml
networks:
- elk
deploy:
mode: replicated
replicas: 1
configs:
elastic_config:
file: ./elasticsearch/config/elasticsearch.yml
logstash_config:
file: ./logstash/config/logstash.yml
logstash_pipeline:
file: ./logstash/pipeline/logstash.conf
kibana_config:
file: ./kibana/config/kibana.yml
networks:
elk:
driver: overlay
Then add this following lines to kibana.yml:
elasticsearch.username: "elastic"
elasticsearch.password: "MyPw123"
Did not managed to get it working without adding XPACK_MONITORING & SECURITY flags to kibana's container and there was no need for a config file
However I was not able to use kibana user, even after logging in with elastic user and changing kibana's password through the UI.
NOTE: looks like you can't setup default built-in users other than elastic superuser in docker-compose through it's environment. I've tried several times with kibana and kibana_system to no success.
version: "3.7"
services:
elasticsearch:
image: elasticsearch:7.4.0
restart: always
ports:
- 9200:9200
environment:
- discovery.type=single-node
- xpack.security.enabled=true
- ELASTIC_PASSWORD=123456
kibana:
image: kibana:7.4.0
restart: always
ports:
- 5601:5601
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- XPACK_MONITORING_ENABLED=true
- XPACK_MONITORING_COLLECTION_ENABLED=true
- XPACK_SECURITY_ENABLED=true
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD="123456"
depends_on:
- elasticsearch
SOURCE
NOTE: looks like this won't work with 8.5.3, Kibana won't accept superuser elastic.
Update
I was able to setup 8.5.3 but with a couple twists. I would build the whole environment, then in elastic's container run the setup-passwords auto
bin/elasticsearch-setup-passwords auto
Grab the auto generated password for kibana_system user and replace it in docker-compose then restart only kibana's container
Kibana 8.5.3 with environment variables:
kibana:
image: kibana:8.5.3
restart: always
ports:
- 5601:5601
environment:
- ELASTICSEARCH_USERNAME="kibana_system"
- ELASTICSEARCH_PASSWORD="sVUurmsWYEwnliUxp3pX"
Restart kibana's container:
docker-compose up -d --build --force-recreate --no-deps kibana
NOTE: make sure to use --no-deps flag otherwise it will restart elastic container if tagged to kibana's
I've the following docker compose file. I'm trying to connect elastic search running in another machine to kibana.
version: '3.3'
services:
kibana_ci:
image: docker.elastic.co/kibana/kibana:6.3.2
environment:
ELASTICSEARCH_URL: http://my_domain:9200
container_name: kibana_ci
command: kibana
ports:
- "5601:5601"
But kibana is keep trying to connect to http://elasticsearch:9200/ url. I've also tried with following options which didnt work.
environment:
- "ELASTICSEARCH_URL=http://my_domain:9200"
environment:
- "KIBANA_ELASTICSEARCH_URL=http://my_domain:9200"
environment:
KIBANA_ELASTICSEARCH_URL: http://my_domain:9200
environment:
elasticsearch.url: http://my_domain:9200
How can I change the url in docker compose file (without overriding kibana.yml file).
This compose file works for me:
version: '3.3'
services:
kibana:
image: docker.elastic.co/kibana/kibana:6.3.2
environment:
SERVER_NAME: kibana.example.org
ELASTICSEARCH_URL: http://my_domain
You don't need to define default port 9200.
kibana_1 | {"type":"log","#timestamp":"2018-09-20T16:58:31Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://my_domain:9200/"}
kibana_1 | {"type":"log","#timestamp":"2018-09-20T16:58:31Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibana_1 | {"type":"log","#timestamp":"2018-09-20T16:58:34Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://my_domain:9200/"}
kibana_1 | {"type":"log","#timestamp":"2018-09-20T16:58:34Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
For those who will face the same issue in Kibana 7.5, you will have to use the ELASTICSEARCH_HOSTS environment variable instead of ELASTICSEARCH_URL, like below:
kibana:
image: docker.elastic.co/kibana/kibana:7.5.2
container_name: kibana
environment:
ELASTICSEARCH_HOSTS: http://es01:9200
ports:
- 5601:5601
depends_on:
- es01
networks:
- elastic
You can also consult via the following link the list of all environment variables available, and how to setup in a docker environment:
https://www.elastic.co/guide/en/kibana/7.5/docker.html
I'm not getting any logging output from the RestApi which is suppose to output in Kibana. All I get is this from Kibana: No results found :(
Containers are running fine in docker with RestApi, Elasticsearch and Kibana. So I do suspect there is something in docker-compose.yml that is missing and I can't seem to figure that out. Been looking for a long time with no luck. Hopefully you can see where I have gone wrong.
Still new to programming and eager to learn. Appreciate your help.
This is how the yml file looks like in .net core:
version: '3.1'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
container_name: elasticsearch
ports:
- "9200:9200"
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
kibana:
image: docker.elastic.co/kibana/kibana:6.2.4
container_name: kibana
ports:
- "5601:5601"
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
RestApi:
build:
context: C:\foo\RestApi
dockerfile: Dockerfile
image: docker.restapi
container_name: RestApi
ports:
- "9000:80"
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
volumes:
elasticsearch-data
I suggest to add the links tag.
Here a snippet of my configuration
elasticsearch:
image: elasticsearch:5.6.8
expose:
- 9200
ports:
- "9200:9200"
kibana:
image: kibana:6.2.3
links:
- "elasticsearch"
ports:
- "5601:5601"
Hope this helps.
I have an elk docker-compose file that I think cannot run on my server because the docker-compose version is too old...
$ docker-compose -version
docker-compose version 1.6.2, build 4d72027
And here is my docker-compose file...
version: '2'
services:
elasticsearch:
image: elasticsearch:5
command: elasticsearch
environment:
# This helps ES out with memory usage
- ES_JAVA_OPTS=-Xmx1g -Xms1g
volumes:
# Persist elasticsearch data to a volume
- elasticsearch:/usr/share/elasticsearch/data
# Extra ES configuration options
- ./es/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
ports:
- "9200:9200"
- "9300:9300"
logstash:
image: logstash:5
command: logstash -w 4 -f /etc/logstash/conf.d/logstash.conf
environment:
# This helps Logstash out if it gets too busy
- LS_HEAP_SIZE=2048m
volumes:
# volume mount the logstash config
- ./logstash/logstash.conf:/etc/logstash/conf.d/logstash.conf
- /Users/rriviere/workspace/parks-dpe/activities-api-v1/app-logs/:/tmp/app-logs
ports:
# GELF port for Docker logs
- "12201:12201/udp"
# UDP port for syslogs
- "5000:5000/udp"
# Default TCP port
- "5001:5001"
links:
- elasticsearch
kibana:
image: kibana:5
environment:
# Point Kibana to the elasticsearch container
- ELASTICSEARCH_URL=http://elasticsearch:9200
ports:
- "5601:5601"
links:
- elasticsearch
kopf:
image: rancher/kopf:v0.4.0
ports:
- "8080:80"
environment:
KOPF_ES_SERVERS: "elasticsearch:9200"
links:
- elasticsearch
volumes:
elasticsearch:
Whilst I'm not looking for an exact answer here can someone help me with what is required to create myself a docker compose v1 file that would do the same things.
thanks
you better check the docs for more details but just looking at your file, I think if you remove following lines then you are good to go:
version: '2'
services:
version 2 has some additional features which I dont see in your case, so you are pretty compatible with v1.