How to connect metricbeat to elasticsearch and kibana with docker - docker

Question

I've setup elasticsearch and kibana with docker compose. elasticsearch is deployed on: localhost:9200 while kibana is deployed on localhost:5601
When trying to deploy metricbeat with docker run I got the following errors:
$ docker run docker.elastic.co/beats/metricbeat:6.3.2 setup -E setup.kibana.host=kibana:5601 -E output.elasticsearch.hosts=["localhost:9200"]
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://localhost:9200: Get http://localhost:9200: dial tcp [::1]:9200: connect: cannot assign requested address]
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://elasticsearch:9200: Get http://elasticsearch:9200: lookup elasticsearch on 192.168.65.1:53: no such host]
My docker-compose.yml:
# ./docker-compose.yml
version: "3.7"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
environment:
# - cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- elkdata:/usr/share/elasticsearch/data
ports:
- "9200:9200"
restart: always
kibana:
image: docker.elastic.co/kibana/kibana:6.3.2
volumes:
- kibana:/usr/share/kibana/config
ports:
- "5601:5601"
depends_on:
- elasticsearch
restart: always
volumes:
elkdata:
kibana:

Answer 1

First edit your docker-compose file by adding a name for default docker network:
version: "3.7"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
environment:
# - cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- elkdata:/usr/share/elasticsearch/data
ports:
- "9200:9200"
networks:
- my-network
restart: always
kibana:
image: docker.elastic.co/kibana/kibana:6.3.2
volumes:
- kibana:/usr/share/kibana/config
ports:
- "5601:5601"
networks:
- my-network
depends_on:
- elasticsearch
restart: always
volumes:
elkdata:
kibana:
networks:
my-network:
name: awesome-name
Execute docker-compose up and then start metricbeat with the below command:
$ docker run docker.elastic.co/beats/metricbeat:6.3.2 --network=awesome-name setup -E setup.kibana.host=kibana:5601 -E output.elasticsearch.hosts=["elasticsearch:9200"]
Explanation:
When you try to deploy metricbeat, you provide below envars:
setup.kibana.host=kibana:5601
output.elasticsearch.hosts=["localhost:9200"]
I will start with the second one. With docker run command, when you are starting metricbeat, you are telling the container that it can access elastic search on localhost:9200. So when the container starts, it will access localhost on port 9200 expecting to find elasticsearch running. But, as the container is a host isolated process with its own network layer, localhost resolves to the container itself, not to your docker host machine as you are expecting.
Regarding the setup of kibana host, you should firstly understand how docker-compose works. By default, when you execute docker-compose up, a docker network is created and all services defined on yml file are added to this network. Inside this network and only, services are accessible through their service name. For your case, as defined on yml file, their names would be elasticsearch, kibana.
So in order metricbeat container to be able to communicate with elasticsearch and kibana containers, it should be added to the same docker network. This can be achieved with setting --network flag on docker run command.
Another approach would be to share docker host's network with your containers by using network mode host, but I would not recommend that.
References:
Docker compose
docker run