Here's my docker compose file:
version: '3.5'
services:
elasticsearch:
restart: always
build:
context: elasticsearch/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
# data
- ./data:/usr/share/elasticsearch/data:rw
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
ELASTIC_PASSWORD: password
networks:
- elk-network
kibana:
restart: always
build:
context: kibana/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
ports:
- "5601:5601"
networks:
- elk-network
depends_on:
- elasticsearch
networks:
elk-network:
name: elk-network
driver: bridge
I do docker-compose up, each day my drive space gets lower by 3-4gb. If I bash into the containers and look for biggest files, there are no big files. If I do container ls, the containers are only 2gb and don't grow in size.
But if I do docker-compose down, and up again, all my disk space is restored. What is going on? How do I debug what is taking all my space for apparently no reason?
if I look for biggest files on my host machine I believe it's all in the /var/lib/docker (or something similar I lost the exact location before clearing all the space).
Try turning off swap in your host
sudo swapoff -a
Related
So, I've setup several container apps that use MariaDB as their db backend, using docker-compose.
Containers are setup as needed and therefore MariaDB gets installed each time on every container that uses the db.
For example, I have some containers (PHPMyAdmin, NGiNX-PM, etc.) that use MariaDB, and they, in turn, have a version of it installed within their container. I also have a separate container (MariaDB) that I would rather have shared amongst the other containered apps and, thereby, I'd only have to maintain one version of the db.
I've searched for a solution, but no luck. Needless to say, I'm a noob at docker.
The only thing I can come up with is that all the apps need to be installed through the same docker-compose.yaml file to use the same db? That would make for a very long file if I had many containers running, and I'd prefer to have a directory per app and all the app's contents available in this one location.
I'm sure there is a way, I just haven't been able to figure it out.
So this is what I've tried:
The following setup is what I've tried but I am unable to get it to work:
(/docker/apps/mariadb/mariadb.yml)
version: '3.9'
networks:
NET:
external: true
services:
#############################################################################################
# MariaDB (docker-compose -f mariadb.yml up -d) #
#############################################################################################
mariadb:
image: jsurf/rpi-mariadb:latest
restart: unless-stopped
environment:
- TZ=${TIMEZONE}
- MYSQL_DATABASE=dockerApps
- MYSQL_USER=root
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
volumes:
- $HOME/docker/apps/mariadb/db:/var/lib/mysql
expose:
- '3306'
networks:
- NET
(/docker/apps/nginxpm/nginxpm.yml)
version: '3.9'
networks:
NET:
external: true
services:
#############################################################################################
# NGiNX Proxy Manager (docker-compose -f nginxpm.yml up -d) #
#############################################################################################
nginxpm:
container_name: NGiNX_Proxy_Manager
image: 'jc21/nginx-proxy-manager:latest'
ports:
- '80:80'
- '81:81'
- '443:443'
volumes:
- ./config.json:/app/config/production.json
- ./data:/data
- ./letsencrypt:/etc/letsencrypt
networks:
- NET
depends_on:
- mariadb
(/docker/apps/phpmyadmin/phpmyadmin.yml)
version: "3.9"
networks:
NET:
external: true
services:
#############################################################################################
# phpMyAdmin (docker-compose up -d -OR- docker-compose -f phpmyadmin.yml up -d) #
#############################################################################################
phpmyadmin:
image: phpmyadmin:latest
container_name: phpMyAdmin
restart: unless-stopped
environment:
PMA_HOST: mariadb
PMA_USER: root
PMA_PASSWORD: ${MYSQL_PASSWORD}
volumes:
# Must add ServerName directive to end of file "ServerName 127.0.0.1"
- $HOME/docker/apps/phpmyadmin/apache2.conf:/etc/apache2/apache2.conf
ports:
- '8004:80'
networks:
- NET
Any help in this matter is greatly appreciated.
Ok, so after some more reading and testing, I've found the answer to my issue. I was assuming that "depends_on" was supposed to connect the containers, somehow. Not true!
I found that "external_links" is the correct way of connecting them.
So, my final docker-compose file looks like this:
(/docker/apps/nginxpm/nginxpm.yml)
version: '3.9'
networks:
NET:
external: true
services:
#############################################################################################
# NGiNX Proxy Manager (docker-compose -f nginxpm.yml up -d) #
#############################################################################################
nginxpm:
container_name: NGiNX_Proxy_Manager
image: 'jc21/nginx-proxy-manager:latest'
ports:
- '80:80'
- '81:81'
- '443:443'
volumes:
- ./config.json:/app/config/production.json
- ./data:/data
- ./letsencrypt:/etc/letsencrypt
networks:
- NET
external_links:
- mariadb
I am trying to build images for my app. However, when I run "docker-compose up" command, it builds some of the containers twice. I couldn't figure the reason of it. I think the tags cause this kind of situation, but I couldn't figure where 'latest' tag come from.
Here it is my docker-compose.yml:
version: '3.2'
services:
elasticsearch:
build:
context: elasticsearch/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./elasticsearch/config/elasticsearch.yml
target: /usr/share/elasticsearch/config/elasticsearch.yml
read_only: true
- type: volume
source: elasticsearch
target: /usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
ELASTIC_PASSWORD: changeme
# Use single node discovery in order to disable production mode and avoid bootstrap checks.
# see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
discovery.type: single-node
networks:
- elk
logstash:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./logstash/config/logstash.yml
target: /usr/share/logstash/config/logstash.yml
read_only: true
- type: bind
source: ./logstash/pipeline
target: /usr/share/logstash/pipeline
read_only: true
ports:
- "5044:5044"
- "5000:5000/tcp"
- "5000:5000/udp"
- "9600:9600"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
depends_on:
- elasticsearch
kibana:
build:
context: kibana/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./kibana/config/kibana.yml
target: /usr/share/kibana/config/kibana.yml
read_only: true
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
zookeeper:
image: 'bitnami/zookeeper:latest'
container_name: zookeeper
ports:
- "2181:2181"
networks:
- elk
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'bitnami/kafka:latest'
depends_on:
- zookeeper
ports:
- "9092:9092"
- "9093:9093"
networks:
- elk
environment:
KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
ALLOW_PLAINTEXT_LISTENER: 'yes'
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_LISTENERS: CLIENT://:9092,EXTERNAL://:9093
KAFKA_CFG_ADVERTISED_LISTENERS: CLIENT://kafka:9092,EXTERNAL://localhost:9093
KAFKA_INTER_BROKER_LISTENER_NAME: CLIENT
links:
- logstash
app:
container_name: "ml-pipeline"
build: .
ports:
- "7000:7000"
- "5001:5001"
depends_on:
- kafka
- elasticsearch
- logstash
networks:
- elk
links:
- kafka
networks:
elk:
driver: bridge
volumes:
elasticsearch:
And output of this is:
As you can see there are duplicate images. How can I solve it ?
Actually there is nothing that indicates that docker-compose built the images twice. Your screenshot shows that the images have multiple tag names. But without further context it's hard to say how this happened and how docker-compose was involved in this.
One possible cause for this:
the pre-built images from docker.elastic.co were downloaded by docker pull docker.elastic.co/... or another docker run command
docker-compose up was looking for images named twitter-stream-dl-docker_* and since it couldn't find them triggered a docker-compose build
docker-compose build built the images - but using the docker build cache it could re-use all layers of the existing docker.elastic.co/... images which must have been built from the same source
the new built images resulted in the same final images which were then tagged with the name expected by docker-compose, i.e. twitter-stream-dl-docker_*
If you want to force a new local built either:
build without using the cache: docker-compose build --no-cache
delete the downloaded images: docker rmi docker.elastic.co/...
All 3 ELK containers have a build context with a Dockerfile that by default only consists of a FROM line. In the Dockerfiles you could add additional plugins.
part of your docker-compose.yml:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
logstash/Dockerfile:
ARG ELK_VERSION
# https://github.com/elastic/logstash-docker
FROM docker.elastic.co/logstash/logstash:${ELK_VERSION}
# Add your logstash plugins setup here
# Example: RUN logstash-plugin install logstash-filter-json
docker-compose pulls the Image docker.elastic.co/logstash/logstash:${ELK_VERSION} and builds it's own version version twitter-stream-dl-docker_XXX. Since the build doesn't do anything it simply tags the old image with the new tag so they have the same Image ID.
In case you're wondering your folder's name is twitter-stream-dl-docker so the images have that tag (or you used docker-compose -p twitter-stream-dl-docker).
I hope that clears things up, but feel free to ask anything that's ambigious.
I'm running one elasticsearch with
version: '3'
services:
elasticsearch:
build:
context: .
dockerfile: ./compose/elasticsearch/Dockerfile
args:
- VERSION=${VERSION}
- MEM=${MEM}
- ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT_DEV}
- CLUSTER_NAME=${CLUSTER_NAME_DEV}
- ENV=${ENV_DEV}
container_name: elasticsearch
network_mode: host
environment:
- discovery.type=single-node
volumes:
- /var/lib/elasticsearch:/usr/share/elasticsearch/data
logstash:
build:
context: .
dockerfile: ./compose/logstash/Dockerfile
args:
- VERSION=${VERSION}
- ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST_DEV}
- ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT_DEV}
- DB_HOST=${DB_HOST_DEV}
- DB_NAME=${DB_NAME_DEV}
- ENV=${ENV_DEV}
container_name: logstash
network_mode: host
volumes:
- /opt/logstash/data:/usr/share/logstash/data
dns:
- 192.168.1.1 # IP necessary to connect to a database instance external to where the server in which the container is running
kibana:
build:
context: .
dockerfile: ./compose/kibana/Dockerfile
args:
- VERSION=${VERSION}
- ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST_DEV}
- ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT_DEV}
container_name: kibana
depends_on:
- elasticsearch
network_mode: host
nginx:
build:
context: .
dockerfile: ./compose/nginx/Dockerfile
args:
- KIBANA_HOST=${KIBANA_HOST_DEV}
- KIBANA_PORT=${KIBANA_PORT_DEV}
container_name: nginx
network_mode: host
depends_on:
- kibana
apm:
build:
context: .
dockerfile: ./compose/apm/Dockerfile
args:
- VERSION=${VERSION}
- ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST_DEV}
- ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT_DEV}
- APM_PORT=${APM_PORT_DEV}
container_name: apm
depends_on:
- elasticsearch
network_mode: host
(I think this one uses host's /var/lib/elasticsearch when container access /usr/share/elasticsearch/data and the data is persisted in the /var/lib/elasticsearch of the host)
Another one with
version: '3'
services:
elasticsearch-search:
restart: always
build:
context: .
dockerfile: ./compose/elasticsearch/Dockerfile
args:
- VERSION=${VERSION}
- ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT_SEARCH_DEV}
- MEM=${MEM_SEARCH}
- CLUSTER_NAME=${CLUSTER_NAME_SEARCH_DEV}
- ENV=${ENV_DEV}
container_name: elasticsearch-search
network_mode: host
environment:
- discovery.type=single-node
volumes:
- /etc/localtime:/etc/localtime:ro
- data:/usr/share/elasticsearch/data
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
kibana:
build:
context: .
dockerfile: ./compose/kibana/Dockerfile
args:
- VERSION=${VERSION}
- ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST_SEARCH_DEV}
- ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT_SEARCH_DEV}
container_name: kibana-search
depends_on:
- elasticsearch-search
network_mode: host
volumes:
- /etc/localtime:/etc/localtime:ro
- data:/usr/share/elasticsearch/data
volumes:
data:
(I'm not sure how this one works out, but I guess docker provides persistant storage that can be accessed via /usr/share/elasticsearch/data from container)
When I run them at the same time, I expect the two elasticsearch uses separate data. but it seems they are interfering with each other.
I have a kibana running which looks at the first ES.
When I run the first ES alone, I can see the data , but as soon as I run the second ES, there's nothing, no index-pattern, no dashboard.
What am I misunderstanding?
.env
ELASTICSEARCH_PORT_DEV=29200
ELASTICSEARCH_PORT_SEARCH_DEV=29300
most probably something is wrong with your docker-compose in term of volumes: sections.
second example has this at the top
volumes:
- data:/usr/share/elasticsearch/data
and this at the bottom:
volumes:
- /etc/localtime:/etc/localtime:ro
- data:/usr/share/elasticsearch/data
which means that at least two separate container have binding to the same local folder data. which is definitely way to see strange things, because something inside of those containers (ES is one of those) will try to recreate data storage hierarchy in hosts data folder.
can you just try defining volumes for first ES as:
volumes:
- ./data/es1:/usr/share/elasticsearch/data
and for second one as:
volumes:
- ./data/es2:/usr/share/elasticsearch/data
just make sure that ./data/es1 and ./data/es2 folders are there on your host before doing docker-compose up.
or you can post whole docker-compose.yml file so we can say what is wrong with it...
I'm using elk docker image and using the below docker-compose file to kick start the ELK containers and storing the data in volume.
version: '3.5'
services:
elasticsearch:
build:
context: elasticsearch/
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
logstash:
build:
context: logstash/
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
ports:
- "5000:5000"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
depends_on:
- elasticsearch
kibana:
build:
context: kibana/
volumes:
- ./kibana/config/:/usr/share/kibana/config:ro
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
networks:
elk:
driver: bridge
Kibana version 6.6.0
I used below command to start:
docker-compose up -d
Observed that all three containers are up and running and I can able to publish my data into kibana and It can be visualized.
I just had situation to down this compose file and start up . But When I do that activity all the earlier kibana data has been lost.
docker-compose down
Is there any way that I can permanently store those records in machine (Some where in linux box) as backup else any database?
Please help me on this.
You have to move elasticsearch data dir (/usr/share/elasticsearch/data) into a persistent docker volume like this:
elasticsearch:
build:
context: elasticsearch/
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
- ./es_data:/usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
I am trying to use kompose convert on my docker-compose.yaml files however, when I run the command:
kompose convert -f docker-compose.yaml
I get the output:
WARN Volume mount on the host "/home/centos/Sprint0Demo/Servers/elasticSearchConnector/etc/kafka-connect" isn't supported - ignoring path on the host
WARN Volume mount on the host "/home/centos/Sprint0Demo/Servers/elasticSearchConnector/etc/kafka-elasticsearch" isn't supported - ignoring path on the host
WARN Volume mount on the host "/home/centos/Sprint0Demo/Servers/elasticSearchConnector/etc/kafak" isn't supported - ignoring path on the host
It also says more warning for the other persistent volumes
My docker-compose file is:
version: '3'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.1
container_name: es01
environment:
[env]
ulimits:
nproc: 3000
nofile: 65536
memlock: -1
volumes:
- /home/centos/Sprint0Demo/Servers/elasticsearch:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- kafka_demo
zookeeper:
image: confluentinc/cp-zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
volumes:
- /home/centos/Sprint0Demo/Servers/zookeeper/zk-data:/var/lib/zookeeper/data
- /home/centos/Sprint0Demo/Servers/zookeeper/zk-txn-logs:/var/lib/zookeeper/log
networks:
kafka_demo:
kafka0:
image: confluentinc/cp-kafka
container_name: kafka0
environment:
[env]
volumes:
- /home/centos/Sprint0Demo/Servers/kafkaData:/var/lib/kafka/data
ports:
- "9092:9092"
depends_on:
- zookeeper
- es01
networks:
kafka_demo:
schema_registry:
image: confluentinc/cp-schema-registry:latest
container_name: schema_registry
environment:
[env]
ports:
- 8081:8081
networks:
- kafka_demo
depends_on:
- kafka0
- es01
elasticSearchConnector:
image: confluentinc/cp-kafka-connect:latest
container_name: elasticSearchConnector
environment:
[env]
volumes:
- /home/centos/Sprint0Demo/Servers/elasticSearchConnector/etc/kafka-connect:/etc/kafka-connect
- /home/centos/Sprint0Demo/Servers/elasticSearchConnector/etc/kafka-elasticsearch:/etc/kafka-elasticsearch
- /home/centos/Sprint0Demo/Servers/elasticSearchConnector/etc/kafak:/etc/kafka
ports:
- "28082:28082"
networks:
- kafka_demo
depends_on:
- kafka0
- es01
networks:
kafka_demo:
driver: bridge
Does anyone know how I can fix this issue? I was thinking it has to do with the error message saying that its a volume mount vs host mount?
I have made some research and there are three things to point out:
kompose does not support volume mount on host. You might consider using emptyDir instead.
Kubernetes makes it difficult to pass in host/root volumes. You can try with
hostPath.
kompose convert --volumes hostPath works for k8s.
Also you can check out Compose on Kubernetes if you'd like to run things on a single machine.
Please let me know if that helped.