Downgrading elasticsearch in docker-compose.yml leads to license error - docker

I have the following docker-compose.yml file:
...
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.4.3
container_name: elasticsearch-cust-comp
...
I've previously run it with another elasticsearch version:
...
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
container_name: elasticsearch-cust-comp
...
Since changing the version to 5.4.3, I'm getting this error in docker logs for the Elasticsearch container:
Unknown license version found, please upgrade all nodes to the latest elasticsearch-license plugin
My guess is that version 6.5.4 of Elasticsearch is still running somewhere and that it creates issues when I'm trying to run 5.4.3. But as far as I know, I've shut down all Elasticsearch containers currently running with docker-compose down, and docker ps shows no processes after this. Still, when I run docker-compose up -d with version 5.4.3, it gives me this error. Running 6.5.4 works fine. What do I need to do to be able to run version 5.4.3?
EDIT:
This is the whole part regarding elasticsearch in the docker-compose.yml. As you can see, xpack is already disabled:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.4.3
container_name: elasticsearch-cust-comp
ports:
- 9200:9200
- 9300:9300
volumes:
- cust-comp-elastic:/usr/share/elasticsearch/data
- ./cust/externalConfig/elasticsearch/config/hunspell/:/usr/share/elasticsearch/config/hunspell/
# - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
# - ./fwhome/elasticsearch/cust-comp:/usr/share/elasticsearch/config/cust-comp
networks:
- cust-comp
environment:
- cluster.name=i3-elasticsearch
- xpack.security.enabled=false
- xpack.monitoring.enabled=false
- xpack.ml.enabled=false
- xpack.graph.enabled=false
- xpack.watcher.enabled=false
restart: unless-stopped

In doubt, try:
docker system prune
This will clean unused containers/images.

Related

Dockerimage working on pull but not on pull image directive in yml file?

I have a dockerimage on a gitlab registry.
when I (after login on a target machine)
docker run -d -p 8081:8080/tcp gitlab.somedomain.com:5050/root/app
the laravel app is available and running and reachable. Things like php artisan config:clear are working. when I enter the container everything looks fine.
But I don't have any services running. So I had the idea to create a yml file to docker-compose run to set things up in docker-compose-gitlab.yml
version: '3'
services:
mysql:
image: mysql:5.7
container_name: my-mysql
environment:
- MYSQL_ROOT_PASSWORD=***
- MYSQL_DATABASE=dbname
- MYSQL_USER=username
- MYSQL_PASSWORD=***
volumes:
- ./data/mysql:/var/lib/mysql
ports:
- "3307:3306"
application:
image: gitlab.somedomain.com:5050/root/app:latest
build:
context: .
dockerfile: ./Dockerfile
container_name: my-app
ports:
- "8081:8080"
volumes:
- .:/application
env_file: .env.docker
working_dir: /application
depends_on:
- mysql
links:
- mysql
calling docker-compose --verbose -f docker-compose-gitlab.yml up shows me that the mysql service is created and working, the app seems also be creeated but then fails ... exiting with code 0 - no further message.
If I add commands in my yml like php artisan config:clear the error gets even unclearer for me: it says it cannot find artisan and it seems as if the command is executed outside the container ... exiting with code 1. (artisan is a helper and executed via php)
When I call the docker-compose with -d and then do docker ps I can only see mysql running but not the app.
When I use both strategies, the problem is, the two container do not share a common network and can so not work together.
What did I miss? Is this the wrong strategy?
The problem is, that I let a volume directive left over which overwrites my entier application with an empty directory.
You can just leave that out.
version: '3'
services:
mysql:
image: mysql:5.7
container_name: my-mysql
environment:
- MYSQL_ROOT_PASSWORD=***
- MYSQL_DATABASE=dbname
- MYSQL_USER=username
- MYSQL_PASSWORD=***
volumes:
- ./data/mysql:/var/lib/mysql
ports:
- "3307:3306"
application:
image: gitlab.somedomain.com:5050/root/app:latest
build:
context: .
dockerfile: ./Dockerfile
container_name: my-app
ports:
- "8081:8080"
## volumes:
## - .:/application ## this would overwrite the app
env_file: .env.docker
working_dir: /application
depends_on:
- mysql
links:
- mysql
You can debug the network of the containers listing the networks with docker network ls
then when the list is shown inspect the compose network with docker inspect <ComposeNetworkID>
Once you are shure that your services are not in the same network, remove your containers and recreate it again with docker-compose -f docker-compose-gitlab.yml up
If you notice they are in the same network try to use the container name instead localhost to reach each other, if it is the case.

Restart docker compose with a different command

I've an application running on a server, but somehow the server rebooted but some docker services could restart, another not.
docker-compose ps:
Name Command State Ports
------------------------------------------------------------------------------------------------------------
elasticsearch /usr/local/bin/docker-entr ... Up 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
kibana sh -c ./bin/kibana-plugin ... Restarting
logstash /usr/local/bin/docker-entr ... Up 5044/tcp, 9600/tcp
If I try to see the logs of kibana by docker kibana ps:
Plugin kbn_radar already exists, please remove before installing a new version
Found previous install attempt. Deleting...
Attempting to transfer from file:///usr/share/kibana/config/kbn_radar.zip
Transferring 3686700 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
The problem is: kbn_radar takes a long time to restart, so I want to restart the kibana service without needing to restart the other applications. I've tried to change my .yml file where I've run the commands to start de plugins:
kibana:
image: docker.elastic.co/kibana/kibana:6.8.0
command:
- sh
- -c
- './bin/kibana-plugin install file:///usr/share/kibana/config/kbn_radar.zip && ./bin/kibana-plugin install file:///usr/share/kibana/config/ob-kb-funnel-6.8.zip && exec /usr/local/bin/kibana-docker'
So at the end, my docker compose was:
docker-compose.yml:
version: "3"
networks:
elasticsearch-net-624:
services:
elasticsearch-products-624-service:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
container_name: elasticsearch
restart: always
networks:
- elasticsearch-net-624
ports:
- "9200:9200"
- "9300:9300"
expose:
- "9200"
volumes:
- /home/docker/elastic.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /home/docker/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
- /docker/elastic/data:/usr/share/elasticsearch/data
- /docker/elastic/data/snapshots:/usr/share/elasticsearch/data/snapshots
kibana:
image: docker.elastic.co/kibana/kibana:6.8.0
command:
- sh
- -c
- 'exec /usr/local/bin/kibana-docker'
container_name: kibana
restart: always
hostname: kibana
networks:
- elasticsearch-net-624
environment:
- SERVER_NAME=kibana.localhost
- ELASTICSEARCH_URL=http://elasticsearch:9200
- ELASTICSEARCH_HOST=elasticsearch
- ELASTICSEARCH_PORT=9200
- XPACK_GRAPH_ENABLED=true
- XPACK_WATCHER_ENABLED=true
- XPACK_ML_ENABLED=true
- XPACK_MONITORING_ENABLED=true
- XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED
ports:
- "5601:5601"
expose:
- "5601"
links:
- elasticsearch-products-624-service
depends_on:
- elasticsearch-products-624-service
volumes:
- /home/docker/kibana.yml:/usr/share/kibana/config/kibana.yml
- /home/docker/ob-kb-funnel-6.8.zip:/usr/share/kibana/config/ob-kb-funnel-6.8.zip
- /home/docker/kbn_radar.zip:/usr/share/kibana/config/kbn_radar.zip
- /home/morpheus/docker/dashboard_app.js:/usr/share/kibana/src/legacy/core_plugins/kibana/public/dashboard/dashboard_app.js
logstash:
image: docker.elastic.co/logstash/logstash:6.8.0
container_name: logstash
restart: always
volumes:
- /home/docker/logstash.yml:/usr/share/logstash/config/logstash.yml
Finally I've tried to restart the service:
docker-compose -f docker-kibana.yml restart kibana
But, the service keeps trying to restart the plugins and if I run docker-compose ps, the command continues "sh -c ./bin/kibana-plugin ..."
How could I restart docker service with another command? Or restart my service without restarting the plugin that already exists?
I recommend that you create a build for your plugin and not do everything at the container start.
A simple dockerfile to fix your issue would look somewhat like this
FROM docker.elastic.co/kibana/kibana:6.8.0
COPY ob-kb-funnel-6.8.zip kbn_radar.zip /usr/share/kibana/config/
RUN ./bin/kibana-plugin install file:///usr/share/kibana/config/kbn_radar.zip &&
./bin/kibana-plugin install file:///usr/share/kibana/config/ob-kb-funnel-6.8.zip
ENTRYPOINT /usr/local/bin/kibana-docker
Next you would need to use docker-compose to build your image. We can do that by updating your service definition
kibana:
build:
context: ./kibana
container_name: kibana
restart: always
hostname: kibana
networks:
- elasticsearch-net-624
environment:
- SERVER_NAME=kibana.localhost
- ELASTICSEARCH_URL=http://elasticsearch:9200
- ELASTICSEARCH_HOST=elasticsearch
- ELASTICSEARCH_PORT=9200
- XPACK_GRAPH_ENABLED=true
- XPACK_WATCHER_ENABLED=true
- XPACK_ML_ENABLED=true
- XPACK_MONITORING_ENABLED=true
- XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED
ports:
- "5601:5601"
expose:
- "5601"
links:
- elasticsearch-products-624-service
depends_on:
- elasticsearch-products-624-service
volumes:
- /home/docker/kibana.yml:/usr/share/kibana/config/kibana.yml
- /home/morpheus/docker/dashboard_app.js:/usr/share/kibana/src/legacy/core_plugins/kibana/public/dashboard/dashboard_app.js
As you can see in the service definition we replaced image with build. We assume that your Dockerfile for kibana resides in a folder called kibana and also contains your plugin zip files.
next you can run docker-compose build and it will build you the required images for your compose stack.
The problem is that when you run a docker-compose or a docker stack, a context is created with all the initial data. If you later change this data, for example the command in your case, it will not take effect unless you restart the whole context, that is, unless you bring down and up again the docker-compose or stack.
However, you might try your luck with the following:
Edit the compose with the command you want to run now.
Remove the kibana container at all. I mean, don't try to restart kibana with docker-compose, but remove the container. docker rm -f dir_kibana
Run docker-compose up again. It should detect that kibana is missing and run it again.

docker-compose.yml for elasticsearch 7.0.1 and kibana 7.0.1

I am using Docker Desktop with linux containers on Windows 10 and would like to launch the latest versions of the elasticsearch and kibana containers over a docker compose file.
Everything works fine when using some older version like 6.2.4.
This is the working docker-compose.yml file for 6.2.4.
version: '3.1'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
container_name: elasticsearch
ports:
- "9200:9200"
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
networks:
- docker-network
kibana:
image: docker.elastic.co/kibana/kibana:6.2.4
container_name: kibana
ports:
- "5601:5601"
depends_on:
- elasticsearch
networks:
- docker-network
networks:
docker-network:
driver: bridge
volumes:
elasticsearch-data:
I deleted all installed docker containers and adapted the docker-compose.yml file by changing 6.2.4 to 7.0.1.
By starting the new compose file everything looks fine, both the elasticsearch and kibana containers are started. But after a couple of seconds the elasticsearch container exits (the kibana container is running further). I restarted everything, attached a terminal to the elasticsearch container and saw the following error message:
...
ERROR: [1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
...
What must be changed in the docker-compose.yml file to get elasticsearch 7.0.1 working?
Making a few changes worked for me -
Add cluster.initial_master_nodes to the elasticsearch service in compose -
environment:
- cluster.initial_master_nodes=elasticsearch
vm.max_map_count on the linux box kernel setting needs to be set to at least 262144 -
$ sudo sysctl -w vm.max_map_count=262144
For development mode, you can use below settings as well -
environment:
- discovery.type=single-node
Working compose file for me -
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.0.1
container_name: es01
environment:
- cluster.initial_master_nodes=es01
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200
For production mode, you must consider having multiple ES nodes/containers as suggested in the official documentation
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/docker.html#docker-cli-run-prod-mode

Docker service restart without order sequence when using docker-compose depends_on

I'm trying to setup a Sonarqube service in docker for windows and use mysql as database.
I am using below compose file, and using compose-file depends_on to control start up order:
db->sonarqube
But when docker services/windows restart, containers start up with no sequence.
Which will take error when sonarqube trying to connect mysql but mysql service didnt startup first.
version: '3'
services:
sonarqube:
image: sonarqube:6.5
container_name: sonarqube
restart: always
environment:
- SONARQUBE_JDBC_URL=jdbc:mysql://db:3306/sonar?useSSL=true&useUnicode=true&characterEncoding=utf8
- SONARQUBE_JDBC_USERNAME=sonar
- SONARQUBE_JDBC_PASSWORD=sonar
ports:
- 9000:9000
- 9002:9002
- 9092:9092
volumes:
- ../../volumes/data/sonarqube/conf:/opt/sonarqube/conf
- ../../volumes/data/sonarqube/data:/opt/sonarqube/data
- ../../volumes/data/sonarqube/extensions:/opt/sonarqube/extensions
- ../../volumes/data/sonarqube/lib/bundled-plugins:/opt/sonarqube/lib/bundled-plugins
depends_on:
- db
grafana:
container_name: grafana
image: grafana/grafana
ports:
- 3000:3000
volumes:
- ../../volumes/data/grafana-storage:/var/lib/grafana
restart: always
depends_on:
- db
# mysql service for sonarqube & grafana
db:
image: mysql:5.7
container_name: sonar-mysql
restart: always
environment:
- MYSQL_DATABASE=sonar
- MYSQL_USER=sonar
- MYSQL_PASSWORD=sonar
- MYSQL_ROOT_PASSWORD=Password1
- MAX_ALLOWED_PACKET=13421772800
volumes:
- ../../volumes/data/mysql:/var/lib/mysql
ports:
- 3306:3306
command: mysqld --max_allowed_packet=80M --federated --event_scheduler=1
After reading some of docker-compose official document,
They said the best way is to check the application code, like updating entrypoint scripts using waiting for other service online then start application.
It's the right way I think, but I still want to know if there is any way we can control containers startup sequence when docker service restart?
Thanks.

Docker-compose re-write file for version 2 to version 1

I have an elk docker-compose file that I think cannot run on my server because the docker-compose version is too old...
$ docker-compose -version
docker-compose version 1.6.2, build 4d72027
And here is my docker-compose file...
version: '2'
services:
elasticsearch:
image: elasticsearch:5
command: elasticsearch
environment:
# This helps ES out with memory usage
- ES_JAVA_OPTS=-Xmx1g -Xms1g
volumes:
# Persist elasticsearch data to a volume
- elasticsearch:/usr/share/elasticsearch/data
# Extra ES configuration options
- ./es/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
ports:
- "9200:9200"
- "9300:9300"
logstash:
image: logstash:5
command: logstash -w 4 -f /etc/logstash/conf.d/logstash.conf
environment:
# This helps Logstash out if it gets too busy
- LS_HEAP_SIZE=2048m
volumes:
# volume mount the logstash config
- ./logstash/logstash.conf:/etc/logstash/conf.d/logstash.conf
- /Users/rriviere/workspace/parks-dpe/activities-api-v1/app-logs/:/tmp/app-logs
ports:
# GELF port for Docker logs
- "12201:12201/udp"
# UDP port for syslogs
- "5000:5000/udp"
# Default TCP port
- "5001:5001"
links:
- elasticsearch
kibana:
image: kibana:5
environment:
# Point Kibana to the elasticsearch container
- ELASTICSEARCH_URL=http://elasticsearch:9200
ports:
- "5601:5601"
links:
- elasticsearch
kopf:
image: rancher/kopf:v0.4.0
ports:
- "8080:80"
environment:
KOPF_ES_SERVERS: "elasticsearch:9200"
links:
- elasticsearch
volumes:
elasticsearch:
Whilst I'm not looking for an exact answer here can someone help me with what is required to create myself a docker compose v1 file that would do the same things.
thanks
you better check the docs for more details but just looking at your file, I think if you remove following lines then you are good to go:
version: '2'
services:
version 2 has some additional features which I dont see in your case, so you are pretty compatible with v1.

Resources