docker-compose command is unable to stop gracefully - docker

My custom docker-compose command won't stop gracefully. Why isn't the below working and how can I fix it properly (i.e. no SIGKILL)?
This is mentioned in the docs: https://docs.docker.com/compose/faq
Similar questions (none have helped - see test below):
Docker: gracefully stop django server
How to gracefully stop a Dockerized Python ROS2 node when run with docker-compose up?
Gunicorn graceful stopping with docker-compose
I've written these tests to demonstrate:
version: '3.3'
services:
# My personal use case
test1_string:
image: python:3.9.0rc1-alpine3.12
command: 'python -u -m smtpd -n -c DebuggingServer 0.0.0.0:1025'
# https://docs.docker.com/compose/faq/
# "Compose always uses the JSON form, so don’t worry if you override the command or entrypoint in your Compose file."
test2_list:
image: python:3.9.0rc1-alpine3.12
command: ['python', '-u', '-m', 'smtpd', '-n', '-c', 'DebuggingServer', '0.0.0.0:1025']
# Maybe it's an issue with networking and syscalls?
# Using nc once for a comparison to smtpd DebuggingServer above
test3_bash:
image: bash:5.0.18
command: ['bash', '-c', 'nc -k -l 4444 > filename.out']
# Something simpler, just a sleep.
test4_bash_sleep:
image: bash:5.0.18
command: ['bash', '-c', 'sleep 100']
# Apparently bash doesn't forward signals, but exec can help (?)
test5_bash_exec:
image: bash:5.0.18
command: ['bash', '-c', 'exec sleep 100']
# Print any signals sent to the executable
test6_bash_trap:
image: bash:5.0.18
command: ['bash', '-c', 'for s in HUP INT TERM KILL EXIT ; do trap "echo $$s; exit 0" $$s ; done ; sleep 100']
# Finally, use SIGKILL instead of SIGTERM. This works, but is overkill and not at all graceful.
test7_kill:
image: python:3.9.0rc1-alpine3.12
command: ['python', '-u', '-m', 'smtpd', '-n', '-c', 'DebuggingServer', '0.0.0.0:1025']
stop_signal: SIGKILL
This is the output, hitting Ctrl-C after theyre up:
$ docker-compose --version
docker-compose version 1.25.4, build unknown
$ docker --version
Docker version 19.03.11, build 42e35e6
$ docker-compose up
Creating docker_stop_test2_list_1 ... done
Creating docker_stop_test1_string_1 ... done
Creating docker_stop_test6_bash_trap_1 ... done
Creating docker_stop_test4_bash_sleep_1 ... done
Creating docker_stop_test7_kill_1 ... done
Creating docker_stop_test5_bash_exec_1 ... done
Creating docker_stop_test3_bash_1 ... done
Attaching to docker_stop_test4_bash_sleep_1, docker_stop_test5_bash_exec_1, docker_stop_test2_list_1, docker_stop_test3_bash_1, docker_stop_test6_bash_trap_1, docker_stop_test7_kill_1, docker_stop_test1_string_1
^CGracefully stopping... (press Ctrl+C again to force)
Stopping docker_stop_test5_bash_exec_1 ...
Stopping docker_stop_test3_bash_1 ...
Stopping docker_stop_test6_bash_trap_1 ...
Stopping docker_stop_test7_kill_1 ... done
Stopping docker_stop_test1_string_1 ...
Stopping docker_stop_test2_list_1 ...
Stopping docker_stop_test4_bash_sleep_1 ...
test7_kill is the only one that stops but it's using SIGKILL. How can I make any of the others work?

Related

I can't input after running command in docker, but I can if I execute it manually

I have a docker-compose file for starting a Terraria server, but after starting the server, I can't input any commands. If I start the server directly in my shell, I am able to input commands. How can I get the same result in docker as if I had run the command myself in a shell?
This is the desired behavior, which is what happens when I run it from my shell:
$ TerrariaServerVolume/TerrariaServer -pass xxx -port 7777 -world ~/absolute/path/TerrariaWorldsVolume/testWorldName.wld
Terraria Server v1.4.2.2
Listening on port 7777
Type 'help' for a list of commands.
: Server started
help // my input
Available commands:
... //list of commands
: % //I pressed Ctrl+c
$
This is what actually happens in my docker container:
$ sudo docker-compose up
Terraria Server v1.4.2.2
TerrariaServer_1 |
TerrariaServer_1 | Listening on port 7777
TerrariaServer_1 | Type 'help' for a list of commands.
TerrariaServer_1 |
TerrariaServer_1 | : Server started
^[[6;23
I don't know what ^[[6;23 is, but then here's me trying to input commands:
...
TerrariaServer_1 | : Server started
^[[6;23Rhelp
help
exit
stop
ljadgkljasdgl
^CGracefully stopping... (press Ctrl+C again to force)
Stopping terraria_TerrariaServer_1 ... done
$
This is my setup:
docker-compose.yml
version: "3"
services:
TerrariaServer:
image: "mono:6.8.0.96-slim"
ports:
- 7777:7777
expose:
- 7777
volumes:
- "./TerrariaServerVolume:/Terraria/Server"
- "./TerrariaWorldsVolume:/Terraria/Worlds"
environment:
- WorldName=testWorldName.wld
command: bash -c "/Terraria/Server/TerrariaServer -pass <password> -port 7777 -world /Terraria/Worlds/$WorldName"
stdin_open: true
tty: true
To type other commands after running docker-compose You need to use -d parameter.
Example:
docker-compose up -d
From docs:
-d, --detach Detached mode: Run containers in the background, print new container names.

Docker wait untill a service is completely ready

I'm dockerizing my existing Django application.
I have an entrypoint.sh script which run as entrypoint by the Dockerfile
ENTRYPOINT ["/app/scripts/docker/entrypoint.sh"]
It's content contains script to run migration when environment variable is set to migrate
#!/bin/sh
#set -e
# Run the command and exit with the custom message when the comamnd fails to run
safeRunCommand() {
cmnd="$*"
echo cmnd="$cmnd"
eval "$cmnd"
ret_code=$?
if [ $ret_code != 0 ]; then
printf "Error : [code: %d] when executing command: '$cmnd'\n" $ret_code
exit $ret_code
else
echo "Command run successfully: $cmnd"
fi
}
runDjangoMigrate() {
echo "Migrating database"
cmnd="python manage.py migrate --noinput"
safeRunCommand "$cmnd"
echo "Done: Migrating database"
}
# Run Django migrate command.
# The command is run only when environment variable `DJANGO_MANAGE_MIGRATE` is set to `on`.
if [ "x$DJANGO_MANAGE_MIGRATE" = 'xon' ] && [ ! "x$DEPLOYMENT_MODE" = 'xproduction' ]; then
runDjangoMigrate
fi
# Accept other commands
exec "$#"
Now, in the docker-compose file, I have the services like
version: '3.7'
services:
database:
image: mysql:5.7
container_name: 'qcg7_db_mysql'
restart: always
web:
build: .
command: ["./wait_for_it.sh", "database:3306", "--", "./docker_start.sh"]
volumes:
- ./src:/app
depends_on:
- database
environment:
DJANGO_MANAGE_MIGRATE: 'on'
But when I build the image using
docker-compose up --build
It fails to run the migration command from entrypoint script with error
(2002, "Can't connect to MySQL server on 'database' (115)")
This is due to the fact that the database server has not still started.
How can I make web service to wait untill the database service is completely started and is ready to accept connections?
Unfortunately, there is not a native way in Docker to wait for the database service to be ready before Django web app attempts to connect. Depends_on will only ensure that the web app is started after the database container is launched.
Because of this limitation you will need to solve this problem in how your container runs. The easiest solution is to modify the entrypoint.sh to sleep for 10-30 seconds so that your database has time to initialize before executing any additional commands. This official MySQL entrypoint.sh shows an example of how to block until the database is ready.

Docker-compose does not start containers

I'm new with docker-compose. I have a problem when I use the command "docker-compose up -d" to start a multi-container application what should start the containers with the status "up" but all the time a execute the command the status is "Exit", I'm not sure if I'm doing something wrong, this is my docker-compose.yml file
version: '3'
services: catalog:
image: ciscatalog
hostname: catalogHost
command: hostname
volumes:
- /home/docker:/opt/host
container:
image: dis/ciscontainer
hostname: containerHost
command: hostname
volumes:
- /home/docker:/opt/host
inbound:
image: dsi/cisinbound
hostname: inboundHost
depends_on:
- catalog
links:
- catalog
command: hostname
volumes:
- /home/docker:/opt/host
outbound:
image: dsi/cisoutbound
hostname: outboundHost
depends_on:
- catalog
links:
- catalog
command: hostname
volumes:
- /home/docker:/opt/host
example run:
root#docker1:/home/docker/DSI# docker-compose scale catalog=3 container=4 inbound=1 outbound=1
Creating and starting dsi_catalog_1 ... done
Creating and starting dsi_catalog_2 ... done
Creating and starting dsi_catalog_3 ... done
Creating and starting dsi_container_1 ... done
Creating and starting dsi_container_2 ... done
Creating and starting dsi_container_3 ... done
Creating and starting dsi_container_4 ... done
Creating and starting dsi_inbound_1 ... done
Creating and starting dsi_outbound_1 ... done
root#docker1:/home/docker/DSI# docker-compose up -d
Starting dsi_container_4
Starting dsi_catalog_3
Starting dsi_catalog_1
Starting dsi_container_3
Starting dsi_catalog_2
Starting dsi_container_1
Starting dsi_outbound_1
Starting dsi_inbound_1
Starting dsi_container_2
root#docker1:/home/docker/DSI# docker-compose ps
Name Command State Ports
-------------------------------------------
dsi_catalog_1 hostname Exit 0
dsi_catalog_2 hostname Exit 0
dsi_catalog_3 hostname Exit 0
dsi_container_1 hostname Exit 0
dsi_container_2 hostname Exit 0
dsi_container_3 hostname Exit 0
dsi_container_4 hostname Exit 0
dsi_inbound_1 hostname Exit 0
dsi_outbound_1 hostname Exit 0
Please, can anybody help me? docker-compose version 1.13.
I think I got it: you are overriding the command you give in the dockerfile because you have this line in each of the services
command: hostname
so the only command you give is "hostname", which is actually what is run.
If you run an image with docker, you are probably running a completely different command!
If this is a linux based image, 'hostname' will just print the hostname and then exit. So then the command is stopped which logically will result in a stopped container (exit 0)
Remove the command-override so the containers actually run their respective commands.

HEALTHCHECK of a Docker container running Celery tasks?

I know one of the ways to check health for Docker container is using the commmand
HEALTHCHECK CMD curl --fail http://localhost:3000/ || exit 1
But in case of workers there is no such URL to hit , How to check the container's health in that case ?
The celery inspect ping command comes in handy, as it does a whole trip: it sends a "ping" task on the broker, workers respond and celery fetches the responses.
Assuming your app is named tasks.add, you may ping all your workers:
/app $ celery inspect ping -A tasks.add
-> celery#aa7c21dd0e96: OK
pong
-> celery#57615db15d80: OK
pong
With aa7c21dd0e96 being the Docker hostname, and thus available in $HOSTNAME.
To ping a single node, you would have to run:
celery inspect ping -A tasks.add -d celery#$HOSTNAME
Here, d stands for destination.
The line to add to your Dockerfile:
HEALTHCHECK CMD celery inspect ping -A tasks.add -d celery#$HOSTNAME
Sample outputs:
/app $ celery inspect ping -A tasks.add -d fake_node
Error: No nodes replied within time constraint.
/app $ echo $?
69
Unhealthy if the node does not exist or does not reply
/app $ celery inspect ping -A tasks.add -d celery#$HOSTNAME
-> celery#d39b3d31cc13: OK
pong
/app $ echo $?
0
Healthy when the node replies pong.
/app $ celery inspect ping -d celery#$HOSTNAME
Traceback (most recent call last):
...
raise socket.error(last_err)
OSError: [Errno 111] Connection refused
/app $ echo $?
1
Unhealthy when the broker is not available - I removed the app, so it tries to connect to a local AMPQ and fails
This might not suit your needs, the broker is unhealthy, not the worker.
The below example snippet, derived from that posted by #PunKeel, is applicable for those looking to implement health check in docker-compose.yml which could be used through docker-compose or docker stack deploy.
worker:
build:
context: .
dockerfile: Dockerfile
image: myimage
links:
- rabbitmq
restart: always
command: celery worker --hostname=%h --broker=amqp://rabbitmq:5672
healthcheck:
test: celery -b amqp://rabbitmq:5672 inspect ping -d celery#$$HOSTNAME
interval: 30s
timeout: 10s
retries: 3
Notice the extra $ in the command, so that $HOSTNAME actually gets passed into the container. I also didn't use the -A flag.
Ideally, rabbitmq should also have its own health check, perhaps with curl guest:guest#localhost:15672/api/overview, since docker wouldn't be able to discern if worker is down or the broker is down with celery inspect ping.
For celery 5.2.3 I used celery -A [celery app name] status for the health check. This is how my docker-compose file looks like
worker:
build: .
healthcheck:
test: celery -A app.celery_app status
interval: 10s
timeout: 10s
retries: 10
volumes:
- ./app:/app
depends_on:
- broker
- redis
- database
Landed on this question looking for a health check for Celery workers as part of an Airflow setup (Airflow 2.3.4, Celery 5.2.7), which I eventually figured out. This is a very specific use case of the original question, but might still be useful for some:
# docker-compose.yml
worker:
image: ...
hostname: local-worker
entrypoint: airflow celery worker
...
healthcheck:
test: [ "CMD-SHELL", 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery#$${HOSTNAME}"' ]
interval: 5s
timeout: 10s
retries: 10
restart: always
...
I got inspiration from Airflow's quick-start Docker Compose.

Spring boot app fail to link consul in docker

I am trying to use Consul as discovery service, and another two spring boot app to register with Consul; and put them into docker;
following are my codes:
app:
server:
port: 3333
spring:
application:
name: adder
cloud:
consul:
host: consul
port: 8500
discovery:
preferIpAddress: true
healthCheckPath: /health
healthCheckInterval: 15s
instanceId: ${spring.application.name}:${spring.application.instance_id:${server.port}}
2 docker-compose.yml
consul1:
image: "progrium/consul:latest"
container_name: "consul1"
hostname: "consul1"
command: "-server -bootstrap -ui-dir /ui"
adder:
image: wsy/adder
ports:
- "3333:3333"
links:
- consul1
environment:
WAIT_FOR_HOSTS: consul1:8500
There is another similar question Cannot link Consul and Spring Boot app in Docker;
the answer suggests, the app should wait for consul to fully work by using depends_on, which I tried, but didn't work;
the error message is as following:
adder_1 | com.ecwid.consul.transport.TransportException: java.net.ConnectException: Connection refused
adder_1 | at com.ecwid.consul.transport.AbstractHttpTransport.executeRequest(AbstractHttpTransport.java:80) ~[consul-api-1.1.8.jar!/:na]
adder_1 | at com.ecwid.consul.transport.AbstractHttpTransport.makeGetRequest(AbstractHttpTransport.java:39) ~[consul-api-1.1.8.jar!/:na]
besides spring boot application.yml and docker-compose.yml, following is App's Dockerfile
FROM java:8
VOLUME /tmp
ADD adder-0.0.1-SNAPSHOT.jar app.jar
RUN bash -c 'touch /app.jar'
ADD start.sh start.sh
RUN bash -c 'chmod +x /start.sh'
EXPOSE 3333
ENTRYPOINT ["/start.sh", " java -Djava.security.egd=file:/dev/./urandom -jar /app.jar"]
and the start.sh
#!/bin/bash
set -e
wait_single_host() {
local host=$1
shift
local port=$1
shift
echo "waiting for TCP connection to $host:$port..."
while ! nc ${host} ${port} > /dev/null 2>&1 < /dev/null
do
echo "TCP connection [$host] not ready, will try again..."
sleep 1
done
echo "TCP connection ready. Executing command [$host] now..."
}
wait_all_hosts() {
if [ ! -z "$WAIT_FOR_HOSTS" ]; then
local separator=':'
for _HOST in $WAIT_FOR_HOSTS ; do
IFS="${separator}" read -ra _HOST_PARTS <<< "$_HOST"
wait_single_host "${_HOST_PARTS[0]}" "${_HOST_PARTS[1]}"
done
else
echo "IMPORTANT : Waiting for nothing because no $WAIT_FOR_HOSTS env var defined !!!"
fi
}
wait_all_hosts
exec $1
I can infer that your Consul configuration is located in your application.yml instead of bootstrap.yml, that's the problem.
According to this answer, bootstrap.yml is loaded before application.yml and Consul client has to check its configuration before the application itself and therefore look at the bootstrap.yml.
Example of a working bootstrap.yml :
spring:
cloud:
consul:
host: consul
port: 8500
discovery:
prefer-ip-address: true
Run Consul server and do not forget the name option to match with your configuration:
docker run -d -p 8500:8500 --name=consul progrium/consul -server -bootstrap
Consul server is now running, run your application image (builded previously with your artifact) and link it to the Consul container:
docker run -d -name=my-consul-client-app --link consul:consul acme/spring-app
Your problem is that depends_on does only control the startup order of your services. You have to wait until the consul servers are up and running before starting your spring app. You can do this with this script:
#!/bin/bash
set -e
default_host="database"
default_port="3306"
host="${2:-$default_host}"
port="${3:-$default_port}"
echo "waiting for TCP connection to $host:$port..."
while ! (echo >/dev/tcp/$host/$port) &>/dev/null
do
sleep 1
done
echo "TCP connection ready. Executing command [$1] now..."
exec $1
Usage in you docker file:
COPY wait.sh /wait.sh
RUN chmod +x /wait.sh
CMD ["/wait.sh", "java -jar yourApp-jar" , "consulURL" , "ConsulPort" ]
I just want to clarify that, at last I still don't have a solution, and can't understand the situation here; I tried the suggestion from Ohmen, in APP container, I am able to ping consul1; But the APP still fails to connect consul;
If I only start the consul by following command:
docker-compose -f docker-compose-discovery.yml up discovery
Then I can run the APP directly(through Main), and it is able to connect with spring.cloud.consul.host: discovery;
But if I try to run APP in docker container, like following:
docker run --name adder --link discovery-consul:discovery wsy/adder
It fails again with connection refused;
I am very new to docker & docker-compose; I thought it would be a good example to start, but it seems not that easy for me;

Resources