I have a health check defined for my ECS Fargate Service, it works when I test locally and works with Fargate v 1.3.0.
But when I change to Fargate Platform version 1.4.0 it always turns unhealthy. But the actual service is working. I can access the service on the containers public IP.
The health check is defined as:
"CMD-SHELL", "curl --fail http://localhost || exit 1"
So we looked into this and there's an issue in platform version 1.4 where, if the health check outputs anything to stderr a false negative occurs. We will, obviously, fix this but in the meantime you can work around this by (in this case) run curl in silent mode or simply redirect stderr output to /dev/null:
curl -s --fail http://localhost || exit 1
or
curl --fail http://localhost 2>/dev/null || exit 1
Should unblock you for now.
I wanted to collate some answers together and build on them, as follows.
I'm not being funny, but first and foremost make sure you have a healthcheck endpoint running somewhere. Note that this doesn't have to be inside your container! Let me show you what I mean:
curl -s --fail -I https://127.0.0.1:8000/ || exit 1
will only pass if you have a HTTP server running on localhost port 8000 (etc.). This can be anything that returns a 200 - over to you.
Tips:
Make sure curl is installed inside the container
-s is for silent
--fail - ask google
-I header only
If localhost doesn't work try 127.0.0.1
Now, in my case I was not running a HTTP server but rather a long-running python script. In its error state the script exits with 1 (which terminates the task), but otherwise (after a long time) it exits with 0. To fail the healthcheck, the healthcheck call must also return 0 (otherwise there is a 1 and the task is again terminated*). [*exit codes > 1 can be converted to a 1 - see below stolen trick.]
So I had to fake a different endpoint with the same behaviour.
Step forward, Google.
curl -s --fail -I https://www.google.com || exit 1
As before, but now hit an external endpoint kindly provided. Note the || exit 1 which converts any positive-definite integer exit code to the 1 liked by the healthcheck.
Sorry to "state the bleeding obvious", but you really do need a function running here - don't run curl on a local endpoint and expect to get a healthy status!
Remember to expose the https / http ports 443 / 80 in your docker file and in the JSON task definition spec/through the console UI.
TIP! Note that the CMD-SHELL syntax is slightly different depending.
Putting it all together, for ECS Fargate the rest is correct.
You could also try an echo rather than a curl. I am unclear whether a point-to-point call is even required.
Related
My goal is to have a HEALTHCHECK command in my Dockerfile to check if the webserver is working alright by simply making a request to the website and check if it receives a "proper response".
The problem I'm having is that the application has an authentication middleware, which causes the application to return an error (401 Unauthorized), causing CURL to fail and return curl: (7) Failed to connect to host.docker.internal port 8000: Connection refused.
If I remove the authentication middleware it doesn't return anything, which is what I'm aiming for.
The command I'm using is the following (I'm currently just using it inside a container, trying to find a solution):
curl --fail http://host.docker.internal:8000
I know I can tell CURL the username and password but that's something I would rather not do it.
Having a way to tell CURL that Unauthorized (error 401) is fine or to consider a connection refused error (curl: (7)) as the only error would be fine but it would be even better if I could decide what should CURL consider and/or not consider a success. Is there any way to do something like this with one or more CURL options?
Health check is a good practice when microservices or rest services architecture are used.
Default health endpoints and check platforms needs 200 as http code to flag your app as healthy. Any other response is flagged ad unhealthy.
Custom codes with curl
I tried and I can say: with curl is not possible:
https://superuser.com/questions/590099/can-i-make-curl-fail-with-an-exitcode-different-than-0-if-the-http-status-code-i
You need a custom logic.
Custom health
As you are using ubuntu based image, you could use a simple bash function to catch 401 codes and return exit 0 in order to mark as healthy your container.
with curl
The cornerstone here is the option to retrieve just the response code from curl invokation:
curl -o /dev/null -s -w "%{http_code}\n" http://localhost
So you can create a bash script to execute your curl invocation and return:
exit 0 just for 200 & 401
And exit 1 in any other case .
#healthcheck.sh
code=$(curl -o /dev/null -s -w "%{http_code}\n" http://localhost:12345)
echo "response code:$code"
if [ "$code" == "200" ] || [ "$code" == "401" ]
then
echo "success"
exit 0;
else
echo "error"
exit 1;
fi
Finally you can use this script in your healthcheck in any language(php in your case):
FROM node
COPY server.js /
HEALTHCHECK --interval=5s --timeout=10s --retries=3 CMD curl -sS 127.0.0.1:8080 || exit 1
CMD [ "node", "/server.js" ]
Health feature should be public
Common health verification is related to: server status, internet connection, ram, disk, database connectivity, and any other stat that indicates you if you app is running and is ok.
Health check platforms does not allow us to register complex security flows (oauth1, ouath2, openid, etc). Just allow us to register a simple http endpoint. Here an example from aws ELB check configuration
Health feature should not expose any other sensitive data, because of that, this endpoint could be public. Classic and public webpages, web systems or public apis are examples.
Workaround
In some strict cases, privacy is required.
In this case I protected the /health with a simple apikey value as query parameter. In the controller, I validate if it is equal to some value. Final health endpoint will be /health?apiKey=secret and this is easy to register in check platforms.
Using complex configurations you could allow /health just for internal private lan, not for public access. So in this case your /health is secure
I have 5 containers running one after another. First 3, (ABC) are very minimal. ABC containers need to be health checked, but curl,wget cannot be run on them, so currently I just run test:[CMD-SHELL], "whoami || exit 1" in docker-compose.yml. Which seems to bring them to a healthy state. Other 2 (DE) dependent on ABC to be healthy are being checked using test: [CMD-SHELL] , "curl --fail http://localhost" command. My question is how can I properly check health of those minimal containers, without using curl, wget etc. ?
If you can live with a TCP connection test to your internal service's port, you could use /dev/tcp for this:
HEALTHCHECK CMD bash -c 'echo -n > /dev/tcp/127.0.0.1/<port>'
Like this:
# PASS (webserver is serving on 8099)
root#ab7470ea0c8b:/app# echo -n > /dev/tcp/127.0.0.1/8099
root#ab7470ea0c8b:/app# echo $?
0
# FAIL (webserver is NOT serving on 9000)
root#ab7470ea0c8b:/app# echo -n > /dev/tcp/127.0.0.1/9000
bash: connect: Connection refused
bash: /dev/tcp/127.0.0.1/9000: Connection refused
root#ab7470ea0c8b:/app# echo $?
1
Unfortunately, I think this is the best that can be done without installing curl or wget.
What does the "(healthy)" string in STATUS column stands for?
user#user:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
X X X X Up 20 hours X X
X X X X Up 21 hours (healthy) X X
That's the result of the HEALTHCHECK instruction. That instruciton runs a command inside the container every 30 seconds. If the command succeeds, the container is marked healthy. If it fails too many times, it's marked unhealthy.
You can set the interval, timeout, number of retries and start delay.
The following, for example, will check that your container responds to HTTP every 5 minutes with a timeout of 3 seconds.
HEALTHCHECK --interval=5m --timeout=3s \
CMD curl -f http://localhost/ || exit 1
You get a health_status event when the health status changes. You can follow those and others with docker events.
https://ryaneschinger.com/blog/using-docker-native-health-checks/
Normally it's something you launch with, to enable swarm or other services to check on the health of the container.
IE:
$ docker run --rm -it \
--name=elasticsearch \
--health-cmd="curl --silent --fail localhost:9200/_cluster/health || exit 1" \
--health-interval=5s \
--health-retries=12 \
--health-timeout=2s \
elasticsearch
see the health checks enabled at runtime?
Means they are using the command: healthcheck
https://docs.docker.com/engine/reference/builder/#healthcheck
When a container has a healthcheck specified, it has a health status in addition to its normal status. This status is initially starting. Whenever a health check passes, it becomes healthy (whatever state it was previously in). After a certain number of consecutive failures, it becomes unhealthy.
**starting** – Initial status when the container is still starting
**healthy** – If the command succeeds then the container is healthy
**unhealthy** – If a single run of the takes longer than the specified
timeout then it is considered unhealthy. If a health check fails then the
will run retries number of times and will be declared unhealthy
if the still fails.
Reference
I have an issue using Docker swarm.
I have 3 replicas of a Python web service running on Gunicorn.
The issue is that when I restart the swarm service after a software update, an old running service is killed, then a new one is created and started. But in the short period of time when the old service is already killed, and the new one didn't fully start yet, network messages are already routed to the new instance that isn't ready yet, resulting in 502 bad gateway errors (I proxy to the service from nginx).
I use --update-parallelism 1 --update-delay 10s options, but this doesn't eliminate the issue, only slightly reduces chances of getting the 502 error (because there are always at least 2 services running, even if one of them might be still starting up).
So, following what I've proposed in comments:
Use the HEALTHCHECK feature of Dockerfile: Docs. Something like:
HEALTHCHECK --interval=5m --timeout=3s \
CMD curl -f http://localhost/ || exit 1
Knowing that Docker Swarm does honor this healthcheck during service updates, it's relative easy to have a zero downtime deployment.
But as you mentioned, you have a high-resource consumer health-check, and you need larger healthcheck-intervals.
In that case, I recomend you to customize your healthcheck doing the first run immediately and the successive checks at current_minute % 5 == 0, but the healthcheck itself running /30s:
HEALTHCHECK --interval=30s --timeout=3s \
CMD /service_healthcheck.sh
healthcheck.sh
#!/bin/bash
CURRENT_MINUTE=$(date +%M)
INTERVAL_MINUTE=5
[ $((a%2)) -eq 0 ]
do_healthcheck() {
curl -f http://localhost/ || exit 1
}
if [ ! -f /tmp/healthcheck.first.run ]; then
do_healhcheck
touch /tmp/healthcheck.first.run
exit 0
fi
# Run only each minute that is multiple of $INTERVAL_MINUTE
[ $(($CURRENT_MINUTE%$INTERVAL_MINUTE)) -eq 0 ] && do_healhcheck
exit 0
Remember to COPY the healthcheck.sh to /healthcheck.sh (and chmod +x)
There are some known issues (e.g. moby/moby #30321) with rolling upgrades in docker swarm with the current 17.05 and earlier releases (and doesn't look like all the fixes will make 17.06). These issues will result in connection errors during a rolling upgrade like you're seeing.
If you have a true zero downtime deployment requirement and can't solve this with a client side retry, then I'd recommend putting in some kind of blue/green switch in front of your swarm and do the rolling upgrade to the non-active set of containers until docker finds solutions to all of the scenarios.
In my CI chain I execute end-to-end tests after a "docker-compose up". Unfortunately my tests often fail because even if the containers are properly started, the programs contained in my containers are not.
Is there an elegant way to verify that my setup is completely started before running my tests ?
You could poll the required services to confirm they are responding before running the tests.
curl has inbuilt retry logic or it's fairly trivial to build retry logic around some other type of service test.
#!/bin/bash
await(){
local url=${1}
local seconds=${2:-30}
curl --max-time 5 --retry 60 --retry-delay 1 \
--retry-max-time ${seconds} "${url}" \
|| exit 1
}
docker-compose up -d
await http://container_ms1:3000
await http://container_ms2:3000
run-ze-tests
The alternate to polling is an event based system.
If all your services push notifications to an external service, scaeda gave the example of a log file or you could use something like Amazon SNS. Your services emit a "started" event. Then you can subscribe to those events and run whatever you need once everything has started.
Docker 1.12 did add the HEALTHCHECK build command. Maybe this is available via Docker Events?
If you have control over the docker engine in your CI setup you could execute docker logs [Container_Name] and read out the last line which could be emitted by your application.
RESULT=$(docker logs [Container_Name] 2>&1 | grep [Search_String])
logs output example:
Agent pid 13
Enter passphrase (empty for no passphrase): Enter same passphrase again: Identity added: id_rsa (id_rsa)
#host SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6
#host SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6
parse specific line:
RESULT=$(docker logs ssh_jenkins_test 2>&1 | grep Enter)
result:
Enter passphrase (empty for no passphrase): Enter same passphrase again: Identity added: id_rsa (id_rsa)