Background: My Docker container has a very long startup time, and it is hard to predict when it is done. And when the health check kicks in, it first may show 'unhealthy' since the startup is sometimes not finished. This may cause a restart or container removal from our automation tools.
My specific question is if I can control my Docker container so that it shows 'starting' until the setup is ready and that the health check can somehow be started immediately after that? Or is there any other recommendation on how to handle states in a good way using health checks?
Side question: I would love to get a reference to how transitions are made and determined during container startup and health check initiating. I have tried googling how to determine Docker (container) states but I can't find any good reference.
My specific question is if I can control my container so that it shows
'starting' until the setup is ready and that the health check can
somehow be started immediately after that?
I don't think that it is possible with just K8s or Docker.
Containers are not designed to communicate with Docker Daemon or Kubernetes to tell that its internal setup is done.
If the application takes a time to setup you could play with readiness and liveness probe options of Kubernetes.
You may indeed configure readynessProbe to perform the initial check after a specific delay.
For example to specify 120 seconds as initial delay :
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 120
Same thing for livenessProbe:
livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: Custom-Header
value: Awesome
initialDelaySeconds: 120
periodSeconds: 3
For Docker "alone" while not so much configurable you could make it to work with the --health-start-period parameter of the docker run sub command :
--health-start-period : Start period for the container to initialize
before starting health-retries countdown
For example you could specify an important value such as :
docker run --health-start-period=120s ...
Here is my work around. First in docker-compose set long timeout, start_period+timeout should be grater than max expected starting time, eg.
healthcheck:
test: ["CMD", "python3", "appstatus.py", '500']
interval: 60s
timeout: 900s
retries: 2
start_period: 30s
and then run script which can wait (if needed) before return results. In example above it is appstatus.py. In the script is something like:
timeout = int(sys.argv[1])
t0 = time.time()
while True:
time.sleep(2)
if isReady():
sys.exit(os.EX_OK)
t = time.time() - t0
if t > timeout:
sys.exit(os.EX_SOFTWARE)
Related
I cannot get past 1200 RPS no matter if I use 4 or 5 workers.
I tried to start locust in 3 variations -- one, four, and five worker processes (docker-compose up --scale worker_locust=num_of_workers). I use 3000 clients with a hatch rate of 100. The service that I am loading is a dummy that just always returns yo and HTTP 200, i.e., it's not doing anything, but returning a constant string. When I have one worker I get up to 600 RPS (and start to see some HTTP errors), when I have 4 workers I can get up to the ~1200 RPS (without a single HTTP error):
When I have 5 workers I get the same ~1200 RPS, but with a lower CPU usage:
I suppose that if the CPU went down in the 5-worker case (with respect to 4-worker case), than it's not the CPU that is bounding the RPS.
I am running this on a 6-core MacBook.
The locustfile.py I use posts essentially almost empty requests (just a few parameters):
from locust import HttpUser, task, between, constant
class QuickstartUser(HttpUser):
wait_time = constant(1) # seconds
#task
def add_empty_model(self):
self.client.post(
"/models",
json={
"grouping": {
"grouping": "a/b"
},
"container_image": "myrepo.com",
"container_tag": "0.3.0",
"prediction_type": "prediction_type",
"model_state_base64": "bXkgc3RhdGU=",
"model_config": {},
"meta": {}
}
)
My docker-compose.yml:
services:
myservice:
build:
context: ../
ports:
- "8000:8000"
master_locust:
image: locustio/locust
ports:
- "8089:8089"
volumes:
- ./:/mnt/locust
command: -f /mnt/locust/locustfile.py --master
worker_locust:
image: locustio/locust
volumes:
- ./:/mnt/locust
command: -f /mnt/locust/locustfile.py --worker --master-host master_locust
Can someone suggest the direction of getting towards the 2000 RPS?
You should check out the FAQ.
https://github.com/locustio/locust/wiki/FAQ#increase-my-request-raterps
It's probably your server not being able to handle more requests, at least from your one machine. There are other things you can do to make more sure that's the case. You can try FastHttpUser, running on multiple machines, or just upping the number of users. But if you can, check to see how the server is handling the load and see what you can optimize there.
You will need more workers to generate more RPS. I thought one worker will have limited local port range when creating tcp connection to the destination.
You may check this value in your linux worker:
net.ipv4.ip_local_port_range
Try to tweak that number it on your each linux worker, or simply create hundreds of new worker with another powerful machine (your 6-core cpu macbook is to small)
To create many workers you could try Locust in kubernetes with horizontal pod autoscaling for the workers deployment.
Here is some helm chart to start play arround with Locust k8s deployment:
https://github.com/deliveryhero/helm-charts/tree/master/stable/locust
You may need to check these args for it:
worker.hpa.enabled
worker.hpa.maxReplicas
worker.hpa.minReplicas
worker.hpa.targetCPUUtilizationPercentage
simply set the maxReplicas value to get more workers when the load testing is started. Or you can scale it manually with kubectl command to scale worker pods to your desired number.
I've done to generate minimal 8K rps (stable value for my app, it can't serve better) with 1000 pods/worker, with Locust load test parameter like 200K users with 2000 spawn per second.
You may have to scale out your server when you reach higher throughput, but with 1000 pods/worker i thought you can easily reach 15K-20K rps.
I have a customized rabbitmq image that I am using with docker-compose (3.7) to launch a docker cluster. This is necessary because of some peculiar issues when trying to deploy a cluster in docker swarm. The image has a shell script which runs on the primary and secondary nodes and makes the modifications needed to run a cluster. This involves stopping rabbitmq and running rabbitmqctl commands to create the cluster between the two nodes. This configuiration works flawlessly until I try to add in a healthcheck. I have tried adding it in to the image and adding it into the compose file. Both cause the image to crash and constantly restart. I have the following shell script which gets copied into the image:
#!/bin/bash
set -eo pipefail
# A RabbitMQ node is considered healthy if all the below are true:
# * the rabbit app finished booting & it's running
# * there are no alarms
# * there is at least 1 active listener
rabbitmqctl eval '
{ true, rabbit_app_booted_and_running } = { rabbit:is_booted(node()), rabbit_app_booted_and_running },
{ [], no_alarms } = { rabbit:alarms(), no_alarms },
[] /= rabbit_networking:active_listeners(),
rabbitmq_node_is_healthy.
' || exit 1
On an already running image this works and produces the correct result.
I tried the flowing in the compose file:
healthcheck:
interval: 60s
timeout: 60s
retries: 10
start_period: 600s
test: ["CMD", "docker-healthcheck"]
It seems that the start_period is completely ignored. I can see the health status with an error right away. I have also tried the following native rabbitmq diagnostics command:
rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms
This oddly fails with an "unable to find rabbitmq-diagnostics" error, despite the fact the program is definitely in the path. I can execute the command successfully in an already running container.
If I create the container without the healthcheck and then add it in after the fact from the command line with:
docker service update --health-cmd docker-healthcheck --health-interval 60s --health-timeout 60s --health-retries 10 [container id]
it marks the container healthy. So it works but just not in a start up configuration. It seems like to me that the healthcheck should not begin until 10 minutes have passed. It doesn't seem to matter how long I wait for everything to startup using the start_period parameter it still causes the container to fail.
Is this a bug or is there something mysterious about the way start_period works?
Anyone else every have this problem?
When I scale a service up from 1 node (Node A) to 2 nodes (Node A and Node B), I see traffic immediately being routed to both nodes (including the new Node B even though it isn't ready).
As a result, an Nginx proxy will return 502s half the time (until Node B is ready).
Any suggestions how you can delay this traffic?
Note: this isn't waiting for another container to come up as mentioned here: Docker Compose wait for container X before starting Y
This is about delaying the network connection until the container is ready.
If you do not configure a healthcheck section, docker will assume that the container is available as soon as it is started.
Note that the initial healthcheck is only done after the set interval.
So you could add something extremely basic like testing if port 80 is connectable (you need nc in your docker image):
healthcheck:
test: nc -w 1 127.0.0.1 80 < /dev/null
interval: 30s
timeout: 10s
retries: 3
start_period: 5s
The "azds up" command times out before all the steps are done. I have a large Angular app that typically takes 5 minutes+ when npm install is executed. When I execute azds up this is what I get:
Step 1/9 : FROM node
Step 2/9 : ENV PORT 80
Step 3/9 : WORKDIR /app
Step 4/9 : COPY package*.json ./
Step 5/9 : RUN npm install --silent
Waiting for container...
and then it returns to the command line.
Is there a configuration in the azds.yaml where I can tell azds/helm to wait for a longer period of time?
Thanks!
To begin with i will give some examples that might be helpful in your case:
An example of .yaml of how to rolling upgrade
spec:
minReadySeconds: 60
progressDeadlineSeconds: 600
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 50%
maxUnavailable: 50%
minReadySeconds: The bootup time of your application, Kubernetes waits specific time until the next pod creation. For
example minReadySeconds is 60 then after the Pod become healthy the
Deployment must wait for 60 seconds to update the next Pod.
progressDeadlineSeconds Timeout value for updating one pod in this example 10 minutes. If the rollout fails to progress in 10
minutes, then the Deployment is marked as failed and finished, also
all of next job never invoke.
maxSurge: The maxSurge parameter controls how many extra resources can be created during the rollout.(Absolute value or %)
maxUnavailable: The maxUnavailable parameter sets the maximum number of Pods that can be unavailable during a rolling
update.(Absolute value or %)
An example of .yaml Liveness & Readiness
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 1
periodSeconds: 10
failureThreshold: 3
The above manifest configured livenessProbes which confirm whether an container is running properly or not.
It probe the health check by using HTTP GET request to /healthz with port 8080.
The probe sets an initialDelaySeconds=60 which means that it will not
be called until 60 seconds after all the containers in the Pod are
created. And timeoutSeconds=1 was configured it means that the probe
must respond with in the 1 second timeout. The periodSeconds=10 was
configured, it means that the probe invoke every 10 seconds. If more
than 3 probe failed(failureThreshold=3), the container will be
considerd un-healthy and failed and restart.
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 120
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 3
The above readinessProbe is more important than liveness probe in
production environment. the readinessProbe is confirm whether the
service can acceptable or not. If this probe failed, the internal
loadbalancer never send the traffic to this pod. Only successed the
probe, the traffic to this pod will start.
I have the following liveness probe in my service deployment.yaml
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 9081
scheme: HTTP
initialDelaySeconds: 180
timeoutSeconds: 10
periodSeconds: 10
successThreshold: 1
I want to test that the probe is actually triggering a POD redeployment, which is the easiest thing to do to make it fail?
Possibly in a programmatic way.
Update:
Better to clarify the question, I don't want to change the code in the application, neither pausing the container that is running.
I was wondering if it's possible to block someway the endpoint/port at runtime maybe using a kubernetes or docker command.
You could define your liveness probe as follows
livenessProbe:
exec:
command:
- /bin/bash
- '-c'
- /liveness-probe.sh
initialDelaySeconds: 10
periodSeconds: 60
And create an sh file in your root path named
liveness-probe.sh
that contains
#!/bin/bash
#exit 0 #Does not fail and does not trigger a pod restart
exit 1 #Triggers pod restart
If you have the ability to change the underlying applications code, simply change the /health endpoint to make it return something higher than a 400 http status code.
If not, you'll have to make your application fail somehow, probably by logging into the pod using kubectl exec and making changes that affect the application's health.
This is entirely dependent on your application, and kubernetes will simply do what you tell it to.
If you can get to the host where the pod is running, doing a docker pause on the container will pause all the processes in the container, which should fail the liveness probes.
Note: I have not tried this myself but based on the documentation of docker pause here, it sounds like that.