How to check health of docker image running sidekiq - ruby-on-rails

Question

I'm using kubernetes on my cluster with several rails / node docker images. Most of them have :3000/healtz health check that simply returns status 200 with OK in body.
Now I'm trying to discover the best way how this health check can be performed on docker image running sidekiq. How I can verify that the worker is running?

Answer 1

If your image is unix like, you can check if the proccess is running with
$ ps aux | grep '[s]idekiq'
But this don't guarantee that everything is working inside sidekiq and redis.
A better approach is described/developed in this sidekiq plugin https://github.com/arturictus/sidekiq_alive
I'm facing problems with livenessProbe for k8s and trying to solve without using this lib but not successful yet.

Answer 2

Sidekiq 6.0 ships with a new sidekiqmon, which you could use to validate that a process is running on your current machine with redis:
REDIS_URL=redis://redis.example.com:6380/5 sidekiqmon | grep $$(hostname)
Documentation: https://github.com/mperham/sidekiq/wiki/Monitoring#sidekiqmon

Answer 3

You have different approaches depending mostly on how you deployed Sidekiq inside the Kubernetes pod.
Here some of them.
SIDEKIQMON
Since Sidekiq v6 we have sidekiqmon
You can try with something like:
bundle exec sidekiqmon processes | grep $(hostname)
To make it work you must be able to grep the correct running process related to current pod/container.
PS AUX (I used in the past)
Something like this could work:
ps aux | grep '[s]idekiq 6'
Note the 6 (sidekiq version) at the end to differentiate from the other process that you could have.
Anyway here the idea is to just check your processes with ps aux and try to "grep" the sidekiq one you need to monitor.
SIDEKIQ_ALIVE
Another approach is to use sidekiq_alive
But it requires more effort to be configured and managed and I never really tried it. This is the only one that really check the full functionality with also the Redis status.
SYSTEMCTL
If you deployed sidekiq using systemd you can just do:
systemctl status sidekiq | grep '[r]unning'"
So at the end my YAML manifest for Sidekiq deployment looks like this:
livenessProbe:
exec:
command: ["/bin/bash", "-l", "-c", "bundle exec sidekiqmon processes | grep $(hostname)"]
initialDelaySeconds: 120
periodSeconds: 30
successThreshold: 1
failureThreshold: 3
timeoutSeconds: 30
Note that only the sidekiq_alive approach is going to perform a full end to end test, checking also the redis status. Maybe this is too much, you could have other monitoring system for Redis so maybe here, with the kubernetes LivenessProbes, you don't need to check the whole flow but just check if the sidekiq process is still alive or crashed due to a memory leak. This is up to you, based on your needs.
"Why you need this kind of LivenessProbes?"
Because if you are launching Sidekiq via systemd or similar, you will have the main process (PID 1) as /usr/bin/python3 /usr/bin/systemctl start sidekiq and the related process launched with ExecStart as sidekiq 6.1.3 your_app_name [0 of 5 busy]. If you have a memory leak, the last one will crash and not the PID 1.
For this reason you need a sort of livenessProbes for Sidekiq.
Instead if you are not using systemd and your PID 1 is sidekiq 6.1.3 your_app_name [0 of 5 busy] because you are launching it with bundle exec sidekiq you don't need it.