I am making graceful shutdown feature using go lang when kebernetes doing rolling-update on google container engine. Does anyone know what process signal is sent to the running pods when kubectl rolling-update starts?
I've listened to os.Kill, os.Interrupt, syscall.SIGTERM, syscall.SIGKILL, syscall.SIGSTOP signals to be handled, none of those signals was raised while kubectl rolling-update.
I would really appreciate for your answers.
I got a solution! I used shell script file as an ENTRYPOINT and executed go binary in that script file. So process ID of executed go binary was not 1.(shell script process's ID was 1 instead) And docker sent SIGTERM only to PID 1(which is not propagated to it's child processes). So, I had to change my ENTRYPOINT direct to executing go binary, and I got SIGTERM in my go code now. Refer to this link
Related
I'm running Kubernetes service using exec which have few pods in statefulset.
If I kill one of the master pod used by service in exec, it exits with code 137. I want to forward it to another pod immediately after killing or apply wait before exiting. I need help. Waiting for answer. Thank you.
137 means your process exited due to SIGKILL, usually because the system ran out of RAM. Unfortunately no delay is possible with SIGKILL, the kernel just drops your process and that is that. Kubernetes does detect it rapidly and if you're using a Service-based network path it will usually react in 1-2 seconds. I would recommend looking into why your process is being hard-killed and fixing that :)
I have a docker container that runs a custom php file or say a unix shell script.
If the script executes fine the docker container should continue to RUN however, if the script fails due to error or due to a custom check; then I wish to terminate (stop) i.e change the status of that docker container to "Exited".
Sample case 1: The unix shell script periodically checks for a particular file or data on a file-system / URL. If that data / file is not found I would like the docker container to shutdown (Exit status) else it should continue to run.
Sample case 2: The script runs and checks for stuck thread count for a different process. If the stuck thread count is more than 5 I would like the docker process to shutdown (Exit status) else it should continue to run.
I know how to shutdown a container from outside however, in this case I wish to trigger container shutdown from within the container depending upon the custom script's failure condition being met.
Can you please suggest ?
Every Docker container has some main process, whatever was launched as the ENTRYPOINT or CMD. That process has pid 1, with the rights and responsibilities that entails. The lifetime of the container is exactly the length of that main process: the only way to cause the container to exit is to cause pid 1 to exit. Since pid 1 is special, it may not work to kill 1.
If I was going to implement this, I'd write a program (probably in C) that could both execute the health checks and run the main process. If the process exited normally, the supervisor would wait(2) for it and then exit itself, causing the container to exit. If a health check failed, the supervisor would kill(2) its child, wait(2) for it, and then exit itself.
I'm not immediately aware of a prebuilt implementation of this concept. It is not dissimilar from what supervisord does, except that supervisor expects to run as an init process that never exits.
Another possibility is to implement the health checks within your application itself. Then you're just running the one process, and if a health check fails, it can kill itself (exit(3), for example). Higher-level orchestrators like Kubernetes also have a health check concept that can be tied to a network request or a command that runs inside a container (for Kubernetes, see Container probes).
I'm using jenkins to do a few actions in a remote server.
I have an Execute Shell command in which I do the following:
sudo ssh <remote server> 'sudo service supervisor restart'
sleep 30
When jenkins reaches the first line I can see 'Restarting Supervisor' but after a moment I see that jenkins closed the ssh connection and moved on to the second line.
I tried adding a 'sleep 30' after the restart command but it still doesn't work.
Seems jenkins doesn't wait for the supervisor restart command to be completed.
Problem is it's not something that always happens, just sometimes, but it does make a lot of problems when it fails.
I think you can never be certain all processes started by supervisord are in a 'ready' state after a restart. Even is the restart action would wait for processes to be started, it wouldn't know if they are 'ready'.
In docker-compose setups that need to know if a certain service is available I've used an extra 'really ready' check for this - optionally in a loop with a sleep/wait. If the process that you are starting opens a port you can use one of the variations of 'wait-for' for this.
If there is a simple run command specified on the command line, or with CMD, the container stops when the program exits. But, what if:
the program spawns new processes, ant then exits?
'exec' is used on the command line, then the first command exits?
Can you please also point to the docs?
Thanks!
The process you run when you exec docker run will be the process with PID 1 (inside the process namespace of the container). This process is special in UNIX / Linux systems and it's the process in charge of 'adopting' any 'orphaned' process. If this process ends, all the processes will end also.
So, answering your questions, if this initial process (the one executed in docker run) ends, all the processes inside your container will also end . I have not found any official documentation related to this, but there is a great post from phusion discussing about this topic.
After completing a Jenkins task, I execute a Linux shell script by using Jenkins' post-condition configuration section.
This Linux shell script wants to launch a standby service on the backend and can NOT cause Jenkins to pause.
I tried to use "nohup+&", etc., but it does not work.
Is there a good way to do it?
Jenkins is probably waiting for some pipes to close. Your background process has inherited some file descriptors and is keeping them open for as long as it runs.
If you are lucky, the only file descriptors are 0, 1 and 2 (the standard ones.) You might want to check the file descriptors of the background process using lsof -p PID where PID is the process id of the background process.
You should make sure all of those file descriptors (both inputs and outputs) are redirected for the background process, so start it with something like:
nohup daemon </dev/null >/dev/null 2>&1 &
Feel free to direct the output to a file other than /dev/null but make sure you keep the order of the redirections. The order is important.
If you plan to start background processes from a Jenkins job, be advised that Jenkins will kill background processes when build ends. See https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller on how to prevent that.
I had a similar problem with running a shell script from Jenkins as a background process. I fixed it by using the below command:
BUILD_ID=dontKillMe nohup ./start-fitnesse.sh &