jenkins kills ssh session when supervisord restarts - jenkins

I'm using jenkins to do a few actions in a remote server.
I have an Execute Shell command in which I do the following:
sudo ssh <remote server> 'sudo service supervisor restart'
sleep 30
When jenkins reaches the first line I can see 'Restarting Supervisor' but after a moment I see that jenkins closed the ssh connection and moved on to the second line.
I tried adding a 'sleep 30' after the restart command but it still doesn't work.
Seems jenkins doesn't wait for the supervisor restart command to be completed.
Problem is it's not something that always happens, just sometimes, but it does make a lot of problems when it fails.

I think you can never be certain all processes started by supervisord are in a 'ready' state after a restart. Even is the restart action would wait for processes to be started, it wouldn't know if they are 'ready'.
In docker-compose setups that need to know if a certain service is available I've used an extra 'really ready' check for this - optionally in a loop with a sleep/wait. If the process that you are starting opens a port you can use one of the variations of 'wait-for' for this.

Related

Is there a way to have a docker container wait for an external call?

I have looked for a bit on Stack Overflow for a way to have a container start up and wait for an external connection but have not seen anything.
Here is what my process looks like currently:
Non-Docker external process reaches out at X interval and tells system to run a command.
Command runs.
System should remain idle until the next interval.
Now I have seen a few options with --wait or sleep but I would think that would not allow the container to receive the connection.
I also looked at the wait for container script that is often recommended but in this case I need the container to wait for a script to call it on non defined intervals.
I have tried having this just run the help command for my process but it then fails the container after a bit of time and makes it a mess for finding anything.
Additionally I have tried to have the container start with no command just to run the base OS and wait for the call but that did not work either.
I was looking at this wrong.
Ended up just running like any other webserver and database server.

Change docker status to "Exited" i.e shutdown container upon internal process check

I have a docker container that runs a custom php file or say a unix shell script.
If the script executes fine the docker container should continue to RUN however, if the script fails due to error or due to a custom check; then I wish to terminate (stop) i.e change the status of that docker container to "Exited".
Sample case 1: The unix shell script periodically checks for a particular file or data on a file-system / URL. If that data / file is not found I would like the docker container to shutdown (Exit status) else it should continue to run.
Sample case 2: The script runs and checks for stuck thread count for a different process. If the stuck thread count is more than 5 I would like the docker process to shutdown (Exit status) else it should continue to run.
I know how to shutdown a container from outside however, in this case I wish to trigger container shutdown from within the container depending upon the custom script's failure condition being met.
Can you please suggest ?
Every Docker container has some main process, whatever was launched as the ENTRYPOINT or CMD. That process has pid 1, with the rights and responsibilities that entails. The lifetime of the container is exactly the length of that main process: the only way to cause the container to exit is to cause pid 1 to exit. Since pid 1 is special, it may not work to kill 1.
If I was going to implement this, I'd write a program (probably in C) that could both execute the health checks and run the main process. If the process exited normally, the supervisor would wait(2) for it and then exit itself, causing the container to exit. If a health check failed, the supervisor would kill(2) its child, wait(2) for it, and then exit itself.
I'm not immediately aware of a prebuilt implementation of this concept. It is not dissimilar from what supervisord does, except that supervisor expects to run as an init process that never exits.
Another possibility is to implement the health checks within your application itself. Then you're just running the one process, and if a health check fails, it can kill itself (exit(3), for example). Higher-level orchestrators like Kubernetes also have a health check concept that can be tied to a network request or a command that runs inside a container (for Kubernetes, see Container probes).

How can I call a shell script to start a backend Java process?

After completing a Jenkins task, I execute a Linux shell script by using Jenkins' post-condition configuration section.
This Linux shell script wants to launch a standby service on the backend and can NOT cause Jenkins to pause.
I tried to use "nohup+&", etc., but it does not work.
Is there a good way to do it?
Jenkins is probably waiting for some pipes to close. Your background process has inherited some file descriptors and is keeping them open for as long as it runs.
If you are lucky, the only file descriptors are 0, 1 and 2 (the standard ones.) You might want to check the file descriptors of the background process using lsof -p PID where PID is the process id of the background process.
You should make sure all of those file descriptors (both inputs and outputs) are redirected for the background process, so start it with something like:
nohup daemon </dev/null >/dev/null 2>&1 &
Feel free to direct the output to a file other than /dev/null but make sure you keep the order of the redirections. The order is important.
If you plan to start background processes from a Jenkins job, be advised that Jenkins will kill background processes when build ends. See https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller on how to prevent that.
I had a similar problem with running a shell script from Jenkins as a background process. I fixed it by using the below command:
BUILD_ID=dontKillMe nohup ./start-fitnesse.sh &

stop a database container in docker automatically

I'm setup a database container with a script rc.db which provide standard init commands like:
/etc/rc.db start
/etc/rc.db stop
/etc/rc.db status
In Is it possible to install a complex server inside a Docker container?, I know I could use a simple script to start the db container(for example name as /etc/db_run.sh:
#/bin/sh
/etc/rc.db start
wait
And the Dockerfile
...
RUN /etc/db_run.sh
Because close database correctly is important. I wish when the container be stopped, it could call the /etc/rc.db stop.
When Docker tries to stop a container, it sends a SIGTERM signal, followed by a SIGKILL after a grace period. Just catch this signal and either call your script or pass it onto the DB process, whichever is appropriate.
I suspect that if you make the DB the main process running in the foreground, it will handle the signals correctly itself.

Using SSH (Scripts, Plugins, etc) to start processes

I'm trying to finish a remote deployment by restarting the two processes that make my Python App work. Like so
process-one &
process-two &
I've tried to "Execute a Shell Script" by doing this
ssh -i ~/.ssh/id_... user#xxx.xxx ./startup.sh
I've tried using the Jekins SSH Plugin and the Publish Over SSH Plugin and doing the same thing. All of the previous steps, stopping the processes, restarting other services, pulling in new code work fine. But when I get to the part where I start the services. It executes those two lines, and none of the Plugins or the Default Script execution can get off of the server. They all either hang until I restart Jekins or time out int he case of the Publish Over SSH plugin. So my build either requires a restart of Jenkins, or is marked unstable.
Has anyone had any success doing something similar? I've tried
nohup process-one &
But the same thing has happened. It's not that the services are messing up either, because they actually start properly, it's just that Jenkins doesn't seem to understand that.
Any help would be greatly appreciated. Thank you.
What probably happens in that the process when spawned (even with the &) is consuming the same input and output as your ssh connection. Jenkins is waiting for these pipes to be emptied before the job closes, thus waits for the processes to exit. You could verify that by killing your processes and you will see that the jenkins job terminates.
Dissociating outputs and starting the process remotely
There are multiple solutions to your problem:
(preferred) use proper daemon control tools. Your target platform probably has a standard way to manage those services, e.g. init.d scripts. Note, when writing init.d scripts, make sure you detach the process in the background AND ensure the input/output of the daemon are detached from the shell that starts them. There are several techniques, like like http://www.unix.com/man-page/Linux/8/start-stop-daemon/ tools, daemonize, daemontools or something like the shell script described under https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+as+a+Unix+daemon (take note of the su -s bin/sh jenkins -c "YOUR COMMAND; ...disown" etc). I also list some python specific techniques under.
ssh server 'program < /dev/null > /dev/null 2>&1 &'
or
ssh server 'program < /dev/null >> logfile.log 2>&1 &' if you want to have a crude output management (no log rotation, etc...)
potentially using setsid (I haven't tried) https://superuser.com/questions/172043/how-do-i-fork-a-process-that-doesnt-die-when-shell-exits . In my quick tests I wasn't able to get it to work though...
Python daemons
The initial question was focused on SSH, so I didn't fully described how to run the python process as daemon. This is mostly covered in other techniques:
with start-stop-daemon: start-stop-daemon and python
with upstart on ubuntu: Run python script as daemon at boot time (Ubuntu)
some more python oriented approaches:
How to make a Python script run like a service or daemon in Linux
Can I run a Python script as a service?

Resources