I'm building a docker image for running Gitlab CI jobs. One of the components needs systemd up and running inside the container, this is not trivial but there are several guides on the web so I managed to do it. Part of the process requires to define this entrypoint in the Dockerfile:
ENTRYPOINT ["/usr/sbin/init"]
so that systemd runs as PID 1 in the container, as needed. This seems to conflict with Gitlab CI requirements: as far as I understand gitlab-runner overrides the Dockerfile's CMD to spawn a shell which then executes the CI script. But the /usr/sbin/init entrypoint cannot understand the Gitlab's CMD so the shell is not spawned and the execution halts.
I cannot figure out how to solve this:
executing an entrypoint script which starts /usr/sbin/init and then a shell won't work because systemd won't be PID1;
using a shell as ENTRYPOINT and then systemd as CMD won't work since Gitlab CI overrides CMD.
I cannot think of any other possible solution, so any help is much appreciated.
You can always fork a bash shell to do the work and then exec systemd with pid 1 like this:
ENTRYPOINT [ "/bin/bash", "-c", "nohup bash -c \"${#} ; /sbin/init 0\" & exec /sbin/init", "${#}" ]
Finally I have been able to put together a working solution. The ENTRYPOINT executes a script:
ENTRYPOINT ["/entrypoint.sh"]
which retrieves the stdin/stdout fds of PID 1 (i.e. those that will be used by Gitlab CI for the job I/O), pins them by attaching them to a long-running process and then spawns systemd as PID 1:
#!/bin/bash
# Start a long-running process to keep the container pipes open
sleep infinity < /proc/1/fd/0 > /proc/1/fd/1 2>&1 &
# Wait a bit before retrieving the PID
sleep 1
# Save the long-running PID on file
echo $! > /container-pipes-pid
# Start systemd as PID 1
exec /usr/lib/systemd/systemd
The pinning of stdin/stdout is needed since some systemd versions close stdin/stdout at startup, but they are needed since the CI infrastructure uses these to send CI commands to the shell and receive console output. So attaching them to sleep infinity makes them persist even after exec /usr/lib/systemd/systemd.
The shell is then spawned by a systemd unit (previously enabled in the Dockerfile):
[Unit]
Description=Start bash shell attached to container STDIN/STDOUT
[Service]
Type=simple
PassEnvironment=PATH LD_LIBRARY_PATH
ExecStart=/bin/bash -c "echo Attaching to pipes of PID `cat container-pipes-pid` && exec /bin/bash < /proc/`cat container-pipes-pid`/fd/0 > /proc/`cat container-pipes-pid`/fd/1 2>/proc/`cat container-pipes-pid`/fd/2"
ExecStopPost=/usr/bin/systemctl exit $EXIT_STATUS
[Install]
WantedBy=multi-user.target rescue.target
The pinned fds are attached to the shell (no conflict is generated by this since neither systemd nor sleep do I/O on those fds), so the shell correctly receives CI commands and directs console output to CI logs. Then at the end of the job, when stdin is closed by the CI infrastructure, the shell terminates and the unit shuts down the container, returning the job execution result so that the CI correctly retrieves the job outcome.
This is a rather involved implementation which can for sure be refined and improved, but is working beautifully for my purpose.
Related
I'm trying to understand why my Docker container does not stop gracefully and just times out. The container is running crond:
FROM alpine:latest
ADD crontab /etc/crontabs/root
RUN chmod 0644 /etc/crontabs/root
CMD ["crond", "-f"]
And the crontab file is:
* * * * * echo 'Working'
# this empty line required by cron
Built with docker build . -t periodic:latest
And run with docker run --rm --name periodic periodic:latest
This is all good, but when I try to docker stop periodic from another terminal, it doesn't stop gracefully, the time out kicks in and is killed abruptly. It's like crond isn't responding to the SIGTERM.
crond is definitely PID 1
/ # ps
PID USER TIME COMMAND
1 root 0:00 crond -f
6 root 0:00 ash
11 root 0:00 ps
However, if I do this:
docker run -it --rm --name shell alpine:latest ash and
docker exec -it shell crond -f in another terminal, I can kill crond from the first shell with SIGTERM so I know it can be stopped with SIGTERM.
Thanks for any help.
Adding an init process to the container (init: true in docker-compose.yml) solved the problem.
EDIT: I read this https://blog.thesparktree.com/cron-in-docker to understand the issues and solutions around running cron in Docker. From this article:
"Finally, as you’ve been playing around, you may have noticed that it’s difficult to kill the container running cron. You may have had to use docker kill or docker-compose kill to terminate the container, rather than using ctrl + C or docker stop.
Unfortunately, it seems like SIGINT is not always correctly handled by cron implementations when running in the foreground.
After researching a couple of alternatives, the only solution that seemed to work was using a process supervisor (like tini or s6-overlay). Since tini was merged into Docker 1.13, technically, you can use it transparently by passing --init to your docker run command. In practice you often can’t because your cluster manager doesn’t support it."
Since my original post and answer, I've migrated to Kubernetes, so init in docker-compose.yml won't work. My container is based on Debian Buster, so I've now installed tini in the Dockerfile, and changed the ENTRYPOINT to ["/usr/bin/tini", "--", "/usr/local/bin/entrypoint.sh"] (my entrypoint.sh finally does exec cron -f)
The key is that you cannot stop a pid=1 process in docker. It supposes that docker stops (or kills if it was launched with --rm).
That's why if you run -it ... ash, shell has pid 1 and you can kill other processes.
If you want your cron is killable without stopping/killing docker, just launch another process as entrypoint:
Launch cron after docker entrypoint (For example, run as cmd tail -F /dev/null and then launch cron docker run -d yourdocker service cron start)
I know I can use the dockerfile's CMD RUN and ENTRYPOINT commands to run a script when the container initiates, but how can I make the container run a script every time when the container restarts on failure?
entrypoint runs every time a container starts, or restarts. It's common practice to put startup configuration in a shell script that then execs the application's "true" entrypoint at the end. (See What purpose does using exec in docker entrypoint scripts serve? for why exec is important).
Remember, docker is really just a wrapper around filesystem , process, and network namespacing. It can't restart your container in any way other than rerunning the same process it started in the first place.
You can try it yourself with an invocation something like this:
docker run -d --restart=always --entrypoint=sh alpine -c "sleep 5; echo Exiting; exit"
if you docker logs -f that container, you'll see the Exiting come out after every 5 seconds. Note that the container stopping will also stop the log following though, so you'll have to run it again to see the next restart.
Official Docker images like MySQL can be run like this:
docker run -d --name mysql_test mysql/mysql-server:8.0.13
And it can run indefinitely in the background.
I want to try to create an image which does the same, specifically a Flask development server (just for testing). But my container exit immediately. My Dockerfile is like this:
FROM debian:buster
ENV TERM xterm
RUN XXXX # some apt-get and Python installation stuffs
ENTRYPOINT [ "flask", "run", "--host", "0.0.0.0:5000" ]
EXPOSE 80
EXPOSE 5000
USER myuser
WORKDIR /home/myuser
However it exited immediately as soon as it is ran. I also tried "bash" as an entry point just so to make sure it isn't a Flask configuration issue and it also exited.
How do I make it so that it runs as THE process in the container?
EDIT
OK someone posted below (but later deleted), the command to test is to use tail -f /dev/null, and it does run indefinitely. I still don't understand why bash doesn't work as a process which doesn't exist (does it?). But my flask configuration is probably off.
EDIT 2
I see that running without the -d flag print out the stdout (or stderr) so I can diagnose the problem.
Let's clear things out.
In general, a container exits as soon as its entrypoint is successfully executed.
In your case, without being a python expert this ENTRYPOINT [ "flask", "run", "--host", "0.0.0.0:5000" ] would be enough to keep the container alive. But I guess you have some configuration error and due to that error the container exited before running flask command. You can validate this by running docker ps -a and inspect the exit code(possibly 1).
Let's now discuss about the questions in your edits.
The key part of your misunderstanding derives from the -d flag.
You are right to think that setting bash as entrypoint would be enough to keep container alive but you need to attach to that shell.
When running in detach mode(-d), container will execute bash command but as soon as no one is attached to that shell, it will exit. In addition, using this flag will prevent you from viewing container logs lively(however you may use docker logs container_id to debug) which is very useful when you are in an early phase of setting thing up. So I recommend using this flag only when you are sure that everything works as intended.
To attach to bash shell and keep container alive, you should use the -it flag so that the bash shell will be attached to the current shell invoking the docker run command.
-t : Allocate a pseudo-tty
-i : Keep STDIN open even if not attached
Please also consult official documentation about foreground vs background mode.
The answer to your edit is: when do docker run <container> bash it will literally call bash and exit 0, because the command (bash) was successful. Bash isn't a shell, it's a command.
If you ran docker run -it <container> tail -f /dev/null and then docker exec -it /bin/bash. You'd drop into the shell, because its the command you ran.
Your Dockerfile doesn't have a command to run in the background that is persistent, in mysqls case, it runs mysqld, which starts a server on PID 0.
When PID 0 exits, the container stops.
Your entrypoint is most likely failing to start, or starting and exiting because of how your command is running.
I would try changing your entrypoint to a
I have a script /init that launches apache and neo4j. This script is already in the image ubuntu:14. The following is the content of /init:
service apache2 start
service neo4j start
From this image, I am creating another image with the following dockerfile
FROM ubuntu:v14
EXPOSE 80 80
ENTRYPOINT ["/init"]
When I run the command docker run -d ubuntu:v15, the container starts and then exit. As far as I understood, -d option runs the container in the background. Also, the script\init launches two daemons. Why does the container exit ?
In fact, I think your first problem is the #! in init file, if you did not add something like #!/bin/bash at the start, container will complain like next:
shubuntu1#shubuntu1:~$ docker logs priceless_tu
standard_init_linux.go:207: exec user process caused "exec format error"
But even you fix above problem, you will still can't start your container, the reason same as other folks said: the PID 1 should always there, in your case after service xxx start finish, the PID 1 exit which will also result in container exit.
So, to conquer this problem you should set one command never exit, a minimal workable example for your reference:
Dockerfile:
FROM ubuntu:14.04
RUN apt-get update && \
apt-get install -y apache2
COPY init /
RUN chmod +x /init
EXPOSE 80
ENTRYPOINT ["/init"]
init:
#!/bin/bash
# you can add other service start here
# e.g. service neo4j start as you like if you have installed it already
# next will make apache run in foreground, so PID1 not exit.
/usr/sbin/apache2ctl -DFOREGROUND
When your Dockerfile specifies an ENTRYPOINT, the lifetime of the container is exactly the length of whatever its process is. Generally the behavior of service ... start is to start the service as a background process and then return immediately; so your /init script runs the two service commands and completes, and now that the entrypoint process is completed, the container exits.
Generally accepted best practice is to run only one process in a container. That's especially true when one of the processes is a database. In your case there are standard Docker Hub Apache httpd and neo4j images, so I'd start by using an orchestration tool like Docker Compose to run those two containers side-by-side.
I'm new to docker, trying to build my the first docker container on AWS free tier account t2.micro instance
I am able to build my docker based on below Dockerfile.
FROM java:8
COPY content-service /
RUN chmod +x /bin/start.sh
CMD bash -C '/bin/start.sh'
EXPOSE 8081
MAINTAINER Velu
It's failing and exits while is trying to run the container command following error message is getting.
[ec2-user#ip-172-31-30-38 cis-docker]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
content-service latest 9fa3ca5a8ac3 10 minutes ago 705.7 MB
docker.io/java 8 d23bdf5b1b1b 8 months ago 643.1 MB
[ec2-user#ip-172-31-30-38 cis-docker]$ sudo docker --debug run -p 8082:8082 content-service
DEBU[0000] framesize: 18
Starting service.
DEBU[0000] Corrupted prefix: []
DEBU[0000] [hijack] End of stdout
Any help would be appreciated.
I ran your docker file by copying a simple shell script which prints "hello" on to console. It ran successfully.
Also, it does not seem like your docker container is exiting. Docker is running your shell script and exiting gracefully. To verify, check the exit code of your docker run command by running the below command as soon as your docker run is finished.
echo $?
If the above command prints "0" on to the screen then docker did not fail.
If you are expecting your shell script to run as a background daemon you need to make sure that your script does not exit. The reason why docker container is exiting as soon as your shell scriptspecified in CMD line your Dockerfile is finished executing is because Docker's design philosophy is to run one process per container. In your case that single process is your shell script. So start whatever service you are starting in your shell script as a foreground process and the container will keep running as long as that service is running. As soon as that service dies your container dies which is what you might want.
Hope that helps.
I found and fixed this issue.
In my start.sh script last line I have this below line.
java -cp $CLASS_PATH $JVM_OPTIONS $CLASS_NAME ${ARGUMENTS[#]} & echo $! > $PID_FILE
In that line, I did remove this & echo $! > $PID_FILE,
Working:
java -cp $CLASS_PATH $JVM_OPTIONS $CLASS_NAME ${ARGUMENTS[#]}