How to debug docker container that randomly dies with exit code 0 - docker

Question

I use Jenkins as a docker container in a custom Docker image, starting it like:
docker run -dt --privileged -p 8080:8080 -p 80:80 -p 443:443 -p 40440:40440 \
--name jenkins-master-$(hostname) \
--restart unless-stopped \
-h jenkins-master-$(hostname) \
-v $LOCAL_JENKINS_ROOT_DIR/ssl:/etc/nginx/ssl \
-v $LOCAL_JENKINS_ROOT_DIR/users:/var/jenkins/users \
-v $LOCAL_JENKINS_ROOT_DIR/jobs:/var/jenkins/jobs \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-v /var/log/jenkins:/var/log/jenkins \
[artifactory-url-here]/jenkins-master:prod
Lately, I've noticed that my container randomly restarts throughout the day, and using docker events I discovered that everytime it was restarted automatically, there was a quadruple like this:
2020-02-28T12:55:08.022251372+01:00 container die 87c7c5601bdee803072d2c8fe9405cb56b765ba6b1298461fbe17304979d7c2a (exitCode=0, image=ulcentral.ullink.lan:5001/ul-jenkins-master:prod, name=jenkins-master-cos-jenkins-prd-qEvu, version=1.0+beta.2)
2020-02-28T12:55:08.278391656+01:00 network disconnect c5afeafee778f463af73e5b7cc305cb90f0a6d6c0d81e8814cd43e5d73bac24e (container=87c7c5601bdee803072d2c8fe9405cb56b765ba6b1298461fbe17304979d7c2a, name=bridge, type=bridge)
2020-02-28T12:55:08.426582932+01:00 network connect c5afeafee778f463af73e5b7cc305cb90f0a6d6c0d81e8814cd43e5d73bac24e (container=87c7c5601bdee803072d2c8fe9405cb56b765ba6b1298461fbe17304979d7c2a, name=bridge, type=bridge)
2020-02-28T12:55:08.993064785+01:00 container start 87c7c5601bdee803072d2c8fe9405cb56b765ba6b1298461fbe17304979d7c2a (image=ulcentral.ullink.lan:5001/ul-jenkins-master:prod, name=jenkins-master-cos-jenkins-prd-qEvu, version=1.0+beta.2)
The exitcode is 0, I need more info on how to debug the underlying reason that the container stops.
Docker image (some sections omitted due to privacy reasons):
FROM debian:stretch
LABEL version 1.0+beta.2
ENV JENKINS_HOME /var/jenkins
ENV DEBIAN_FRONTEND noninteractive
RUN echo "moonlight:/share/dev-common/Applications/x86-64/linux /mnt/applis nfs defaults 0 0" >> /etc/fstab && \
echo "moonlight:/share/home /home nfs defaults 0 0" >> /etc/fstab && \
echo "sharing:/mnt/samba/share /mnt/share nfs defaults 0 0" >> /etc/fstab
# Global config
# FIXME: nfs mounting hangs forever, so no path, etc...
RUN echo "nslcd nslcd/ldap-base string dc=openldap,dc=ullink,dc=lan" | debconf-set-selections && \
echo "nslcd nslcd/ldap-uris string ldap://ldap" | debconf-set-selections && \
echo "libnss-ldapd:amd64 libnss-ldapd/nsswitch multiselect group, passwd, shadow" | debconf-set-selections
RUN apt-get upgrade -y && apt-get update
RUN apt-get -y install \
git \
libnss-ldapd \
libpam-ldapd \
locales \
maven \
nfs-common \
ntp \
openjdk-8-jdk \
openssh-server \
python2.7 \
sudo \
supervisor \
unzip \
vim \
wget \
ca-certificates \
nginx \
--no-install-recommends
RUN wget https://bootstrap.pypa.io/get-pip.py && \
python get-pip.py && \
rm get-pip.py && \
pip install pywinrm
RUN sed 's/#PermitRootLogin yes/PermitRootLogin yes/' -i /etc/ssh/sshd_config && \
sed 's/# fr_FR.UTF-8/fr_FR.UTF-8/' -i /etc/locale.gen && \
sed 's/# en_US.UTF-8/en_US.UTF-8/' -i /etc/locale.gen && \
locale-gen && \
update-locale LANG=en_US.UTF-8 && \
echo 'Europe/Paris' > /etc/timezone && \
cp /usr/share/zoneinfo/Europe/Paris /etc/localtime
RUN mkdir -p /var/run/sshd \
/var/log/supervisor \
/var/log/jenkins \
/mnt/applis \
/mnt/share \
$JENKINS_HOME/plugins
# until https://github.com/jenkinsci/jenkins/pull/3293 is merged we use a custom Jenkins build
# For some reason wget doesn't recognize the GoDaddy certs, even with `ca-certificates` being installed
RUN wget --no-proxy --no-check-certificate https://ulcentral.itiviti.com/artifactory/ext-release-local/org/jenkins-ci/main/jenkins-war/2.187-ullink/jenkins-war-2.187-ullink.war \
-O $JENKINS_HOME/jenkins.war
COPY packaged/install_plugin.sh $JENKINS_HOME/install_plugin.sh
RUN JENKINS_HOME=$JENKINS_HOME \
$JENKINS_HOME/install_plugin.sh \
[A LIST OF JENKINS PLUGINS]
COPY packaged/bootstrap.sh /var/bootstrap.sh
COPY packaged/subversion_servers /root/.subversion/servers
ADD packaged/ssh $JENKINS_HOME/ssh
COPY packaged/proxy.xml $JENKINS_HOME/proxy.xml
COPY packaged/commit-jenkins-config.sh /usr/bin/commit-jenkins-config.sh
COPY packaged/ntp.conf /etc/ntp.conf
COPY packaged/default.conf /etc/nginx/sites-available/default
RUN chmod +x /var/bootstrap.sh
RUN chmod +x /usr/bin/commit-jenkins-config.sh
EXPOSE 8080
EXPOSE 443
# For jnlp agents
EXPOSE 40440
WORKDIR $JENKINS_HOME
CMD ["/var/bootstrap.sh"]
# OUTPUT OF DOCKER INFO:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 57
Server Version: 18.09.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-693.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.639GiB
Name: cos-jenkins-prd
ID: NTNU:KI65:HCY3:5EG4:AGN5:NUGB:HS7U:I75I:LSVE:EEBN:ZY7D:HU3M
Docker Root Dir: /opt/docker/lib
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: http://proxy.ullink.lan:9876/
HTTPS Proxy: http://proxy.ullink.lan:9876/
No Proxy: ulcentral,.ullink.lan,localhost,127.0.0.1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
localhost:5001
ulcentral.ullink.lan:5001
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
EDIT
Contents of the bootstrap.sh file (used to start the jenkins process):
#!/bin/bash
export http_proxy https_proxy no_proxy
systemctl start commit-jenkins-config.timer
nslcd
rpcbind
service ntp start
service nginx start
$(which sshd)
cd $JENKINS_HOME && git pull
java -XX:+ExitOnOutOfMemoryError -Dhttp.proxyHost=proxy.ullink.lan -Dhttp.proxyPort=9876 -Dhttps.proxyHost=proxy.ullink.lan -Dhttps.proxyPort=9876 -Dhttp.nonProxyHosts="LIST OF HOSTS HERE" -Dhudson.remoting.ExportTable.unexportLogSize=0 -Dhudson.model.ParametersAction.keepUndefinedParameters=false -Dhudson.model.DirectoryBrowserSupport.CSP="" -jar jenkins.war -httpPort=8080 --sessionTimeout=10080 --httpKeepAliveTimeout=60000 2>&1 | tee /var/log/jenkins/jenkins.log

Answer 1

docker logs {containername} - it shows what was going on inside of the container.
or
kubectl logs {containername} - shows you logs, if you are using kuberneties.
for me it shows logs even if container was stopped(restarted) which is very useful to se the reason why.
And you need to make sure that you output as much as you can inside of container - if you are running some application inside do console output for any error you have.
But in general i bet that that is happening for you because of memory lack - you can just try increase amount of memory available for container. (depends on the way you are running container, would definitely recommend kubernetes for production - way more control on everything).