Debugging docker builds in jenkins server - docker

Question

I have a Jenkins server which builds docker images (using the docker maven plugin).
These Builds normally take about 40 seconds. But sometimes they take up to 1.5 hours.
Now I am wondering why. And I am also wondering how I can debug the situation.
The output of the build in the jenkins console just tells me that it hangs during the maven docker build. Example output where it hangs:
[INFO] ------------------------------------------------------------------------
[INFO] Building MY Docker Image MY Image 0.5.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> docker-maven-plugin:0.15.14:build (default-cli) > initialize # docker-image-my-image >>>
[INFO]
[INFO] <<< docker-maven-plugin:0.15.14:build (default-cli) < initialize # docker-image-my-image <<<
[INFO]
[INFO] --- docker-maven-plugin:0.15.14:build (default-cli) # docker-image-my-image ---
[INFO] Building tar: /usr/home/jenkinshome/workspace/JOB_NAME/source-repo/docker/image-my-image/docker-build.tar
[INFO] DOCKER> docker-build.tar: Created [my-image] in 69 milliseconds
And now it hangs.
Can I somehow watch at which operation docker is hanging?
UPDATE
The build also hangs on the host maschine, when started directly via docker build.
This is the end of the output of doing this with strace:
futex(0xc820028908, FUTEX_WAKE, 1) = 1
clock_gettime(CLOCK_REALTIME, {1486567620, 329675667}) = 0
clock_gettime(CLOCK_REALTIME, {1486567620, 329756643}) = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, 23) = 0
clock_gettime(CLOCK_REALTIME, {1486567620, 330059768}) = 0
epoll_create1(EPOLL_CLOEXEC) = 4
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3892185696, u64=139659048782432}}) = 0
getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
getpeername(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, [23]) = 0
futex(0xc820028908, FUTEX_WAKE, 1) = 1
read(3, 0xc820349000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
write(3, "POST /v1.24/build?buildargs=%7B%"..., 349) = 349
futex(0xc820028d08, FUTEX_WAKE, 1) = 1
write(3, "7ff\r\nockerfile\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2054) = 2054
ioctl(0, TIOCGWINSZ, {ws_row=57, ws_col=105, ws_xpixel=0, ws_ypixel=0}) = 0
ioctl(0, TIOCGWINSZ, {ws_row=57, ws_col=105, ws_xpixel=0, ws_ypixel=0}) = 0
) = 481, "Sending build context to Docker "..., 48Sending build context to Docker daemon 2.048 kB
write(1, "\r\n", 2
) = 2
write(3, "0\r\n\r\n", 5) = 5
futex(0xc820028d08, FUTEX_WAKE, 1) = 1
futex(0x12d3a48, FUTEX_WAIT, 0, NULLStep 1 : FROM jboss-db
---> 74a0020a9922
Step 2 : MAINTAINER ***
---> Using cache
---> 5d38cbd5501b
Step 3 : USER root:root
---> Running in 64b14554d8be
And at this point it hangs

Answer 1

You have a bunch of options here, and you'll have to feel out which one works best for you. You can:
strace the process on the docker host.
Use something like sysdig on the host, which is easier to use and more detailed.
do printf debugging or some sort of logging that indicates your build is still alive to stdout/stderr and the docker log -f <build container> to see if it's the build that's stuck or the jenkins slave/master processes.
Any more detailed advice will probably need more information about your setup, what you mean by hang and which component is hung exactly.