Docker build inside kubernetes pod fails with "could not find bridge docker0" - docker

Question

I moved our build agents into Kubernetes / Container Engine. They used to run on container vm (version container-vm-v20160321) and mount docker.sock into the docker container so we can run docker build from inside the container.
This used the following manifest:
apiVersion: v1
kind: Pod
metadata:
name: gocd-agent
spec:
containers:
- name: gocd-agent
image: travix/gocd-agent:16.8.0
imagePullPolicy: Always
volumeMounts:
- name: ssh-keys
mountPath: /var/go/.ssh
readOnly: true
- name: gcloud-keys
mountPath: /var/go/.gcloud
readOnly: true
- name: docker-sock
mountPath: /var/run/docker.sock
- name: docker-bin
mountPath: /usr/bin/docker
env:
- name: "GO_SERVER_URL"
value: "https://server:8154/go"
- name: "AGENT_KEY"
value: "***"
- name: "AGENT_RESOURCES"
value: "docker"
- name: "DOCKER_GID_ON_HOST"
value: "107"
restartPolicy: Always
dnsPolicy: Default
volumes:
- name: ssh-keys
gcePersistentDisk:
pdName: sh-keys
fsType: ext4
readOnly: true
- name: gcloud-keys
gcePersistentDisk:
pdName: gcloud-keys
fsType: ext4
readOnly: true
- name: docker-sock
hostPath:
path: /var/run/docker.sock
- name: docker-bin
hostPath:
path: /usr/bin/docker
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
Now after moving it into a full-blown Container Engine cluster - version 1.3.5 - with the following manifest it fails.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gocd-agent
spec:
replicas: 2
strategy:
type: Recreate
revisionHistoryLimit: 1
selector:
matchLabels:
app: gocd-agent
template:
metadata:
labels:
app: gocd-agent
spec:
containers:
- name: gocd-agent
image: travix/gocd-agent:16.8.0
imagePullPolicy: Always
securityContext:
privileged: true
volumeMounts:
- name: ssh-keys
mountPath: /k8s-ssh-secret
- name: gcloud-keys
mountPath: /var/go/.gcloud
- name: docker-sock
mountPath: /var/run/docker.sock
- name: docker-bin
mountPath: /usr/bin/docker
env:
- name: "GO_SERVER_URL"
value: "https://server:8154/go"
- name: "AGENT_KEY"
value: "***"
- name: "AGENT_RESOURCES"
value: "docker"
- name: "DOCKER_GID_ON_HOST"
value: "107"
volumes:
- name: ssh-keys
secret:
secretName: ssh-keys
- name: gcloud-keys
secret:
secretName: gcloud-keys
- name: docker-sock
hostPath:
path: /var/run/docker.sock
- name: docker-bin
hostPath:
path: /usr/bin/docker
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
It seems to start building just fine, but eventually it fails with a no such interface error:
Executing "docker build --force-rm=true --no-cache=true --file=target/docker/Dockerfile --tag=****:1.0.258 ."
Sending build context to Docker daemon 557.1 kB
...
Sending build context to Docker daemon 78.04 MB
Step 1 : FROM travix/base-debian-jre8
---> a130b5e1b4d4
Step 2 : ADD ***-1.0.258.jar ***.jar
---> 8d53e68e93a0
Removing intermediate container d1a758c9baeb
Step 3 : ADD target/newrelic newrelic
---> 9dbbb1c1db58
Removing intermediate container 461e66978c53
Step 4 : RUN bash -c "touch /***.jar"
---> Running in 6a28f48c9fd1
Removing intermediate container 6a28f48c9fd1
failed to create endpoint stupefied_shockley on network bridge: adding interface veth095b905 to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface
Is it impossible to run docker build inside a pod due to Kubernetes networking or do I need to configure the pod differently? Or is it a bug in the particular docker version on the host?
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:20:08 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:20:08 2016
OS/Arch: linux/amd64
The bridge actually seems to exist on the host:
$ sudo brctl show
bridge name bridge id STP enabled interfaces
cbr0 8000.063c847a631e no veth0a58740b
veth1f558898
veth8797ea93
vethb11a7490
vethc576cc01
docker0 8000.02428db6a46e no
And docker info for completeness
$ sudo docker info
Containers: 15
Running: 14
Paused: 0
Stopped: 1
Images: 67
Server Version: 1.11.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 148
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 25.57 GiB
Name: gke-tooling-default-pool-1fa283a6-8ufa
ID: JBQ2:Q3AR:TFJG:ILTX:KMHV:M67A:NYEM:NK4G:R43J:K5PS:26HY:Q57S
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
And
$ uname -a
Linux gke-tooling-default-pool-1fa283a6-8ufa 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux