Failed to use hostPath '/var/lib/docker/containers' as volume in Kubernetes - docker

Question

I have failed to use HostPath /var/lib/docker/containers as a volume with the following error:
Error response from daemon: linux mounts: Path /var/lib/docker/containers is
mounted on /var/lib/docker/containers but it is not a shared or slave mount.
Here is my YAML spec (note: this is just an example for reproducing my problem in doing log collection):
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
namespace: logging
labels:
app: test
spec:
selector:
matchLabels:
app : test
template:
metadata:
labels:
app: test
spec:
containers:
- name: nginx
image: nginx:stable-alpine
securityContext:
privileged: true
ports:
- containerPort : 8003
volumeMounts:
- name: docker
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: docker
hostPath:
path: /var/lib/docker/containers
And my kubernetes version.
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1",
GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean",
BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc",
Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0",
GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean",
BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc",
Platform:"linux/amd64"}
Very much appreciating your help!

Answer 1

very appreciated for any help!
You are most probably hit by a version specific issue:
/var/lib/docker/containers is intentionally mounted by Docker with private mount
propagation and thus conflicts with Kubernetes trying to mount this directory
as rslave when running the container
You should try with 1.10.3+ where it is resolved. See the official changelog for kubernetes and check entry related to "Default mount propagation". Also check related (see the error) fluentd issue for more in-depth analysis.
Now, with that said...
David's seasoned comment with question and caution word still stands and I second that: this is quite an eyebrow raiser - nginx pod digging deep into docker engine internals (hope it is just for sake of minimal reproducible example, or log collection case, you know, something...)... Just make sure you know exactly what you are doing and why.