I'm trying to make an application work on Windows that's been developed only on Unices. It's all dockerized and it uses the traefik load balancer. The volumes for the docker for running traefik looks like this:
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro,delegated
- ${PWD}/load_balancer/traefik.toml:/etc/traefik/traefik.toml:ro,delegated
The first volume works fine on Mac or Linux, but does it on Windows? The application is failing (the load balancer is giving a 404) and it might be related to that volume. When I start the image, the socket looks like a socket:
/ # ls -laF /var/run/docker.sock
srw-rw---- 1 root root 0 Sep 2 11:04 /var/run/docker.sock=
Is this working? Any way to test it? What's the correct way of doing this?
Trying to figure this out, I tried replacing it with this:
volumes:
- //./pipe/docker_engine:/var/run/docker.sock
based on various articles and bug reports I found online. The docker image starts but it fails in the same way and now in the docker container it looks like a directory:
/ # ls -laF /var/run/docker.sock
total 4
drwxr-xr-x 2 root root 40 Sep 3 14:52 ./
drwxr-xr-x 1 root root 4096 Sep 3 14:57 ../
Following Marc ABOUCHACRA's answers, I tried:
volumes:
- type: npipe
source: ////./pipe/docker_engine
target: /var/run/docker.sock
consistency: delegated
but that also looks like a directory:
/ # ls -laF /var/run/docker.sock
total 4
drwxr-xr-x 2 root root 40 Sep 3 14:52 ./
drwxr-xr-x 1 root root 4096 Sep 3 14:57 ../
I also tried this:
volumes:
- npipe:////./pipe/docker_engine:/var/run/docker.sock:ro,delegated
but that fails with this error:
ERROR: Volume npipe:////./pipe/docker_engine:/var/run/docker.sock:ro,delegated has incorrect format, should be external:internal[:mode]
The whole docker-compose.yml section looks like this:
lb:
image: load-balancer
build: ${WORKSPACE}/go-home/load_balancer
ports:
- 80:80
- 443:443
links:
- wifi-ui-dev
- wifi-ui-prod
- portal
- wifi-api
env_file:
- .env
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro,delegated
- ${PWD}/load_balancer/traefik.toml:/etc/traefik/traefik.toml:ro,delegated
My question is specifically about running this docker image, which is a Linux, on a Windows host, running Docker for Windows. I understand that I can run it on a Linux host by installing Linux on another machine or a VM on the Windows machine, it's equivalent. Running Windows guests is not what I'm after either in case there's a way of exposing sockets from Windows to Windows only.
If you cannot nor want use network sockets, then you can use named pipes.
The syntax depends whether you run Linux or Windows containers and on the shell you use.
Linux containers
If you run Linux containers on a Windows machine, this seems to work using Powershell or bash:
docker run --rm -it -v "//var/run/docker.sock://var/run/docker.sock" image_with_docker docker version
Please note the extra / in front of /var/run/docker.sock, both for the source and destination volumes.
Windows containers
If you run Windows containers on a Windows machine, this seems to work using Powershell or bash:
docker run -v "//./pipe/docker_engine://./pipe/docker_engine" --rm -it image-with-docker docker version
Note that this works only in Powershell:
docker run -v "\\.\pipe\docker_engine:\\.\pipe\docker_engine" --rm -it image-with-docker docker version
Therefore, it's better to use the version with /.
Extra - docker-compose.yml
If you use a docker-compose.yaml file, this works with Windows containers.
version: '3.7'
services:
docker:
image: image-with-docker
command:
- docker
- version
volumes:
- type: npipe
source: \\.\pipe\docker_engine
target: \\.\pipe\docker_engine
With Linux containers, you can use the shortened form:
docker:
image: image-with-docker
command:
- docker
- version
volumes:
- //var/run/docker.sock://var/run/docker.sock
Extra - Kubernetes
If you are running Windows containers on a Windows node in Kubernetes, this seems to work:
apiVersion: v1
kind: Pod
spec:
containers:
- name: docker
image: image-with-docker
command:
- powershell
args:
- Start-Sleep
- "999999"
volumeMounts:
- mountPath: \\.\pipe\docker_engine
name: dockersock
volumes:
- name: dockersock
hostPath:
path: \\.\pipe\docker_engine
type: null
nodeSelector:
kubernetes.io/os: windows
In this case, beside using the \, please note the type: null in the definition of the dockersock volume: if you don't set it, it will not work.
Notes
Everything was tested on docker 19.03 and on Kubernetes 1.18.
Client:
Version: 19.03.3
API version: 1.40
Go version: go1.12.10
Git commit: 2355349d-
Built: 10/14/2019 16:41:26
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.24)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:37:20 2020
OS/Arch: windows/amd64
Experimental: false
If you encounter the following error on windows:
cannot create container for service portainer: Unrecognised volume spec: file '\.\pipe\docker_engine' cannot be mapped. Only directories can be mapped on this platform
ERROR: Encountered errors while bringing up the project.
Try adding an extra slash to it, resulting in following volumes section:
volumes:
- source: \\.\pipe\docker_engine\
target: \\.\pipe\docker_engine\
type: npipe
Tested with compose 3.7 and docker CE 19.03.12
Using short syntax with the type of the bind mount is not possible : npipe:////./pipe/docker_engine:/var/run/docker.sock:ro,delegated
You need to use the long syntax in your compose file :
volumes:
- type: npipe
source: ////./pipe/docker_engine
target: /var/run/docker.sock
consistency: delegated
You can find some documentation about the long syntax in the official documentation. This syntaxe is from v3.2
Also keep in mind what #lucas-ramage said about using windows container only when using npipe.
In 2022, on Windows 11, what worked for me is below volume configuration (Note the doube slash (//) on the host side of configruation. Not sure why docker_enginer variant does not work for me.
-v "//var/run/docker.sock:/var/run/docker.sock"
Per the Docker for Windows FAQ,
On Docker Desktop for Windows, clients can connect to the Docker Engine through a named pipe: npipe:////./pipe/docker_engine
See also this issue on GitHub,
The windows version of docker doesn't use unix socket (/var/run/docker.sock) but npipe (npipe:////./pipe/docker_engine). So you have either
to switch to linux container which runs docker in a full virtualized linux with unix socket
to pass the npipe instead of the unix socket to the container (windows container only)
to use a network socket (should work with linux and windows container)
However, since this is a Linux container, your options are either A) Run docker in a virtual machine (first choice above), or B) Use a network socket (the third choice).
Related
I've been binding the host docker socket and cli so that I can run docker and compose commands from within running containers for over a year without issue but since updating to docker version 20.10.7 and compose version 1.29.2 I can't get my containerised environment to launch without the following error:
invalid mount config for type "bind": bind source path does not exist: /usr/local/bin/docker
Nothing has changed other than I updated Docker Desktop.
The location of the docker binary (symlink) on the host is still present:
0 lrwxr-xr-x 1 aadams-mbp staff 54 3 Aug 2018 /usr/local/bin/docker -> /Applications/Docker.app/Contents/Resources/bin/docker
The target of the symlink permissions look like this:
133608 -rwxr-xr-x 1 root admin 68405888 7 Jul 17:59 /Applications/Docker.app/Contents/Resources/bin/docker
This snippet is from my docker-compose.yaml file:
volumes:
# Bind docker CLI so can run docker commands
# from inside the container. Double check the
# location of the source binary on hosts that
# are not Mac OS. Docker might be in /usr/bin/docker,
# but on Mac OS it is at /usr/local/bin/docker.
- type: bind
source: /var/run/docker.sock
target: /var/run/docker.sock
- type: bind
source: ${DOCKER_BIN_SRC}
target: /usr/bin/docker
The ${DOCKER_BIN_SRC} is pulled in from a .env file (snippet):
##
# Docker bind
#
DOCKER_BIN_SRC=/usr/local/bin/docker
I am running on Mac OS Mojave version 10.14.6
A few days ago, my watchtower updated Confluence on Docker with the 6.15.1-alpine tag. It's hosted using Atlassians official image. Since those update, Confluence shows the setup screen. Haven't any chance to get inside the admin panel. When continue the wizard end entering server credentials of the existing installation, it gave an error that an installation already exists that would be overwritten if continued.
It was a re-push of the exact version tag 6.15.1 tag, not a regular version update. So there seems no possibility to use the old, working image. Also other versions seems re-pushed. Tried some older ones and also a new one, without success.
docker-compose.yml
version: "2"
volumes:
confluence-home:
services:
confluence:
container_name: confluence
image: atlassian/confluence-server:6.15.1-alpine
#restart: always
mem_limit: 6g
volumes:
- confluence-home:/var/atlassian/application-data/confluence
- ./confluence.cfg.xml:/var/atlassian/application-data/confluence/confluence.cfg.xml
- ./server.xml:/opt/atlassian/confluence/conf/server.xml
- ./mysql-connector-java-5.1.42-bin.jar:/opt/atlassian/confluence/lib/mysql-connector-java-5.1.42-bin.jar
networks:
- traefik
environment:
- "TZ=Europe/Berlin"
- JVM_MINIMUM_MEMORY=4096m
- JVM_MAXIMUM_MEMORY=4096m
labels:
- "traefik.port=8090"
- "traefik.backend=confluence"
- "traefik.frontend.rule=Host:confluence.my-domain.com"
networks:
traefik:
external: true
I found out that there were the following changes on the images:
Ownership
The logs throwed errors about not beinng able to write on log files because nearly the entire home directory was owned by an user called bin:
root#8ac38faa94f1:/var/atlassian/application-data/confluence# ls -l
total 108
drwx------ 2 bin bin 4096 Aug 19 00:03 analytics-logs
drwx------ 3 bin bin 4096 Jun 15 2017 attachments
drwx------ 2 bin bin 24576 Jan 12 2019 backups
[...]
This could be fixed by executing a chown:
docker exec -it confluence bash
chown confluence:confluence -R /var/atlassian/application-data/confluence
Moutings inside mount
My docker-compose.yml mounts a volume to /var/atlassian/application-data/confluence and inside those volume, the confluence.cfg.xml file was mounted from current directory. This approach is a bit older and should seperate the user data in the volume from configuration files like docker-compose.yml and also the application itself as confluence.cfg.xml.
Seems not properly working any more on using Docker 17.05 and Docker-Compose 1.8.0 (at least in combination with Confluence), so I simply removed that second mount and placed the configuration file inside the volume.
Atlassian creates config files now dynamically
It was noticeable that my mounted configuration files like confluence.cfg.xml and server.xml were overwritten by Atlassians container. Their source code shows that they now use Jina2, a common Python template engine used in e.g. Ansible. A python script parse those files on startup and create Confluences configuration files, without properly checking on all of those files if they already exists.
Mounting them as read only caused the app to crash because this is also not handled in their Python script. By analyzing their templates, I learned that they replaced nearly every config item by environment variables. Not a bad approach, so I specified my server.xml parameters by env variables instead of mouting the entire file.
In my case, Confluence is behind a Traefik reverse proxy and it's required to tell Confluence it's final application url for end users:
environment:
- ATL_proxyName=confluence.my-domain.com
- ATL_proxyPort=443
- ATL_tomcat_scheme=https
Final working docker-compose.yml
By applying all modifications above, accessing the existing installation works again using the following docker-compose.yml file:
version: "2"
volumes:
confluence-home:
services:
confluence:
container_name: confluence
image: atlassian/confluence-server:6.15.1
#restart: always
mem_limit: 6g
volumes:
- confluence-home:/var/atlassian/application-data/confluence
- ./mysql-connector-java-5.1.42-bin.jar:/opt/atlassian/confluence/lib/mysql-connector-java-5.1.42-bin.jar
networks:
- traefik
environment:
- "TZ=Europe/Berlin"
- JVM_MINIMUM_MEMORY=4096m
- JVM_MAXIMUM_MEMORY=4096m
- ATL_proxyName=confluence.my-domain.com
- ATL_proxyPort=443
- ATL_tomcat_scheme=https
labels:
- "traefik.port=8090"
- "traefik.backend=confluence"
- "traefik.frontend.rule=Host:confluence.my-domain.com"
networks:
traefik:
external: true
I'm trying to setup the Concourse CI locally by following the documentation. Everything goes great until I try to run a sample hello-world pipeline. The job results in this error...
runc create: exit status 1: container_linux.go:264: starting container process caused "process_linux.go:339: container init caused \"rootfs_linux.go:56: mounting \\\"/worker-state/3.8.0/assets/bin/init\\\" to rootfs \\\"/worker-state/volumes/live/a8d3b403-19e7-4f16-4a8a-40409a9b017f/volume/rootfs\\\" at \\\"/worker-state/volumes/live/a8d3b403-19e7-4f16-4a8a-40409a9b017f/volume/rootfs/tmp/garden-init\\\" caused \\\"open /worker-state/volumes/live/a8d3b403-19e7-4f16-4a8a-40409a9b017f/volume/rootfs/tmp/garden-init: permission denied\\\"\""
Looks like I'm getting a permissions error but I've double-checked that the container is running in privileged mode.
$ docker inspect --format='{{.HostConfig.Privileged}}' concourse_concourse_1
true
Docker Compose File
version: '3'
services:
concourse-db:
image: postgres
environment:
- POSTGRES_DB=concourse
- POSTGRES_PASSWORD=concourse_pass
- POSTGRES_USER=concourse_user
- PGDATA=/database
concourse:
image: concourse/concourse
command: quickstart
privileged: true
depends_on: [concourse-db]
ports: ["8080:8080"]
environment:
- CONCOURSE_POSTGRES_HOST=concourse-db
- CONCOURSE_POSTGRES_USER=concourse_user
- CONCOURSE_POSTGRES_PASSWORD=concourse_pass
- CONCOURSE_POSTGRES_DATABASE=concourse
- CONCOURSE_EXTERNAL_URL
- CONCOURSE_BASIC_AUTH_USERNAME
- CONCOURSE_BASIC_AUTH_PASSWORD
- CONCOURSE_NO_REALLY_I_DONT_WANT_ANY_AUTH=true
- CONCOURSE_WORKER_GARDEN_NETWORK
Pipeline
---
jobs:
- name: job-hello-world
public: true
plan:
- task: hello-world
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
run:
path: echo
args: [hello world]
Concourse Version
$ curl http://192.168.99.100:8080/api/v1/info
{"version":"3.12.0","worker_version":"2.0"}
Other Versions
$ docker --version
Docker version 18.04.0-ce, build 3d479c0
$ docker-machine --version
docker-machine version 0.14.0, build 89b8332
$ docker-compose --version
docker-compose version 1.21.0, build unknown
$ system_profiler SPSoftwareDataType
Software:
System Software Overview:
System Version: macOS 10.13.1 (17B1003)
Kernel Version: Darwin 17.2.0
Boot Volume: OSX
While everything on the surface may look like it's up to date. It's important to note that docker-machine runs docker inside of VMs which can get stale if you're not updating them on a regular basis and Concourse needs kernel 3.19 or higher.
Running docker info can shed some light on the situation from Docker server's perspective.
What worked for me was...
$ docker-compose down
$ docker-machine upgrade
$ docker-compose up -d
When run command:
docker run -it -v some_volume:/abc/xyz --volume-driver=btrfs a_docker_image /bin/bash
terminal shows:
docker: Error response from daemon: create some_volume: Error looking up volume plugin btrfs: plugin not found.
====================
But if create volume first:
docker volume create --opt type=btrfs --name some_volume
It will create volume successfully. Now if I try to run container and create a new volume:
docker run -it -v some_volume:/abc/xyz --volume-driver=btrfs a_docker_image /bin/bash
It shows (of course it makes sense, since the same name volume has been already created):
docker: Error response from daemon: create some_volume: conflict: volume name must be unique.
And if I try to run container with the existing volume:
docker run -it -v some_volume:/abc/xyz a_docker_image /bin/bash
It returns:
docker: Error response from daemon: missing device in volume options.
====================
Could anyone help tell me how to install the volume plugin btrfs for docker? I haven't found any useful information regarding that except some introduction about plugin (but not how to install). Thanks in advance.
As suggested by #forevergenin in comments, here is my docker environment:
docker version
Client:
Version: 1.11.0
API version: 1.23
Go version: go1.5.4
Git commit: 4dc5990
Built: Wed Apr 13 18:13:28 2016
OS/Arch: darwin/amd64
Server:
Version: 1.11.0
API version: 1.23
Go version: go1.5.4
Git commit: 4dc5990
Built: Wed Apr 13 19:36:04 2016
OS/Arch: linux/amd64
docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 39
Server Version: 1.11.0
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 121
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.1.19-boot2docker
Operating System: Boot2Docker 1.11.0 (TCL 7.0); HEAD : 32ee7e9 - Wed Apr 13 20:06:49 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.1 MiB
Name: default
ID: 74TB:OVH5:S3GD:UQUG:ILWG:5NVH:2MSH:5H7R:A5H4:GSLV:2Q6D:ZIR6
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug mode (client): false
Debug mode (server): true
File Descriptors: 15
Goroutines: 32
System Time: 2016-08-15T13:57:03.866016657Z
EventsListeners: 0
Username: thyrlian
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
I am new to btrfs with docker, but here is my understanding:
Using btrfs as a storage driver means that docker will use btrfs internally for the images and containers (that is explained here). Specifically, look at the installation details here: they make you create a btrfs partition and mount /var/lib/docker on it. When you restart your docker daemon after that, docker info should tell you "Storage Driver: btrfs".
Using the btrfs driver, the image's base is saved int /var/lib/docker/btrfs/subvolumes, and then they do snapshots (but I am not sure where they save them exactly). But that is done automatically without you specifying the driver (I would guess that specifying the driver is useful when you have multiple drivers that can run on a given filesystem. But the btrfs driver seems to be the default when /var/lib/docker is formatted in btrfs.
Regarding volumes, I believe that they are not saved as btrfs subvolumes. They seem to be simple folders in /var/lib/docker/volumes/. Again, I can imagine this as being the normal behavior of docker: images and containers are layered, but volumes are simple directories.
At least, that is the behavior I observe:
If I pull an image or create a container, I get btrfs subvolumes created.
I could create a volume by simply using docker volume create testvol1 and mount it in a container. But then it is not a btrfs subvolume.
If you want to have your volumes in btrfs subvolumes, then I believe that you might need to create the subvolumes manually and mount the volumes in them directly.
System
OSX 10.9.5
Docker version 1.4.1, build 5bc2ff8
Image niallo/strider (latest) a51ba391459b
Goal
Setup a docker instance as per this guide
First attempt
I followed the guide steps, but when I ran the final image:
docker run -d -p 3000:3000 -p 27000:27017 -p 44:22 niallo/strider
I was not able to access the dashboard through localhost
Second attempt: boot2docker ip
I did some googling and found these OSX specific instructions. Including the boo2docker ip:
curl $(boot2docker ip 2> /dev/null):3000
This succeeds in getting the dashboard html.
Outstanding problem
However in the browser the html asset loads, but the front end assets scripts/app.js and styles/style.css are all broken links.
curl $(boot2docker ip 2> /dev/null):3000/styles/styles.css
Note: all the other assets are fine
Does anyone have any insight? I really wanna play with docker!
More details
Info
bash-3.2$ docker info
Containers: 9
Images: 29
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Dirs: 47
Execution Driver: native-0.2
Kernel Version: 3.16.7-tinycore64
Operating System: Boot2Docker 1.4.1 (TCL 5.4); master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014
CPUs: 4
Total Memory: 1.961 GiB
Name: boot2docker
ID: DA3Y:GVFJ:6NO7:FFNL:RNLW:2QXY:UV3F:YWAS:OBFF:42YG:LRU7:CBHV
Debug mode (server): true
Debug mode (client): false
Fds: 18
Goroutines: 17
EventsListeners: 0
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Processes
IMAGE COMMAND PORTS
niallo/strider:latest "/usr/bin/supervisor 0.0.0.0:44->22/tcp, 0.0.0.0:3000->3000/tcp, 0.0.0.0:27000->27017/tcp
All the docker links in tutorial you have linked are broken. I can't even find the source dockerfile for this build and hence don't recommend using it and can't really help you debug.
However, The Strider github page recommends using docker-strider image for running strider in docker. The instructions seem straight forward but you can ask another question on SO if you get stuck.