I am experimenting docker and I plan to dockerize my project's CI infratsructure.
I am building dockerfiles on 3 different machines: Fedora, Ubuntu, and Boot2Docker (a virtiual machine under windows).
Docker builds on fedora are a lot slower than on the other 2 machines, especially the opration that creates a new image after a Step in the Dockerfile.
So my question is: Is device mapper a lot slower than AUFS or should I search for some other reason ? Should I expect better if I put in place a RHEL7 config ?
Config1: fedora 21 (3.18.3-201.fc21.x86_64)
sudo docker info
Containers: 27
Images: 1353
Storage Driver: devicemapper
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data Space Used: 82.77 GB
Data Space Total: 107.4 GB
Metadata Space Used: 103.9 MB
Metadata Space Total: 2.147 GB
Udev Sync Supported: true
Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 3.18.3-201.fc21.x86_64
Operating System: Fedora 21 (Twenty One)
CPUs: 8
Total Memory: 31.38 GiB
Config2: Ubuntu 14.04.2 LTS
Containers: 89
Images: 589
Storage Driver: aufs
Backing Filesystem: extfs
Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-49-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 31.38 GiB
Config3: Boot2Docker (virtual linux machine running under Windows. docker default solution for Windows)
docker info
Containers: 14
Images: 215
Storage Driver: aufs
Backing Filesystem: extfs
Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.18.11-tinycore64
Operating System: Boot2Docker 1.6.0 (TCL 5.4); master : a270c71 - Thu Apr
CPUs: 8
Total Memory: 1.961 GiB
I read this article but it still did not help me to clear things out.
Device Mapper's "slowness" has been documented, especially when used with a loop device.
Here's a useful presentation I found: http://jpetazzo.github.io/assets/2015-03-03-not-so-deep-dive-into-docker-storage-drivers.html.
I would look into overlay.
Related
When I want to rerun container with another volumes or update image.
I stop and try to remove container, but often geterror on rm command
# docker rm containername
Error response from daemon: Driver devicemapper failed to remove root filesystem dbe6....f91f: Device is Busy
I need to restart docker daemon to remove container.
~ # docker version root#CentOS-72-64-minimal
Client:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:23:59 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:23:59 2016
OS/Arch: linux/amd64
------------------------------------------------------------
~ # docker info root#CentOS-72-64-minimal
Containers: 40
Running: 11
Paused: 0
Stopped: 29
Images: 32
Server Version: 1.12.5
Storage Driver: devicemapper
Pool Name: docker-8:3-28705145-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 14.83 GB
Data Space Total: 107.4 GB
Data Space Available: 92.54 GB
Metadata Space Used: 21.15 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.126 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-09-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge overlay host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.2.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 30.96 GiB
Name: CentOS-72-64-minimal
ID: SMTY:72HJ:5QIS:AT63:6GPI:U2UQ:KUYY:C7M6:UIOY:37AR:JS53:JAGA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
I've been experiencing this issue quite a bit on a Red Hat host. The fix according to the reported issue on this is to upgrade to a newer kernel. As a workaround for places where that's not an option, I've been using docker rm -f ... which still throws the error but the container does get cleaned up. Much quicker and less intrusive than a restart of the daemon.
I experienced the same problem, service docker restart almost always fixes the issue.(restarting docker service)
More information about this issue availbale here.
I'm new to docker. I've been trying it for less than two weeks. We have a service that we'd like to migrate into a container. The service makes use of about 50G worth of data, so we expect the image to be very large. We've written a Dockerfile for it. When we run the build it fails with the ff:
ApplyLayer exit status 1 stdout: stderr: write /mnt/spine_features/spine_features_subset.lmdb/data.mdb: input/output error
When we check docker ps -a for containers, we can see the build container listed with status:
Exited (1) About a minute ago
When we try to commit the container, we get the same error:
Error response from daemon: ApplyLayer exit status 1 stdout: stderr: write /mnt/spine_features/spine_features_subset.lmdb/data.mdb: input/output error
We can also docker inspect the container. When we exclude downloading the largest files, we are able to complete building the service image. Is there some sort of configuration we can change succeed while still including the larger files?
docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.7
Git commit: 23cf638
Built: Fri Aug 19 02:03:02 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.7
Git commit: 23cf638
Built: Fri Aug 19 02:03:02 2016
OS/Arch: linux/amd64
docker info
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 22
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-8:2-7603782-pool
Pool Blocksize: 65.54 kB
Base Device Size: 214.7 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 67.78 GB
Data Space Total: 107.4 GB
Data Space Available: 39.59 GB
Metadata Space Used: 37.04 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.11 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.134 (2016-09-07)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.7.4-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 15.58 GiB
Name: mega-haro
ID: MDQ5:JIT3:BVQX:XYO6:YTXI:HTRE:N2UQ:ML4V:ENIE:DDCO:ZGYF:3P5F
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
misty:5000
127.0.0.0/8
As Haoming Zhang recommended, mounting the host directory into the container is an acceptable solution. We are also exploring the possibility of using FUSE to load the data into the container at runtime instead of baking it into the image during build or having the host pass it into the container when the container is run.
I'm new to docker and trying to implement docker using chef on centos 7.1
Below is the basic code I wrote for installing, pulling centos image and creating container.
All the 3 tasks are executing successfully. Since the containers are in stop mode I tried to start manually by typing docker start containerid. When I checked docker ps I found the container is not started . I tried for several times but couldn't start the container.
Docker code using chef
docker_service 'default' do
action [:create, :start]
end
docker_image 'centos' do
action :pull
end
docker_container 'check2' do
repo 'centos'
action :create
end
Docker info:
Containers: 6
Running: 1
Paused: 0
Stopped: 5
Images: 3
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-8:1-523814-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 441.3 MB
Data Space Total: 107.4 GB
Data Space Available: 28.08 GB
Metadata Space Used: 1.159 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-229.4.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 6.807 GiB
ID: R24R:ORHY:XJQW:2HNI:U5TV:UGF7:B7VX:P6Z6:UHSR:YIMR:VGJT:4URU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Would you please help me
By default, the centos image will run /bin/bash which will exit immediately without a tty (-t) and standard input (-i) available.
Try running something in the container
docker_container 'check2' do
repo 'centos'
command 'top -b -d 5'
end
If I kill my docker daemon process and then restart it, any containers that were running are now listed with Exited status and cannot be restarted using docker-compose as it will complain about the container name already being in use.
Docker containers can be started again using docker start but this could be hard when you've many containers.
Is there any way to restart the docker daemon, leave containers running (so as to not disrupt traffic) and have the daemon reattach to the containers?
uname -a:
Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
docker info:
Containers: 23
Running: 2
Paused: 0
Stopped: 21
Images: 16
Server Version: 1.11.1
Storage Driver: devicemapper
Pool Name: docker-253:0-1567975-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.738 GB
Data Space Total: 107.4 GB
Data Space Available: 28 GB
Metadata Space Used: 7.688 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.14 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 7.64 GiB
Name:
ID:
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username:
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Looks like this is something that will be addressed in 1.12:
https://github.com/docker/docker/issues/2658
Ok, let's start over after a bunch of investigation. Here is what we know:
Ubuntu host
15.04 on i7 3820 (quad 3.6) and Samsung 850 pro SSD 512gb SATA 6
$ docker info
Containers: 2
Images: 101
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-28-generic
Operating System: Ubuntu 15.04
CPUs: 8
Total Memory: 15.61 GiB
Name: camacho
ID: ZOYN:QGDO:UGMJ:TDDM:WEEM:ZEHJ:4OKB:V5WR:RGCL:NOKG:F5W5:SDEL
WARNING: No swap limit support
OSX host
10.10.5 on i7 (quad 2.7) and Apple SSD 512gb SD512E SATA 6 (2+ years old)
$ docker info
Containers: 3
Images: 185
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 191
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
CPUs: 8
Total Memory: 3.858 GiB
Name: dinghy
ID: PNNP:PI3E:CRUK:27RI:IPHW:HROF:NQA2:XKV6:VGCZ:WT7B:BZ7R:USWD
Debug mode (server): true
File Descriptors: 21
Goroutines: 54
System Time: 2015-09-24T19:16:01.715069994Z
EventsListeners: 1
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
provider=virtualbox
Observations
dockerized rspec running 2x+ faster on OSX
sysbench io results are terrible on Ubuntu
dockerized iozone results are as expected (ubuntu slightly faster)
hardware check complete, ubuntu host is on a SATA 6 cable and port
iozone results
docker run -it threadx/docker-ubuntu-iozone
$ iozone -R -l 5 -u 5 -r 4k -s 100m -F /home/f1 /home/f2 /home/f3 /home/f4 /home/f5 | tee -a /tmp/iozone_results.txt &
Summary
I'm dockerizing our test process because we need stability and concurrency. We've done a bunch of work to limit test times, and a 2x increase in times is a terrible step backwards.
Sysbench shows the ubuntu host performing 4x worse than OSX, and I have no understanding of that. dd and iozone tests show the ubuntu host to be performing as expected.
Question
Why is my rspec performance worse on the ubuntu host which has faster resources? Where should I investigate? Is sysbench io test an indicator or an anomaly? What are sources of bad performance in dockerized ubuntu?
The difference is PCIe vs SATA 6 interfaces. The latest generation of Macbook Pro PCIe is even faster.