I'm running Kubernetes/Docker on Google Container Optimized OS on a GCE instance. When I run docker info it says
$ docker info
Containers: 116
Running: 97
Paused: 0
Stopped: 19
Images: 8
Server Version: 1.11.2
Storage Driver: overlay
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.4.21+
Operating System: Container-Optimized OS from Google
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.67 GiB
Name: REDACTED
ID: REDACTED
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
The last line says that there is no swap limit support. I'm having trouble figuring out how to enable swap limit support. I found instructions for Ubuntu/Debian here.
My problem is that my docker containers get OOMKilled as soon as they reach their memory limit instead of trying swapping. I want the containers to use swap as a buffer instead of dying immediately.
Container-Optimized OS (COS) is actually configured with swap disabled completely. You can verify this via running cat /proc/meminfo | grep SwapTotal in a COS VM, which will say that it is configured to 0 kB.
I'm not sure whether it's a good idea to enable swap in your environment, as it may cause more problems (e.g. disk IO starvation/slowdown, kernel hung) if you are using swap frequently.
But if you wanna try it out, these commands might help you (run all of them as root):
cos-swap / # sysctl vm.disk_based_swap=1
vm.disk_based_swap = 1
cos-swap / # fallocate -l 1G /var/swapfile
cos-swap / # chmod 600 /var/swapfile
cos-swap / # mkswap /var/swapfile
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=406d3dfc-3780-44bf-8add-d19a24fdbbbb
cos-swap / # swapon /var/swapfile
cos-swap / # cat /proc/meminfo | grep Swap
SwapCached: 0 kB
SwapTotal: 1048572 kB
SwapFree: 1048572 kB
Related
I have a server with a docker registry, and have pushed a lot of times a build the same :latest tag now my DD is full and I can't get how to diet it.
disk is full
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 48G 45G 397M 100% /
udev 10M 0 10M 0% /dev
tmpfs 794M 81M 713M 11% /run
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/dm-1 9.8G 56M 9.2G 1% /var/lib/docker/devicemapper/mnt/2e895760700ac3e1575e496a4ac6adde4de6129226febba8c0c3126af1655ad9
shm 64M 0 64M 0% /var/lib/docker/containers/5aa47e34d1b8be22deeae473729b4e587e6e4bfe7fb3e262eda891bad4b05042/shm
there is no dangling volume nor images
# docker volume ls -qf dangling=true
#
# docker images -f "dangling=true" -q
#
docker images
[root#kvm22:/etc/cron.daily] # docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
jwilder/nginx-proxy 0.5.0 72b65b5a6f38 4 weeks ago 248.4 MB
registry 2 c9bd19d022f6 11 weeks ago 33.27 MB
registry 2.5 c9bd19d022f6 11 weeks ago 33.27 MB
disk usage
# du -h -d 7 /var/lib/docker/volumes/
12K /var/lib/docker/volumes/24000fbe2e81da06924be8f7ce81e07101824036bca5f87d4d811f2a6f7bfa7b/_data
16K /var/lib/docker/volumes/24000fbe2e81da06924be8f7ce81e07101824036bca5f87d4d811f2a6f7bfa7b
42G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/blobs/sha256
42G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/blobs
5.9M /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/repositories/labor-prod
5.9M /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/repositories
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data
43G /var/lib/docker/volumes/registry_docker-registry-volume
43G /var/lib/docker/volumes/
Output of docker version:
# docker --version
Docker version 1.12.4, build 1564f02
Output of docker info:
# docker info
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 5
Server Version: 1.12.4
Storage Driver: devicemapper
Pool Name: docker-8:1-1184923-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 1.058 GB
Data Space Total: 107.4 GB
Data Space Available: 3.036 GB
Metadata Space Used: 2.142 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.145 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.90 (2014-09-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.873 GiB
Name: kvm22
ID: G6OC:EKKY:ER4W:3JVZ:25BI:FF2Y:YXVA:RZRR:WPAP:SASB:AJJA:DM6J
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Insecure Registries:
127.0.0.0/8
I had the same problem. I can't believe there's no ready solution for this. Anyway I hacked a tool together and it seems to work.
You can find it here: https://github.com/Richie765/docker-tools
Basically it uses a bash script to find out which manifests are untagged. Then deletes them through the registry API. Afterwards you can run a garbage collection to actually delete the data.
I'm sure the script isn't perfect. Any improvements are welcome!
Found another tool here.
https://github.com/burnettk/delete-docker-registry-image
Included a clean_old_version.py script.
I give it a try too.
In case anyone still has this problem:
This is the 'reset' way how I solved it:
You can stop and delete the registry
docker stop registry && docker rm -v registry
and restart it afterwards:
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Then you would have to rebuild your images localy and push them to the registry again.
I'm new to docker and trying to implement docker using chef on centos 7.1
Below is the basic code I wrote for installing, pulling centos image and creating container.
All the 3 tasks are executing successfully. Since the containers are in stop mode I tried to start manually by typing docker start containerid. When I checked docker ps I found the container is not started . I tried for several times but couldn't start the container.
Docker code using chef
docker_service 'default' do
action [:create, :start]
end
docker_image 'centos' do
action :pull
end
docker_container 'check2' do
repo 'centos'
action :create
end
Docker info:
Containers: 6
Running: 1
Paused: 0
Stopped: 5
Images: 3
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-8:1-523814-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 441.3 MB
Data Space Total: 107.4 GB
Data Space Available: 28.08 GB
Metadata Space Used: 1.159 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-229.4.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 6.807 GiB
ID: R24R:ORHY:XJQW:2HNI:U5TV:UGF7:B7VX:P6Z6:UHSR:YIMR:VGJT:4URU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Would you please help me
By default, the centos image will run /bin/bash which will exit immediately without a tty (-t) and standard input (-i) available.
Try running something in the container
docker_container 'check2' do
repo 'centos'
command 'top -b -d 5'
end
I've started a docker master with:
docker swarm init --advertise-addr <MANAGER-IP>
so, im trying to have my shell point to swarm master via:
eval $(docker-machine env --swarm <MANAGER-IP>)
but it's giving me an error: Host does not exists
docker info:
-bash-4.2$ docker info
Containers: 18
Running: 1
Paused: 0
Stopped: 17
Images: 20
Server Version: 1.12.0
Storage Driver: devicemapper
Pool Name: docker-253:1-25646-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.124 GB
Data Space Total: 107.4 GB
Data Space Available: 13.4 GB
Metadata Space Used: 5.071 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.142 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null overlay
Swarm: active
NodeID: 05szzy2z96ypgl5k21swggoil
Is Manager: true
ClusterID: a2wrfuga2tu4cm4k0lxxorqtm
Managers: 1
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot interval: 10000
Heartbeat tick: 1
Election tick: 3
Dispatcher:
Heartbeat period: 5 seconds
CA configuration:
Expiry duration: 3 months
Node Address: 10.193.46.89
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51 GiB
Name: scsor0004331002.rtp.openenglab.netapp.com
ID: T52U:6MWQ:XEDM:2TGH:ITLQ:YD6B:R3MR:MWF5:CFBM:G6PX:W4LG:6SR7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: eugenepark3
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Anyone know what i need to put for eval $(docker-machine env --swarm <MANAGER-IP>) so my compose can run on swarm cluster?
Im supposed to put master name but i dont know how to find it
-bash-4.2$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
05szzy2z96ypgl5k21swggoil * scsor0004331002.rtp.openenglab.netapp.com Ready Active Leader
59t110b0wjhitj1fr8erys600 scsor0004331003.rtp.openenglab.netapp.com Ready Active
dhm6utu2w3dw1to0zh3n71moq scsor0004331004.rtp.openenglab.netapp.com Ready Active
You're mixing up the container based swarm commands with the newer swarmkit based Swarm that's been embedded directly into the Docker CLI. With the new version of Swarm, docker-compose isn't directly supported, yet. Consider this a beta product that works well for a limited scope. You can try the experimental release of the docker engine which adds support for DAB files that are managed with the docker stack CLI. The DAB files are exported from docker-compose bundle and then imported into Docker. This feature is still very experimental and expected to change.
Without that, anything with docker-compose will only operate on a single docker engine since the swarm access is all done under a different docker service CLI interface.
If I kill my docker daemon process and then restart it, any containers that were running are now listed with Exited status and cannot be restarted using docker-compose as it will complain about the container name already being in use.
Docker containers can be started again using docker start but this could be hard when you've many containers.
Is there any way to restart the docker daemon, leave containers running (so as to not disrupt traffic) and have the daemon reattach to the containers?
uname -a:
Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
docker info:
Containers: 23
Running: 2
Paused: 0
Stopped: 21
Images: 16
Server Version: 1.11.1
Storage Driver: devicemapper
Pool Name: docker-253:0-1567975-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.738 GB
Data Space Total: 107.4 GB
Data Space Available: 28 GB
Metadata Space Used: 7.688 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.14 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 7.64 GiB
Name:
ID:
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username:
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Looks like this is something that will be addressed in 1.12:
https://github.com/docker/docker/issues/2658
Ok, let's start over after a bunch of investigation. Here is what we know:
Ubuntu host
15.04 on i7 3820 (quad 3.6) and Samsung 850 pro SSD 512gb SATA 6
$ docker info
Containers: 2
Images: 101
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-28-generic
Operating System: Ubuntu 15.04
CPUs: 8
Total Memory: 15.61 GiB
Name: camacho
ID: ZOYN:QGDO:UGMJ:TDDM:WEEM:ZEHJ:4OKB:V5WR:RGCL:NOKG:F5W5:SDEL
WARNING: No swap limit support
OSX host
10.10.5 on i7 (quad 2.7) and Apple SSD 512gb SD512E SATA 6 (2+ years old)
$ docker info
Containers: 3
Images: 185
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 191
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
CPUs: 8
Total Memory: 3.858 GiB
Name: dinghy
ID: PNNP:PI3E:CRUK:27RI:IPHW:HROF:NQA2:XKV6:VGCZ:WT7B:BZ7R:USWD
Debug mode (server): true
File Descriptors: 21
Goroutines: 54
System Time: 2015-09-24T19:16:01.715069994Z
EventsListeners: 1
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
provider=virtualbox
Observations
dockerized rspec running 2x+ faster on OSX
sysbench io results are terrible on Ubuntu
dockerized iozone results are as expected (ubuntu slightly faster)
hardware check complete, ubuntu host is on a SATA 6 cable and port
iozone results
docker run -it threadx/docker-ubuntu-iozone
$ iozone -R -l 5 -u 5 -r 4k -s 100m -F /home/f1 /home/f2 /home/f3 /home/f4 /home/f5 | tee -a /tmp/iozone_results.txt &
Summary
I'm dockerizing our test process because we need stability and concurrency. We've done a bunch of work to limit test times, and a 2x increase in times is a terrible step backwards.
Sysbench shows the ubuntu host performing 4x worse than OSX, and I have no understanding of that. dd and iozone tests show the ubuntu host to be performing as expected.
Question
Why is my rspec performance worse on the ubuntu host which has faster resources? Where should I investigate? Is sysbench io test an indicator or an anomaly? What are sources of bad performance in dockerized ubuntu?
The difference is PCIe vs SATA 6 interfaces. The latest generation of Macbook Pro PCIe is even faster.