I want to get all disk devices of host machine in Go or C++ language in a docker container. More information such as free spaces are also needed. What should I do or can this be possible ?
There is nothing special about Go or C++ that is required. You can use any relevant code or libraries that would examine Linux system devices for disk space or free space, because the environment the docker container provides is (typically) a Linux environment.
Docker Solution
docker run --privileged <image> <program> will populate the /dev file system in the container, which contains the device files relevant to your system and allows the container to access those devices.
User Solution
You will have to tell your users, e.g. in DockerHub documentation, or in error messages, to use the
--privileged flag
when running your image or it won't be able to access system devices.
You should expect some scrutiny or cynicism from some of your more knowledgeable users.
Like: why does it need that?
Details
According to Luc Juggery's blog on Medium:
Purpose of the --privileged flag
Running a container with the --privileged flag gives all the capabilities to the container and also access to the host’s devices (everything that is under the /dev >folder)...
However, he confuses the issue for beginners a bit by running docker from vagrant.
He also warns us:
If you use the --privileged flag when running a container, make sure you know what you are doing.
And I agree with that completely. Using --privileged gives the container the permission to modify the host.
It is easier to see what is happening from a Linux host running docker.
Example 1:
From the Linux host we will start an ubuntu container (without --privileged) and run sfdisk to see the disk partitions and ls -l /dev/s* to see the disk devices. It doesn't work because the container has no privileges to access the host in this way. The container's environment can not scan the disks on the host in any way.
paul#somewhere:~$ docker run -it ubuntu /bin/bash
root#175db156cb32:/# sfdisk --list
(blank output)
root#175db156cb32:/# ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
Example 2:
Now we run docker run --privileged
paul#somewhere:~$ docker run --privileged -it ubuntu /bin/bash
root#c62b42161444:/# sfdisk --list
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EE70993B-4640-4899-B142-18B89DD16CB8
Device Start End Sectors Size Type
/dev/sda1 2048 923647 921600 450M Windows recovery environment
/dev/sda2 923648 1128447 204800 100M EFI System
/dev/sda3 1128448 1161215 32768 16M Microsoft reserved
/dev/sda4 1161216 467810878 466649663 222.5G Microsoft basic data
/dev/sda5 467812352 468858879 1046528 511M Windows recovery environment
Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 2F514662-72A3-4126-9868-40CEB6ADA416
Device Start End Sectors Size Type
/dev/sdb1 34 262177 262144 128M Microsoft reserved
/dev/sdb2 264192 5860532223 5860268032 2.7T Microsoft basic data
Partition 1 does not start on physical sector boundary.
Disk /dev/sdc: 232.9 GiB, 250059350016 bytes, 488397168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x299c6114
Device Boot Start End Sectors Size Id Type
/dev/sdc1 * 2048 89843711 89841664 42.9G 83 Linux
/dev/sdc2 89843712 480468991 390625280 186.3G 83 Linux
/dev/sdc3 480471038 488396799 7925762 3.8G 5 Extended
/dev/sdc5 480471040 488396799 7925760 3.8G 82 Linux swap / Solaris
root#c62b42161444:/# ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Aug 11 02:43 /dev/sda
brw-rw---- 1 root disk 8, 1 Aug 11 02:43 /dev/sda1
brw-rw---- 1 root disk 8, 2 Aug 11 02:43 /dev/sda2
brw-rw---- 1 root disk 8, 3 Aug 11 02:43 /dev/sda3
brw-rw---- 1 root disk 8, 4 Aug 11 02:43 /dev/sda4
brw-rw---- 1 root disk 8, 5 Aug 11 02:43 /dev/sda5
brw-rw---- 1 root disk 8, 16 Aug 11 02:43 /dev/sdb
brw-rw---- 1 root disk 8, 17 Aug 11 02:43 /dev/sdb1
brw-rw---- 1 root disk 8, 18 Aug 11 02:43 /dev/sdb2
brw-rw---- 1 root disk 8, 32 Aug 11 02:43 /dev/sdc
brw-rw---- 1 root disk 8, 33 Aug 11 02:43 /dev/sdc1
brw-rw---- 1 root disk 8, 34 Aug 11 02:43 /dev/sdc2
brw-rw---- 1 root disk 8, 35 Aug 11 02:43 /dev/sdc3
brw-rw---- 1 root disk 8, 37 Aug 11 02:43 /dev/sdc5
root#c62b42161444:/# exit
and the docker container is allowed to access the host devices.
Related
I am trying to set upper write throughput limit per cgroup via blkio cgroup controller.
I have tried it like this:
echo "major:minor 10485760" > /sys/fs/cgroup/blkio/docker/XXXXX/blkio.throttle.write_bps_device
This should limit throughput to 10 MBps. However tool, that's monitoring servers disk, reports this behaviour.
I thought that, the line should hold somewhere around 10M. Can somebody explain this behaviour to me and maybe propose a better way to limit throughput?
Are you sure that the major/minor numbers that you specified in the command line are correct? Moreover, as you are running in docker, the limitation is for the processes running in the docker container not for the processes running outside. So, you need to check from where the information taken by the monitoring tool come from (does it take numbers for all the processes inside and outside the container or only for the processes inside the container?).
To check the setting, the Linux documentation provides an example with the dd command and a device limited to 1MB/second on reads. You can try the same with a limit on the writes to see if the monitoring tool is coherent with the output of dd. Make the latter run in the container.
For example, my home directory is located on /dev/sdb2:
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
[...]
/dev/sdb2 2760183720 494494352 2125409664 19% /home
[...]
$ ls -l /dev/sdb*
brw-rw---- 1 root disk 8, 16 mars 14 08:14 /dev/sdb
brw-rw---- 1 root disk 8, 17 mars 14 08:14 /dev/sdb1
brw-rw---- 1 root disk 8, 18 mars 14 08:14 /dev/sdb2
I check the speed of the writing in a file:
$ dd oflag=direct if=/dev/zero of=$HOME/file bs=4K count=1024
1024+0 records in
1024+0 records out
4194304 bytes (4,2 MB, 4,0 MiB) copied, 0,131559 s, 31,9 MB/s
I set the 1MB/s write limit on the whole disk (8:16) as it does not work on individual partitions (8:18) on which my home directory resides:
# echo "8:16 1048576" > /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device
# cat /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device
8:16 1048576
dd's output confirms the limitation of the I/O throughput to 1 MB/s:
$ dd oflag=direct if=/dev/zero of=$HOME/file bs=4K count=1024
1024+0 records in
1024+0 records out
4194304 bytes (4,2 MB, 4,0 MiB) copied, 4,10811 s, 1,0 MB/s
So, it is possible to make the same in a container.
I got some application which will call the pvcreate each time.
I can see the volumes in my vm as follow:
$ pvscan
PV /dev/vda5 VG ubuntu-vg lvm2 [99.52 GiB / 0 free]
Total: 1 [99.52 GiB] / in use: 1 [99.52 GiB] / in no VG: 0 [0 ]
$ pvcreate --metadatasize=128M --dataalignment=256K '/dev/vda5'
Can't initialize physical volume "/dev/vda5" of volume group "ubuntu-vg" without -ff
$ pvcreate --metadatasize=128M --dataalignment=256K '/dev/vda5' -ff
Really INITIALIZE physical volume "/dev/vda5" of volume group "ubuntu-vg" [y/n]? y
Can't open /dev/vda5 exclusively. Mounted filesystem?
I have also tried wipsfs and observed the same result for above commands
$ wipefs -af /dev/vda5
/dev/vda5: 8 bytes were erased at offset 0x00000218 (LVM2_member): 4c 56 4d 32 20 30 30 31
How can I execute pvcreate?
Anything to be added for my vm?
It seems your hdd (/dev/vda5) is already been used in your ubuntu-vg. I think you can not use same hdd partition in 2 different PV's. or you can not add it again.
I'm trying to us cos to run some services on GCP.
One of the issues I'm seeing currently is that the VMs I've started very quickly seem to run out of inodes for the /var/lib/docker filesystem. I'd have expected this to be one of the things tuned in a container optimized os?
wouter#nbwm-cron ~ $ df -hi
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 78K 13K 65K 17% /
devtmpfs 463K 204 463K 1% /dev
tmpfs 464K 1 464K 1% /dev/shm
tmpfs 464K 500 463K 1% /run
tmpfs 464K 13 464K 1% /sys/fs/cgroup
tmpfs 464K 9 464K 1% /mnt/disks
tmpfs 464K 16K 448K 4% /tmp
/dev/sda8 4.0K 11 4.0K 1% /usr/share/oem
/dev/sda1 1013K 998K 15K 99% /var
tmpfs 464K 45 464K 1% /var/lib/cloud
overlayfs 464K 39 464K 1% /etc
wouter#nbwm-cron ~ $ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<name>/stackdriver-agent latest 0c4b075e7550 3 days ago 1.423 GB
<none> <none> 96d027d3feea 4 days ago 905.2 MB
gcr.io/<project>/nbwm-ops/docker-php5 latest 5d2c59c7dd7a 2 weeks ago 1.788 GB
nbwm-cron wouter # tune2fs -l /dev/sda1
tune2fs 1.43.3 (04-Sep-2016)
Filesystem volume name: STATE
Last mounted on: /var
Filesystem UUID: ca44779b-ffd5-405a-bd3e-528071b45f73
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Remount read-only
Filesystem OS type: Linux
Inode count: 1036320
Block count: 4158971
Reserved block count: 0
Free blocks: 4062454
Free inodes: 1030756
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Reserved GDT blocks: 747
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8160
Inode blocks per group: 510
Flex block group size: 16
Filesystem created: Thu Jun 15 22:39:33 2017
Last mount time: Wed Jun 28 13:51:31 2017
Last write time: Wed Jun 28 13:51:31 2017
Mount count: 5
Maximum mount count: -1
Last checked: Thu Nov 19 19:00:00 2009
Check interval: 0 (<none>)
Lifetime writes: 67 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 66aa0e7f-57da-41d0-86f7-d93270e53030
Journal backup: inode blocks
How do I tune the filesystem to have more inodes available?
This is a known issue with the overlay storage driver in docker and is addressed by the overlay2 driver.
The new cos-61 releases use docker 17.03 with overlay2 storage driver. Could you please give it a try and see if the issue happens again?
Thanks!
I have witnessed the same issue with all COS versions from 57.9202.64.0 (docker 1.11.2) on GKE 1.5 to 65.10323.85.0 (docker 17.03.2) on GKE 1.8.12-gke.1. Older version were certainly affected too.
Those all use the overlay driver:
pdecat#gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ docker info 2>&1 | grep "Storage Driver"
Storage Driver: overlay
pdecat#gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ grep "\(CHROMEOS_RELEASE_VERSION\|CHROMEOS_RELEASE_CHROME_MILESTONE\)" /etc/lsb-release
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_VERSION=10323.85.0
The overlay2 driver is only used for GKE 1.9+ clusters (fresh or upgraded) with the same COS version:
pdecat#gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ docker info 2>&1 | grep "Storage Driver"
Storage Driver: overlay2
pdecat#gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ grep "\(CHROMEOS_RELEASE_VERSION\|CHROMEOS_RELEASE_CHROME_MILESTONE\)" /etc/lsb-release
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_VERSION=10323.85.0
When the free space/inodes issue occurs with the overlay driver, I resolve it using spotify's docker-gc:
# docker run --rm --userns host -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc
Before:
# df -hi /var/lib/docker/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6.0M 5.0M 1.1M 83% /var
# df -h /var/lib/docker/
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 95G 84G 11G 89% /var
# du --inodes -s /var/lib/docker/*
180 /var/lib/docker/containers
4093 /var/lib/docker/image
4 /var/lib/docker/network
4906733 /var/lib/docker/overlay
1 /var/lib/docker/tmp
1 /var/lib/docker/trust
25 /var/lib/docker/volumes
After:
# df -hi /var/lib/docker/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6.0M 327K 5.7M 6% /var/lib/docker
# df -h /var/lib/docker/
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 95G 6.6G 88G 7% /var/lib/docker
# du --inodes -s /var/lib/docker/*
218 /var/lib/docker/containers
1792 /var/lib/docker/image
4 /var/lib/docker/network
279002 /var/lib/docker/overlay
1 /var/lib/docker/tmp
1 /var/lib/docker/trust
25 /var/lib/docker/volumes
Note: using the usual docker rmi $(docker images --filter "dangling=true" -q --no-trunc) and docker rm $(docker ps -qa --no-trunc --filter "status=exited") did not help to recover resources in /var/lib/docker/overlay.
Docker Version :17.04.0-ce
os :windows 7
I start container using the command :docker run -it -memory 4096MB <container-id>
check the memory using the command :docker stats --no-stream | grep <container-id>
the result is :
5fbc6df8f90f 0.23% 86.52 MB / 995.8 Mib 2.59% 648B / 0B 17.2G / 608 MB 31
when update the memory,the result is also the same:
$ docker update -m 4500MB --memory-swap 4500MB --memory-reservation 4500MB 5fbc6df8f90f
5fbc6df8f90f
$ docker stats --no-stream | grep 5fbc6df8f90f
5fbc6df8f90f 0.23% 86.52 MB / 995.8 Mib 2.59% 648B / 0B 17.2G / 608 MB 31
why "--memory" can not work ,the memory is always the same 995.8Mib?
The docker stats command is showing you how much memory the entire docker host has, or with D4W, how much memory you have in the Linux VM. To increase this threshold, go into the settings of Docker to change the memory allocated to the VM. See this documentation for more details.
I'm running my rethinkdb container in Kubernetes cluster. Below is what I notice:
Running top in the host which is CoreOS, rethinkdb process takes about 3Gb:
$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
981 root 20 0 53.9m 34.5m 20.9m S 15.6 0.4 1153:34 hyperkube
51139 root 20 0 4109.3m 3.179g 22.5m S 15.0 41.8 217:43.56 rethinkdb
579 root 20 0 707.5m 76.1m 19.3m S 2.3 1.0 268:33.55 kubelet
But running docker stats to check the rethinkdb container, it takes about 7Gb!
$ docker ps | grep rethinkdb
eb9e6b83d6b8 rethinkdb:2.1.5 "rethinkdb --bind al 3 days ago Up 3 days k8s_rethinkdb-3.746aa_rethinkdb-rc-3-eiyt7_default_560121bb-82af-11e5-9c05-00155d070266_661dfae4
$ docker stats eb9e6b83d6b8
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
eb9e6b83d6b8 4.96% 6.992 GB/8.169 GB 85.59% 0 B/0 B
$ free -m
total used free shared buffers cached
Mem: 7790 7709 81 0 71 3505
-/+ buffers/cache: 4132 3657
Swap: 0 0 0
Can someone explain why the container is taking a lot more memory than the rethinkdb process itself?
I'm running docker v1.7.1, CoreOS v773.1.0, kernel 4.1.5
In top command, your are looking at physical memory amount. in stats command, this also include the disk cached ram, so it's always bigger than the physical amount of ram. When you really need more RAM, the disk cached will be released for the application to use.
In deed, the memmory usage is pulled via cgroup memory.usage_in_bytes, you can access it in /sys/fs/cgroup/memory/docker/long_container_id/memory.usage_in_bytes. And acording to linux doc https://www.kernel.org/doc/Documentation/cgroups/memory.txt section 5.5:
5.5 usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some
optimization to avoid unnecessary cacheline false sharing.
usage_in_bytes is affected by the method and doesn't show 'exact'
value of memory (and swap) usage, it's a fuzz value for efficient
access. (Of course, when necessary, it's synchronized.) If you want to
know more exact memory usage, you should use RSS+CACHE(+SWAP) value in
memory.stat(see 5.2).