Rancher system started to use a heavy amount of disksspace. Kubernetes was setup by Rancher's RKE.
Diskusage is already over 5TB however I have only 10-12 replicaset, their real data is binded to PV which uses nfs (which has only a size of 10gb).
df -h --total clearly shows which one takes up so many space:
Filesystem Size Used Avail Use% Mounted on
overlay 98G 78G 16G 84% /var/lib/docker/overlay2/84db..somehash/merged
I have ~50-60 entry from these.
How can I cleanup these? Is there any maintenance feature in rancher for this? Couldn't find any though.
Kubernetes's garbage collection should be cleaning up your nodes.
This looks a lot an issue I saw with some log collectors like Splunk and Datadog.
If the following usage numbers do not match up. Then using the script below to release the file descriptors.
df -h /var/lib/docker
docker system df
Workaround:
ps aux | grep dockerd <<== This pid
cd /proc/`pid of dockerd`/fd
ls -l |grep var.log.journal |grep deleted.$ |awk '{print $9}' |while read x; do :> $x; done;
Related
I am running a code optimizer for gzip.c in docker, in this process overlay grows infinitely large that eat up my disk.
root#id17:/var/lib/docker/overlay2/fe6987bf6e686e771ba7b08cda40aa477979512e182ad30120db037024638aa0# df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/sda5 245G 245G 0 100% /
...
overlay 245G 245G 0 100% /var/lib/docker/overlay2/fe6987bf6e686e771ba7b08cda40aa477979512e182ad30120db037024638aa0/merged
By using du -h --max-depth=1 I find it is diff and merged that consumed up my disk(is it?)
root#id17:/var/lib/docker/overlay2/fe6987bf6e686e771ba7b08cda40aa477979512e182ad30120db037024638aa0# du -h --max-depth=1
125G ./diff
129G ./merged
8.0K ./work
254G .
However, when I restart the dockersystemctl restart docker, it returned to normal.
root#eb9bf52aa3a3:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 245G 190G 43G 82% /
...
/dev/sda5 245G 190G 43G 82% /etc/hosts
...
root#id17:/var/lib/docker/overlay2/fe6987bf6e686e771ba7b08cda40aa477979512e182ad30120db037024638aa0# du -h --max-depth=1
125G ./diff
129G ./merged
8.0K ./work
254G .
It has come out for times and I cannot continue to do my work. So I really wonder how can I get out from this problem. Really thank you:-)
If the docker filesystem is growing, that often indicates container logs, or filesystem changes in the container. Logs you can see with docker logs and filesystem changes are shown with docker diff. Since you see a large diff folder, it's going to be the latter.
Those filesystem changes will survive a restart of the container, they get cleaned when the container is removed and replaced with a new container. So if restarting the container resolves it, my suspicion is your application is deleting the files on disk, but still has the file handles open to the kernel, possibly still writing to those file handles.
The other option is the stop or start of your application is deleting the files.
I want to use loop devices in a docker container locally. It means, when running a couple of container all of them should have for instance a /dev/loop0 connected to a file local in the container. I tried
[root#600bbfb452d1 /]# mknod /dev/loop20 b 7 20
[root#600bbfb452d1 /]# dd if=/dev/random of=loopfile1 bs=1M count=2
[root#600bbfb452d1 /]# losetup -a | grep 20
/dev/loop20: [0049]:3553002 (/loopfile1)
so far so good. But going back to host I can see:
[loewe#linux-2 ~]$ losetup -a | grep 20
/dev/loop20: []: (/loopfile1)
the loop device /dev/loop20 was also created in the hosts /dev - as my fear was because of the tmpfs mount - and worst the container local file "loopfile1" is attached to hosts loop dev.
I tries to umount the /dev filesystem in the container but didn't succeed (device busy but no proc visible with lsof).
Any idea what I am doing wrong?
BTW: using iscsi devices in a container should have the same problem.
Thanks Heiko
I changed Docker's storage base directory from /var/lib/docker to /home/docker by changing DOCKER_OPTIONS in /etc/default/docker as explained in this other question. After that, I rsynced the old /var/lib/docker to the new place.
Here is my Docker configuration file:
# Docker Upstart and SysVinit configuration file
# ....
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /home/docker"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
Everything was working fine after I rebooted. However, I started getting a "no space left on device" in my containers from time to time. When this error happens, if my container is up, I can't even do a mkdir. If the container is down and I try to start it, I get the following:
Error response from daemon: rpc error: code = 2 desc = "oci runtime
error: could not synchronise with container process: can't create
pivot_root dir , error mkdir .pivot_root: no space left on device"
However, I have space:
Filesystem Size Used Avail Use% Mounted on
udev 32G 4,0K 32G 1% /dev
tmpfs 6,3G 1,6M 6,3G 1% /run
/dev/sda1 92G 56G 32G 64% /
none 4,0K 0 4,0K 0% /sys/fs/cgroup
none 5,0M 0 5,0M 0% /run/lock
none 32G 472K 32G 1% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda5 1,6T 790G 762G 51% /home
I'm suspecting that perhaps I haven't done the storage migration correctly. Does someone know what might be happening?
Running out of disk space can also include inode limits. You can check those with df -i. This post on Unix.SE walks you through the steps required to increase the number of inodes available. Short of that, you can delete files to free up the inodes.
You can try cleaning up images that aren't in use. This fixed the problem for me:
docker images -aq -f 'dangling=true' | xargs docker rmi
As well as volumes. This will remove dangling volumes:
docker volume ls -q -f 'dangling=true' | xargs docker volume rm
https://success.docker.com/article/error-message-no-space-left-on-device-in-default-machine
I have a container running and by default it uses 10 GB of space. Last night the container space was filled by the log files generated by the system. Since log file grew to 8 GB, I emptied the log file but still my container is 100% disk full. It never released the 8GB space cleared from the log file. Any idea?
root#c7:/app# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-202:1-264176-9aff6 10G 10G 20K 100% /
root#c7:/app# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/docker-202:1-264176-9aff6 68368 67605 763 99% /
Thanks,
Manish Joshi
May be, you can try running this command on host
fstrim /proc/$(docker inspect --format='{{ .State.Pid }}' <cid>)/root
I have a docker running and it gives me disk space warning. How can i increase the docker space and start again? (The same container)
Lets say I want to give like 15gb.
You can also increase disk space through the docker GUI
I assume you are talking about disk space to run your containers.
Make sure that you have enough space on whatever disk drive you are using for /var/lib/docker which is the default used by Docker. You can change it with the -g daemon option.
If you don't have enough space you may have to repartition your OS drives so that you have over 15GB. If you are using boot2docker or docker-machine you will have to grow the volume on your Virtual Machine. It will vary depending on what you are using for Virtualization (i.e VirtualBox, VMware, etc)
For example if you are using VirtualBox and docker-machine you can start with something like this for a 40GB VM.
docker-machine create --driver virtualbox --virtualbox-disk-size "40000" default
I ran into similar problem with my docker-vm (which is 'alpine-linux' on VMware Fusion in OS X):
write error: no space left on device alpinevm:/mnt/hgfs
failed to build: .. no space left on device
.. eventually this guide helped me to resize/expand my docker volume.
TL;DR:
1 - Check size of partition containing /var/lib/docker
> df -h
/dev/sda3 17.6G 4.1G 12.6G 25% /var/lib/docker
look for '/dev/sdaN', where N is your partition for '/var/lib/docker', in my case /dev/sda3
2 - Shut down your VM, open VM Settings > Hard Disk(s) > change size of your 'virtual_disk.vmdk' (or whatever is your machine's virtual disk), then click Apply (see this guide).
3 - Install cfdisk and e2fsprogs-extra which contains resize2fs
> apk add cfdisk
> apk add e2fsprogs-extra
4 - Run cfdisk and resize/expand /dev/sda3
> cfdisk
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 206847 204800 100M 83 Linux
/dev/sda2 206848 4241407 4034560 1.9G 82 Linux swap / Solaris
/dev/sda3 4241408 83886079 79644672 12.6G 83 Linux
[Bootable] [ Delete ] [ Resize ] [ Quit ] [ Type ] [ Help ] [ Write ] [ Dump ]
.. press down/up to select '/dev/sda3'
.. press left/right/enter to select 'Resize' -> 'Write' -> 'Quit'
5 - Run resize2fs to expand the file system of /dev/sda3
> resize2fs /dev/sda3
6 - Verify resized volume
> df -h
/dev/sda3 37.3G 4.1G 31.4G 12% /var/lib/docker
To increase space available for Docker you will have to increase your docker-pool size. If you do a
lvs
You will see the docker-pool logical volume and its size. If your docker pool is sitting on a volume group that has free space you can simply increase the docker-pool LV by
lvextend -l 100%FREE <path_to_lv>
# An example using this may looks like this:
# lvextend -l 100%FREE /dev/VolGroup00/docker-pool
You can check out more docker diskspace tips here
Thanks
Docker stores all layers/images in its file formate (i.e. aufs) in default /var/lib/docker directory.
If you are getting disk space warning because of docker then there must of lot of docker images and you need to clean up it.
If you have option to add disk space then can you create separate partition with bigger size and mount your /var/lib/docker over there which will help you to get rid of filling root partition.
some extra information can be found here on managing disk space for docker .
http://www.scmtechblog.net/2016/06/clean-up-docker-images-from-local-to.html