I'm trying to tar a pretty big folder (~11GB) and while taring, my VM crashes because its disk is full. But... I still have plenty of room available on all disks but /
$ sudo df -h
File system Size Used Avail. Used% Mount on
udev 3,9G 0 3,9G 0% /dev
tmpfs 799M 9,3M 790M 2% /run
/dev/sda1 9,1G 3,1G 5,6G 36% /
/dev/sda2 69G 37G 29G 57% /home
/dev/sdb1 197G 87G 100G 47% /docker
I assume tar is buffering somewhere on / and fulfil it before my OS crashes. By the way, I have no idea on how to prevent this. Do you guy have any idea?
Cheers,
Olivier
Tar normally builds the archive in the current directory, as a hidden file. Try cd'ing to one of your larger partition mounting points and taring from there to see if it makes a difference. You may also be running out of innodes:
No Space Left on Device, Running out of Innodes
I ran into a similar problem with a server because of too many small files. While you have plenty of free space left, you might run into this issue.
Related
I am running a code optimizer for gzip.c in docker, in this process overlay grows infinitely large that eat up my disk.
root#id17:/var/lib/docker/overlay2/fe6987bf6e686e771ba7b08cda40aa477979512e182ad30120db037024638aa0# df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/sda5 245G 245G 0 100% /
...
overlay 245G 245G 0 100% /var/lib/docker/overlay2/fe6987bf6e686e771ba7b08cda40aa477979512e182ad30120db037024638aa0/merged
By using du -h --max-depth=1 I find it is diff and merged that consumed up my disk(is it?)
root#id17:/var/lib/docker/overlay2/fe6987bf6e686e771ba7b08cda40aa477979512e182ad30120db037024638aa0# du -h --max-depth=1
125G ./diff
129G ./merged
8.0K ./work
254G .
However, when I restart the dockersystemctl restart docker, it returned to normal.
root#eb9bf52aa3a3:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 245G 190G 43G 82% /
...
/dev/sda5 245G 190G 43G 82% /etc/hosts
...
root#id17:/var/lib/docker/overlay2/fe6987bf6e686e771ba7b08cda40aa477979512e182ad30120db037024638aa0# du -h --max-depth=1
125G ./diff
129G ./merged
8.0K ./work
254G .
It has come out for times and I cannot continue to do my work. So I really wonder how can I get out from this problem. Really thank you:-)
If the docker filesystem is growing, that often indicates container logs, or filesystem changes in the container. Logs you can see with docker logs and filesystem changes are shown with docker diff. Since you see a large diff folder, it's going to be the latter.
Those filesystem changes will survive a restart of the container, they get cleaned when the container is removed and replaced with a new container. So if restarting the container resolves it, my suspicion is your application is deleting the files on disk, but still has the file handles open to the kernel, possibly still writing to those file handles.
The other option is the stop or start of your application is deleting the files.
I have a server where I run some containers with volumes. All my volumes are in /var/lib/docker/volumes/ because docker is managing it. I use docker-compose to start my containers.
Recently, I tried to stop one of my container but it was impossible :
$ docker-compose down
[17849] INTERNAL ERROR: cannot create temporary directory!
So, I checked how the data is mounted on the server :
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7,8G 0 7,8G 0% /dev
tmpfs 1,6G 1,9M 1,6G 1% /run
/dev/md3 20G 19G 0 100% /
tmpfs 7,9G 0 7,9G 0% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 7,9G 0 7,9G 0% /sys/fs/cgroup
/dev/md2 487M 147M 311M 33% /boot
/dev/md4 1,8T 1,7G 1,7T 1% /home
tmpfs 1,6G 0 1,6G 0% /run/user/1000
As you can see, the / is only 20Go, so it is full and I can't stop my containers using docker-compose.
My questions are :
There is a simple solution to increase the available space in the
/, using /dev/md4 ?
Or can I move volumes to another place without losing data ?
This part of the Docker Daemon is confirgurable. Best practices would have you change the data folder; this can be done with OS-level Linux commands like a symlink... I would say it's better to actually configure the Docker Daemon to store the data elsewhere!
You can do that by editing the Docker command line (e.g. the systemd script that starts the Docker daemon), or change /etc/docker/daemon.json.
The file should have this content:
{
"data-root": "/path/to/your/docker"
}
If you add a new hard drive, partition, or mount point you can add it here and docker will store its data there.
I landed here as I had the very same issue. Even though some sources suggest you could do it with a symbolic link this will cause all kinds of issues.
Depending on the OS and Docker version I had malformed images, weird errors or the docker-daemon refused to start.
Here is a solution, but it seems it varies a little from version to version. For me the solution was:
Open
/lib/systemd/system/docker.service
And change this line
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
to:
ExecStart=/usr/bin/dockerd -g /mnt/WHATEVERYOUR/PARTITIONIS/docker --containerd=/run/containerd/containerd.sock
I solved it creating a symbolic link to a partition with bigger size:
ln -s /scratch/docker_meta /var/lib/docker
/scratch/docker_meta is the folder that I have in a bigger partition.
Do a bind mount.
For example, moving /docker/volumes to /mnt/large.
Append line into /etc/fstab.
/mnt/large /docker/volumes none bind 0 0
And then.
mv /docker/volumes/* /mnt/large/
mount /docker/volumes
Do not forget chown and chmod of /mnt/large first, if you are using non-root docker.
I changed Docker's storage base directory from /var/lib/docker to /home/docker by changing DOCKER_OPTIONS in /etc/default/docker as explained in this other question. After that, I rsynced the old /var/lib/docker to the new place.
Here is my Docker configuration file:
# Docker Upstart and SysVinit configuration file
# ....
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /home/docker"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
Everything was working fine after I rebooted. However, I started getting a "no space left on device" in my containers from time to time. When this error happens, if my container is up, I can't even do a mkdir. If the container is down and I try to start it, I get the following:
Error response from daemon: rpc error: code = 2 desc = "oci runtime
error: could not synchronise with container process: can't create
pivot_root dir , error mkdir .pivot_root: no space left on device"
However, I have space:
Filesystem Size Used Avail Use% Mounted on
udev 32G 4,0K 32G 1% /dev
tmpfs 6,3G 1,6M 6,3G 1% /run
/dev/sda1 92G 56G 32G 64% /
none 4,0K 0 4,0K 0% /sys/fs/cgroup
none 5,0M 0 5,0M 0% /run/lock
none 32G 472K 32G 1% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda5 1,6T 790G 762G 51% /home
I'm suspecting that perhaps I haven't done the storage migration correctly. Does someone know what might be happening?
Running out of disk space can also include inode limits. You can check those with df -i. This post on Unix.SE walks you through the steps required to increase the number of inodes available. Short of that, you can delete files to free up the inodes.
You can try cleaning up images that aren't in use. This fixed the problem for me:
docker images -aq -f 'dangling=true' | xargs docker rmi
As well as volumes. This will remove dangling volumes:
docker volume ls -q -f 'dangling=true' | xargs docker volume rm
https://success.docker.com/article/error-message-no-space-left-on-device-in-default-machine
My hard disk is getting full and I'm suspecting that my Docker container may not have enough disk space.
How can I check that the system allocated enough free disk space for Docker?
My OS is OSX.
Docker for Mac's data is all stored in a VM which uses a thin provisioned qcow2 disk image. This image will grow with usage, but never automatically shrink. (which may be fixed in 1.13)
The image file is stored in your home directories Library area:
mac$ cd ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
mac$ ls -l Docker.qcow2
rw-r--r-- 1 user staff 46671265792 31 Jan 22:24 Docker.qcow2
Inside the VM
Attach to the VM's tty with screen (brew install screen if you don't have it)
$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
If you get a login prompt, user is root with no password. Otherwise just press enter. Then you can run the df commands on the Linux VM.
/ # df -h /var/lib/docker
Filesystem Size Used Available Use% Mounted on
/dev/vda2 59.0G 14.9G 41.1G 27% /var
Note that this matches the df output inside a container (when using aufs or overlay)
mac$ docker run debian df -h
Filesystem Size Used Avail Use% Mounted on
overlay 60G 15G 42G 27% /
tmpfs 1.5G 0 1.5G 0% /dev
tmpfs 1.5G 0 1.5G 0% /sys/fs/cgroup
/dev/vda2 60G 15G 42G 27% /etc/hosts
shm 64M 0 64M 0% /dev/shm
Also note that while the VM is only using 14.9G of the 60G, the file size is 43G.
mac$ du -h Docker.qcow2
43G Docker.qcow2
The easiest way to fix the size is to backup any volume data, "Reset" docker from the Preferences menu and start again. It appears 1.13 has resolved the issue and will run a compaction on shutdown.
screen notes
Exit the screen session with ctrl-a then d
The Docker VM's tty get's messed up after I exit screen and I have to restart Docker to get a functional terminal back for a new session.
I'm trying to make some C code with a simple gcc command in Ubuntu 10, but for some reason, I keep getting an error:
Cannot create temporary file in /tmp/: No space left on device
The thing is, though, I have plenty of space on the disk. Here is output of df -h:
Filesystem Size Used Avail Use% Mounted on
/ 3.7G 2.4G 1.1G 70% /
devtmpfs 312M 112K 312M 1% /dev
none 312M 24K 312M 1% /dev/shm
none 312M 80K 312M 1% /var/run
none 312M 0 312M 0% /var/lock
none 312M 0 312M 0% /lib/init/rw
And df -i, in case you are wondering about the inodes:
Filesystem Inodes IUsed IFree IUse% Mounted on
/ 240960 195198 45762 82% /
devtmpfs 79775 609 79166 1% /dev
none 79798 3 79795 1% /dev/shm
none 79798 41 79757 1% /var/run
none 79798 2 79796 1% /var/lock
none 79798 1 79797 1% /lib/init/rw
I can also touch /tmp/test successfully, so I know I have space on the drive. Any ideas as to why gcc decided to throw a fit all of a sudden? (It was working earlier) Thanks beforehand.
It looks to me that your /tmp directory is actually mounted as a devtmpfs which if I remember correctly is actually your computer's RAM.
You can always reboot and see if that helps, increase your virtual memory partition, or you can close running programs to see if that helps. Additionally, you can maybe delete some unnecessary files from /tmp as they are volatile to at least the life of the session.
The intermediate files are too big for /tmp, so perhaps using another temporary directory (TMPDIR=/var/tmp g++ ...) helps.