Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
I can no longer generate new columns in my database. My stack is below. I am new to managing my own server and am unfamiliar with Apache, Ubuntu and AWS. Currently I am receiving an error saying that I have no space left.
It looks like my folder /dev/xvda1 is at 100% capacity. How can I identify how space is being allocated and how can I open up more space?
I will eagerly share more information if you can help me with what to share.
My Error
irb(main):107:0> Survey.generate_for_appointment(a, b)
PG::DiskFull: ERROR: could not extend file "base/16402/25415": No space left on device
HINT: Check free disk space.
: INSERT INTO "questions" ("choices", "content", "created_at", "position", "question_type", "survey_group", "survey_group_question_id", "survey_id", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING "id"
log writing failed. No space left on device # io_write - /etc/projects/myapp/log/production.log
ActiveRecord::StatementInvalid: PG::DiskFull: ERROR: could not extend file "base/16402/25415": No space left on device
HINT: Check free disk space.
Storage
ubuntu#ip-172-31-33-140:/etc/projects/myapp$ sudo du -h --max-depth=1 /
4.0K /media
752K /tmp
0 /sys
9.6M /bin
1.3G /home
908M /usr
4.0K /mnt
20M /boot
24K /root
4.0K /opt
3.7G /etc
du: cannot access ‘/proc/7489/task/7489/fd/4’: No such file or directory
du: cannot access ‘/proc/7489/task/7489/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/7489/fd/4’: No such file or directory
du: cannot access ‘/proc/7489/fdinfo/4’: No such file or directory
0 /proc
9.4M /sbin
4.0K /srv
16K /lost+found
60M /lib
12K /dev
649M /var
4.0K /lib64
212K /run
7.5G /
ubuntu#ip-172-31-33-140:/etc/projects/myapp$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 7.5G 0 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 284M 12K 284M 1% /dev
tmpfs 59M 212K 59M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 295M 0 295M 0% /run/shm
none 100M 0 100M 0% /run/user
Detailed Storage
ubuntu#ip-172-31-33-140:/etc/projects/myapp$ sudo du -x / | sort -n | tail -40
98880 /home/ubuntu/.rvm/rubies/ruby-2.1.2
98908 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.4.1
98908 /home/ubuntu/.rvm/rubies/ruby-2.1.5/lib
99436 /home/ubuntu/.rvm/rubies/ruby-2.1.5
107920 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp/x86_64-linux-gnu/ports
107924 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp/x86_64-linux-gnu
107928 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp
108520 /var/lib/apt/lists
108572 /var/lib/apt
111044 /usr/src
119420 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri
119424 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext
145780 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1
146972 /var/cache
155528 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/passenger-4.0.45/buildout
155916 /usr/lib/x86_64-linux-gnu
158396 /etc/projects/myapp/public/assets
163068 /etc/projects/myapp/public
198320 /home/ubuntu/.rvm/rubies
200824 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/passenger-4.0.45
207540 /home/ubuntu/.rvm/gems/ruby-2.1.5/gems
231732 /var/lib
264856 /home/ubuntu/.rvm/gems/ruby-2.1.5
271292 /var/log/apache2
284432 /var/log
292684 /usr/share
403300 /usr/lib
583604 /home/ubuntu/.rvm/gems/ruby-2.1.2/gems
650888 /home/ubuntu/.rvm/gems/ruby-2.1.2
664560 /var
915752 /home/ubuntu/.rvm/gems
929616 /usr
1174228 /home/ubuntu/.rvm
1262052 /home/ubuntu
1262056 /home
3470180 /etc/projects/myapp/log
3782520 /etc/projects/myapp
3782524 /etc/projects
3789104 /etc
7795236 /
Nodes in Use
ubuntu#ip-172-31-33-140:/etc/projects/myapp$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 524288 153278 371010 30% /
none 75346 2 75344 1% /sys/fs/cgroup
udev 72620 381 72239 1% /dev
tmpfs 75346 266 75080 1% /run
none 75346 3 75343 1% /run/lock
none 75346 1 75345 1% /run/shm
none 75346 2 75344 1% /run/user
Passanger Version
Phusion Passenger version 4.0.55
Apache Version
Server version: Apache/2.4.7 (Ubuntu)
RVM Version
rvm 1.26.4 (latest) by Wayne E. Seguin <wayneeseguin#gmail.com>, Michal Papis <mpapis#gmail.com> [https://rvm.io/]
Ruby Versions
rvm rubies
ruby-2.1.2 [ x86_64 ]
=* ruby-2.1.5 [ x86_64 ]
# => - current
# =* - current && default
# * - default
You only have 7.5G of total space. Your /etc folder is using up 3.7G, which seems like waaaay too much, and your home directory is 1.3G. Use du -sh * on each of these directories to find out what is using so much space in them and delete them if they are no longer needed. For example, items in your ~/Downloads folder, old logs in /var/log, or whatever is chewing up so much space in /etc.
Related
problem descriptions:
In my system, /var/lib/docker/overlay2/*/merged take too much space than it should be and the detail looks like below.
root#sriov-TaiShan-200:~# df -h --total
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 6.2M 13G 1% /run
/dev/nvme0n1p2 2.9T 2.4T 329G 89% /
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/loop0 128K 128K 0 100% /snap/bare/5
/dev/loop1 49M 49M 0 100% /snap/core18/2406
/dev/loop2 49M 49M 0 100% /snap/core18/2349
/dev/loop3 58M 58M 0 100% /snap/core20/1437
/dev/loop5 229M 229M 0 100% /snap/gnome-3-38-2004/100
/dev/loop6 73M 73M 0 100% /snap/lxd/23043
/dev/loop7 198M 198M 0 100% /snap/gnome-3-34-1804/73
/dev/nvme0n1p1 512M 5.2M 507M 1% /boot/efi
/dev/loop8 66M 66M 0 100% /snap/gtk-common-themes/1519
/dev/loop9 49M 49M 0 100% /snap/snap-store/545
/dev/loop10 229M 229M 0 100% /snap/gnome-3-38-2004/88
/dev/loop11 39M 39M 0 100% /snap/snapd/15541
/dev/loop14 198M 198M 0 100% /snap/gnome-3-34-1804/75
/dev/loop15 73M 73M 0 100% /snap/lxd/23003
/dev/loop16 53M 53M 0 100% /snap/snap-store/559
//10.67.93.97/work 916G 816G 100G 90% /mnt
/dev/loop17 82M 82M 0 100% /snap/gtk-common-themes/1534
/dev/loop13 39M 39M 0 100% /snap/snapd/15909
/dev/loop12 58M 58M 0 100% /snap/core20/1498
tmpfs 13G 16K 13G 1% /run/user/0
tmpfs 126G 12K 126G 1% /var/lib/kubelet/pods/12baa83c-6696-43f3-b060-85ba620867aa/volumes/kubernetes.io~projected/kube-api-access-j26q8
overlay 2.9T 2.4T 329G 89% /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89/merged
shm 64M 0 64M 0% /var/lib/docker/containers/9b02ed0d68aa3c24531b1e2852861a202b36b9d7d08645ee170035a3eab48747/mounts/shm
overlay 2.9T 2.4T 329G 89% /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d/merged
total 9.9T 8.0T 1.4T 86% -
root#sriov-TaiShan-200:~# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 2 2 109.7MB 0B (0%)
Containers 2 2 2.294kB 0B (0%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
root#sriov-TaiShan-200:~# du -h --max-depth=1 /var/lib/docker/overlay2 | sort -rh | head -25
223M /var/lib/docker/overlay2
111M /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d
68M /var/lib/docker/overlay2/6b7c00ae09c4df6f35e65114513bd6216c2436e62ae26c3b91b4c2ab60734662
42M /var/lib/docker/overlay2/94b81b09cc4a7ca3ead72888264e3cff25af77d2fc00334847aa2c3bde225e8a
1.6M /var/lib/docker/overlay2/65ca885064b7df65728ea2864d3f633f035d4b9f3348ae4bcdc8c926157b9552
528K /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89
488K /var/lib/docker/overlay2/5e344fbdc03c21013a38b1a82987d7e9094b5d13fe96d0f89f6e31a096f51c9c
48K /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89-init
36K /var/lib/docker/overlay2/l
32K /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d-init
Things i tried:
extend lv /dev/ubuntu-vg/ubuntu-lv size but no lucky
reinstall the docker with below steps but no lucky
apt-get purge docker-ce
rm -rf /var/lib/docker
rm -rf /etc/docker
apt-get remove docker docker-engine docker.io containerd runc
/etc/apt/sources.list.d/docker.list > /dev/null
apt update
mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
apt update
apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
docker system prune -a but no luck
root#sriov-TaiShan-200:~# docker system prune -a
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all images without at least one container associated to them
- all build cache
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
updates
It seems that docker wasn't using that much disk space. Through the du result, it is other things that take the space. However the overlay 2.9T 2.4T 329G 89% /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d/merged is quite confusing.
root#sriov-TaiShan-200:# du -h --max-depth=1 / --exclude=mnt --exclude=proc --exclude=home | sort -rh
2.3T /
1.6T /var
686G /root
15G /usr
6.1G /snap
3.6G /x86_64-linux-gnu
1.2G /boot
88M /opt
16M /etc
6.2M /run
92K /tmp
16K /lost+found
12K /media
4.0K /srv
4.0K /cdrom
0 /sys
0 /dev
The merged folders are overlay filesystems, they take no disk space themselves, instead df reports the disk usage of the underlying filesysem, which in your case is /:
/dev/nvme0n1p2 2.9T 2.4T 329G 89% /
overlay 2.9T 2.4T 329G 89% /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89/merged
overlay 2.9T 2.4T 329G 89% /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d/merged
The disk usage can be anywhere on the root filesystem, so you need to examine the entire drive to find where diskspace is being used. This may have nothing to do with docker.
extend lv /dev/ubuntu-vg/ubuntu-lv size but no lucky
Your root filesystem is listed as /dev/nvme0n1p2, so I'm not sure how changing the lv would affect that.
With containers, common places to use disk are:
image layers
container RW filesystem (from an app writing to the container filesystem)
logs of container output
The quick cleanup in docker is docker system prune. There are more options to that, and things that aren't included in the prune described in this answer.
I was using Docker on my CentOS machine for a while and had lot of images and containers (around 4GBs). My machine has 8GBs os storage and I kept getting an error from devicemapper whenever trying to remove a Docker container or Docker image with docker rm or docker rmi. The error was: Error response from daemon: Driver devicemapper failed to remove root filesystem. So I stopped the Docker service and tried restarting it, but that failed due to devicemapper. After that I uninstalled Docker and removed all images, containers, and volumes by running the following command: rm -rf /var/lib/docker. However, after running that it does not seem like any space was freed up:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 8.0G 7.7G 346M 96% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 193M 1.6G 11% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
tmpfs 361M 0 361M 0% /run/user/1000
$ du -ch -d 1 | sort -hr
3.6G total
3.6G .
1.7G ./usr
903M ./var
433M ./home
228M ./opt
193M ./run
118M ./boot
17M ./etc
6.4M ./tmp
4.0K ./root
0 ./sys
0 ./srv
0 ./proc
0 ./mnt
0 ./media
0 ./dev
Why does df tell me I am using 7.7G whereas du tells me I am using 3.6G? The figure that du gives (3.6G) should be the correct one since I deleted everything in /var/lib/docker.
I had a similar issue. This ticket was helpful.
Depending on the file system you are using, you will want to use either fstrim, zerofree or add the drive to another machine or and use use xfs_repair
If your file system is xfs and you used xfs_repair then after running that command there should be a lost+found directory at the root of the drive that contains all the data that was taking upspace but unreachable.
You can then delete that and it will actually be reflected in du.
I got a problem of space on a server vps with an application on rails in production environment with passenger and nginx.
I read a lot of similar problems but don't found my solution.
my errors can be
bash: cannot create temp file for here-document: No space left on device
My storage
admin#vps202702:~$ sudo du -h --max-depth=1 /
[sudo] password for admin:
206M /lib
12K /srv
12M /opt
4.0K /mnt
12M /bin
14M /root
20M /boot
2.0G /usr
6.4M /sbin
6.7G /var
4.0K /lib64
16K /lost+found
0 /sys
933M /home
5.3M /run
24K /tmp
du: cannot access ‘/proc/23910/task/23910/fd/4’: No such file or directory
du: cannot access ‘/proc/23910/task/23910/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/23910/fd/4’: No such file or directory
du: cannot access ‘/proc/23910/fdinfo/4’: No such file or directory
0 /proc
0 /dev
4.0K /media
7.8M /etc
9.9G /
Detailed storage
admin#vps202702:~$ sudo du -x / | sort -n | tail -40
120736 /var/www/plannings_ecranvillage/code/.git/objects/pack
121908 /home/admin/.rvm/gems/ruby-2.2.1/cache
128912 /usr/lib/locale
129972 /var/www/plannings_ecranvillage/code/.git/objects
132152 /var/www/plannings_ecranvillage/code/.git
165128 /lib/modules/3.16.0-4-amd64/kernel
168844 /lib/modules/3.16.0-4-amd64
168848 /lib/modules
184432 /usr/share/doc
185980 /usr/share/locale
204432 /usr/bin
208020 /var/www/plannings_ecranvillage/code/log
209936 /lib
225812 /var/www/plannings_ecranvillage/code/bundle/ruby/2.2.0/gems
234060 /var/www/plannings_ecranvillage/code/vendor/bundle/ruby/2.2.0/gems
293548 /var/lib
297180 /var/www/plannings_ecranvillage/code/bundle/ruby/2.2.0
297184 /var/www/plannings_ecranvillage/code/bundle/ruby
297188 /var/www/plannings_ecranvillage/code/bundle
306828 /var/www/plannings_ecranvillage/code/vendor/bundle/ruby/2.2.0
306832 /var/www/plannings_ecranvillage/code/vendor/bundle/ruby
306836 /var/www/plannings_ecranvillage/code/vendor/bundle
306852 /var/www/plannings_ecranvillage/code/vendor
444924 /usr/lib/x86_64-linux-gnu
562076 /home/admin/.rvm/gems/ruby-2.2.1/gems
771244 /home/admin/.rvm/gems/ruby-2.2.1
771252 /home/admin/.rvm/gems
842120 /usr/share
907040 /home/admin/.rvm
941852 /home/admin
946400 /usr/lib
954900 /home
956848 /var/www/plannings_ecranvillage/code
956876 /var/www/plannings_ecranvillage
956892 /var/www
2037752 /usr
5734208 /var/log/nginx
5735444 /var/log
7007748 /var
10282476 /
Node in use
admin#vps202702:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda1 655360 151619 503741 24% /
udev 249112 295 248817 1% /dev
tmpfs 251192 355 250837 1% /run
tmpfs 251192 1 251191 1% /dev/shm
tmpfs 251192 3 251189 1% /run/lock
tmpfs 251192 13 251179 1% /sys/fs/cgroup
tmpfs 251192 4 251188 1% /run/user/1001
admin#vps202702:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 9.9G 9.9G 0 100% /
udev 10M 0 10M 0% /dev
tmpfs 393M 5.3M 388M 2% /run
tmpfs 982M 0 982M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 982M 0 982M 0% /sys/fs/cgroup
tmpfs 197M 0 197M 0% /run/user/1001
I have created a directory mytmp in /home/admin and edited with this line
the /home/admin/.bashrc
export TMPDIR=/home/admin/mytmp
My disk is not so big :
admin#vps202702:~$ sudo fdisk -l
Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000
Device Boot Start End Sectors Size Id Type
/dev/vda1 * 2048 20971519 20969472 10G 83 Linux
Loocking for inodes
root#vps202702:~# for i in /*; do echo $i; find $i |wc -l; done
/bin
151
/boot
7
/dev
297
/etc
2162
/extlinux.conf
1
/home
33553
/initrd.img
1
/ldlinux.c32
1
/ldlinux.sys
1
/lib
4597
/lib64
2
/lost+found
1
/media
1
/mnt
1
/opt
34
/proc
14853
/root
9
/run
360
/sbin
146
/srv
3
/sys
15871
/tmp
6
/usr
85647
/var
25250
/vmlinuz
1
Passenger version
Phusion Passenger 5.0.28
RVM version
admin#vps202702:~$ rvm -v
rvm 1.26.11 (latest) by Wayne E. Seguin <wayneeseguin#gmail.com>, Michal Papis <mpapis#gmail.com> [https://rvm.io/]
Trying to solve with this trick
admin#vps202702:~$ sudo -i
root#vps202702:~# pushd /proc ; for i in [1-9]* ; do ls -l $i/fd | grep "(deleted)" && (echo -n "used by: " ; ps -p $i | grep -v PID ; echo ) ; done ; popd
/proc ~
~
I'm not sure, but I think something is wrong with /proc ?
What's wrong ?
I am getting this error whenever I am trying to build my image. Searched on internet and got some links but none of them solved my problem.
Error:
System error: write /cgroup/docker/5dba72d862bf8171d36aa022d1929455af6589af9fb7ba6220b01842c7a7dee6/cgroup.procs: no space left on device.
This is the output of 'df -h':
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 56G 24G 29G 46% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 1.9G 4.0K 1.9G 1% /dev
tmpfs 385M 1.3M 384M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.9G 324M 1.6G 17% /run/shm
none 100M 20K 100M 1% /run/user
/dev/sda8 761M 3.4M 758M 1% /boot/efi
/dev/sda3 55G 49G 3.8G 93% /home
/dev/sda4 275G 48G 213G 19% /opt/drive2
/dev/sda5 184G 74G 102G 43% /opt/drive3
/dev/sda6 215G 157G 48G 77% /opt/drive4
/dev/sda7 129G 23G 99G 19% /opt/drive1
You have no space in your host machine, where you are trying to build the image. First of all, you have to free some space. If you are using Debian based system this will help you:
df -h (show free space)
apt-get autoclean
apt-get autoremove
BleachBit (CCleaner-like for Linux).
Moreover, you can optimize your image building following these Dockerfile tips.
This is common problem if you are using docker-machine. In VM is disk smaller. Try cleanup your docker using these commands:
# Delete all containers
docker rm $(docker ps -a -q)
# Delete all images
docker rmi $(docker images -q -f dangling=true)
the reason may docker devicemapper Base Device Size.
please consider this or my answer on stackoverflow
I'm trying to make some C code with a simple gcc command in Ubuntu 10, but for some reason, I keep getting an error:
Cannot create temporary file in /tmp/: No space left on device
The thing is, though, I have plenty of space on the disk. Here is output of df -h:
Filesystem Size Used Avail Use% Mounted on
/ 3.7G 2.4G 1.1G 70% /
devtmpfs 312M 112K 312M 1% /dev
none 312M 24K 312M 1% /dev/shm
none 312M 80K 312M 1% /var/run
none 312M 0 312M 0% /var/lock
none 312M 0 312M 0% /lib/init/rw
And df -i, in case you are wondering about the inodes:
Filesystem Inodes IUsed IFree IUse% Mounted on
/ 240960 195198 45762 82% /
devtmpfs 79775 609 79166 1% /dev
none 79798 3 79795 1% /dev/shm
none 79798 41 79757 1% /var/run
none 79798 2 79796 1% /var/lock
none 79798 1 79797 1% /lib/init/rw
I can also touch /tmp/test successfully, so I know I have space on the drive. Any ideas as to why gcc decided to throw a fit all of a sudden? (It was working earlier) Thanks beforehand.
It looks to me that your /tmp directory is actually mounted as a devtmpfs which if I remember correctly is actually your computer's RAM.
You can always reboot and see if that helps, increase your virtual memory partition, or you can close running programs to see if that helps. Additionally, you can maybe delete some unnecessary files from /tmp as they are volatile to at least the life of the session.
The intermediate files are too big for /tmp, so perhaps using another temporary directory (TMPDIR=/var/tmp g++ ...) helps.