Initially I've increased the volume size of "/dev/mapper/rhel-var" from 2g to 9g.
When trying to load docker image I get this error:
Filesistem:
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 4.0K 3.9G 1% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 9.0M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/rhel-root 50G 8.8G 42G 18% /
/dev/sda1 1014M 150M 865M 15% /boot
/dev/mapper/rhel-home 21G 13G 7.9G 62% /home
/dev/mapper/rhel-tmp 5.0G 35M 5.0G 1% /tmp
/dev/mapper/rhel-var 9.0G 152M 8.9G 2% /var
/dev/mapper/rhel-var_log 9.0G 53M 9.0G 1% /var/log
/dev/mapper/rhel-var_tmp 2.0G 74M 2.0G 4% /var/tmp
/dev/mapper/rhel-var_log_audit 3.0G 2.9G 105M 97% /var/log/audit
tmpfs 783M 0 783M 0% /run/user/1000
Is it any other volume that needs aditional storage space?
Docker image (I mean the .tar file) space is around 4.8G
$ ll
total 4634988
-rwxr-x---. 1 admin admin 4746224640 Oct 28 14:51 daas2.tar*
Thanks
Cosmin
Later edit:
Got this crash of the machine
$ sudo docker image load -i daas2.tar
7ea4455e747e: Loading layer [==================================================>] 80.31MB/80.31MB
b3b741e72ab9: Loading layer [==================================================>] 46.68MB/46.68MB
9c79dfcaa270: Loading layer [==================================================>] 3.584kB/3.584kB
2335e0a013ff: Loading layer [==================================================>] 4.608kB/4.608kB
5dabc97cd0ea: Loading layer [======================> ] 812.7MB/1.772GB
Remote side unexpectedly closed network connection
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Session stopped
- Press <return> to exit tab
- Press R to restart session
- Press S to save terminal output to file
h of the machine:
The problem was:
/dev/mapper/rhel-var_log_audit 3.0G 2.9G 105M 97% /var/log/audit
I've check according to "Start loading it, while it is happening, open a second terminal, run df repeatedly and observe which usage chagnes over time. –
KamilCuk
Oct 28 at 17:14"
and this volume went full then crash.
Conclusion:
When loading a docker image here is where it writes things, at least in:
/var/lib/docker
/var/run/docker
/var/log/audit
I've cleared the /var/log/audit and it worked.
Thanks!
Elasticsearch (in docker) uses much more disk space than it shown in the info about indices stat.
It shows only 7.8gb
curl 'localhost:9201/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open my_index LL319QgqSJ6FNDEh2ZQw8g 1 1 24896180 0 7.8gb 7.8gb
yellow open test u0IJ7cocSXSlRST_qCPhHg 1 1 8 0 5.5kb 5.5kb
but when I see the disk space info inside the docker container I see the following
docker exec -it es bash
df -h
Filesystem Size Used Avail Use% Mounted on
overlay 278G 51G 213G 20% /
tmpfs 64M 0 64M 0% /dev
tmpfs 3.0G 0 3.0G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/vda1 278G 51G 213G 20% /etc/hosts
tmpfs 3.0G 0 3.0G 0% /proc/acpi
tmpfs 3.0G 0 3.0G 0% /sys/firmware
I`ve found such a command.
curl 'localhost:9201/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
2 7.8gb 65gb 212.6gb 277.6gb 23 172.17.0.2 172.17.0.2 571fa89f411f 2 UNASSIGNED
But I cant understand what ES spends 65-7.8~57gb of disk space?
As I understood, disk.used shows 'Elasticsearch, including the translog and unassigned shards; The node’s OS; Any other applications or files on the node'. How can I decrease consumption of the disk?
So my question is I have an errant rails app deployed using Dokku with the default Digital Ocean setup. This rails app has eaten all of the disk space as I did not set up anything to clean out the /tmp directory.
So the output of df is:
Filesystem 1K-blocks Used Available Use% Mounted on
udev 1506176 0 1506176 0% /dev
tmpfs 307356 27488 279868 9% /run
/dev/vda1 60795672 60779288 0 100% /
tmpfs 1536772 0 1536772 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 1536772 0 1536772 0% /sys/fs/cgroup
/dev/vda15 106858 3419 103439 4% /boot/efi
tmpfs 307352 0 307352 0% /run/user/0
So I am out of disk space, but I don't know how to enter the container to clean it. Any dokku **** return /home/dokku/.basher/bash: main: command not found
Access denied which I have found out is because I am completely out of HD space.
So 2 questions.
1: How do I get into the container to clear the tmp directory
2: Is there a way to set a max disk size limit so Dokku doesn't eat the entire HD again?
Thanks
Dokku uses docker to deploy your application, you are probably accumulating a bunch of stale docker images, which over time can take over all of your disk space.
Try running this:
docker image ls
Then try removing unused images:
docker system prune -a
For more details, see: https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes
I have two Physical machine installed with Docker 1.11.3 on ubuntu. Following is the configuration of machines -
1. Machine 1 - RAM 4 GB, Hard disk - 500 GB, quad core
2. Machine 2 - RAM 8 GB, Hard disk - 1 TB, octa core
I created containers on both machines. When I check the disk space of individual containers, here are some stats, which I am not able to undestand the reason behind.
1. Container on Machine 1
root#e1t2j3k45432#df -h
Filesystem Size Used Avail Use% Mounted on
none 37G 27G 8.2G 77% /
tmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda9 37G 27G 8.2G 77% /etc/hosts
shm 64M 0 64M 0% /dev/shm
I have nothing installed in the above container, still it is showing
27 GB used.
How come this container got 37 GB of space. ?
2. Container on Machine 2
root#0af8ac09b89c:/# df -h
Filesystem Size Used Avail Use% Mounted on
none 184G 11G 164G 6% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda5 184G 11G 164G 6% /etc/hosts
shm 64M 0 64M 0% /dev/shm
Why only 11GB of disk space is shown as used in this container. Even
though this is also empty container with no packages installed.
How come this container is given 184 GB of disk space ?
The disk usage reported inside docker is the host disk usage of /var/lib/docker (my /var/lib/docker in the example below is symlinked to my /home where I have more disk space):
bash$ df -k /var/lib/docker/.
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/... 720798904 311706176 372455240 46% /home
bash$ docker run --rm -it busybox df -k
Filesystem 1K-blocks Used Available Use% Mounted on
none 720798904 311706268 372455148 46% /
...
So if you run the df command on the same container on different hosts, a different result is expect.
I am trying to resize a logical volume on CentOS7 but am running into the following error:
resize2fs 1.42.9 (28-Dec-2013)
resize2fs: Bad magic number in super-block while trying to open /dev/mapper/centos-root
Couldn't find valid filesystem superblock.
I have tried adding a new partition (using fdisk) and using vgextend to extend the volume group, then resizing.
Resize worked fine for the logical volume using lvextend, but it failed at resize2fs.
I have also tried deleting an existing partition (using fdisk) and recreating it with a larger end block, then resizing the physical volume using lvm pvresize, followed by a resize of the logical volume using lvm lvresize. Again everything worked fine up to this point.
Once I tried to use resize2fs, using both methods as above, I received the exact same error.
Hopefully some of the following will shed some light.
fdisk -l
[root#server~]# fdisk -l
Disk /dev/xvda: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0009323a
Device Boot Start End Blocks Id System
/dev/xvda1 * 2048 1026047 512000 83 Linux
/dev/xvda2 1026048 41943039 20458496 8e Linux LVM
/dev/xvda3 41943040 62914559 10485760 8e Linux LVM
Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-root: 29.5 GB, 29532094464 bytes, 57679872 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
pvdisplay
[root#server ~]# pvdisplay
--- Physical volume ---
PV Name /dev/xvda2
VG Name centos
PV Size 19.51 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 4994
Free PE 0
Allocated PE 4994
PV UUID 7bJOPh-OUK0-dGAs-2yqL-CAsV-TZeL-HfYzCt
--- Physical volume ---
PV Name /dev/xvda3
VG Name centos
PV Size 10.00 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 2559
Free PE 0
Allocated PE 2559
PV UUID p0IClg-5mrh-5WlL-eJ1v-t6Tm-flVJ-gsJOK6
vgdisplay
[root#server ~]# vgdisplay
--- Volume group ---
VG Name centos
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 6
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 2
Act PV 2
VG Size 29.50 GiB
PE Size 4.00 MiB
Total PE 7553
Alloc PE / Size 7553 / 29.50 GiB
Free PE / Size 0 / 0
VG UUID FD7k1M-koJt-2veW-sizL-Srsq-Y6zt-GcCfz6
lvdisplay
[root#server ~]# lvdisplay
--- Logical volume ---
LV Path /dev/centos/swap
LV Name swap
VG Name centos
LV UUID KyokrR-NGsp-6jVA-P92S-QE3X-hvdp-WAeACd
LV Write Access read/write
LV Creation host, time localhost, 2014-10-09 08:28:42 +0100
LV Status available
# open 2
LV Size 2.00 GiB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
--- Logical volume ---
LV Path /dev/centos/root
LV Name root
VG Name centos
LV UUID ugCOcT-sTDK-M8EV-3InM-hjIg-2nwS-KeAOnq
LV Write Access read/write
LV Creation host, time localhost, 2014-10-09 08:28:42 +0100
LV Status available
# open 1
LV Size 27.50 GiB
Current LE 7041
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:1
I've probably done something stupid, so any help would be greatly appreciated!
After a bit of trial and error... as mentioned in the possible answers, it turned out to require xfs_growfs rather than resize2fs.
CentOS 7,
fdisk /dev/xvda
Create new primary partition, set type as linux lvm.
n
p
3
t
8e
w
Create a new primary volume and extend the volume group to the new volume.
partprobe
pvcreate /dev/xvda3
vgextend /dev/centos /dev/xvda3
Check the physical volume for free space, extend the logical volume with the free space.
vgdisplay -v
lvextend -l+288 /dev/centos/root
Finally perform an online resize to resize the logical volume, then check the available space.
xfs_growfs /dev/centos/root
df -h
In Centos 7 default filesystem is xfs.
xfs file system support only extend not reduce. So if you want to resize the filesystem use xfs_growfs rather than resize2fs.
xfs_growfs /dev/root_vg/root
Note: For ext4 filesystem use
resize2fs /dev/root_vg/root
I ran into the same exact problem around noon today and finally found a solution here --> Trying to resize2fs EB volume fails
I skipped mounting, since the partition was already mounted.
Apparently CentOS 7 uses XFS as the default file system and as a result resize2fs will fail.
I took a look in /etc/fstab, and guess what, XFS was staring me in the face... Hope this helps.
resize2fs Command will not work for all file systems.
Please confirm the file system of your instance using below command.
Please follow the procedure to expand volume by following the steps mentioned in Amazon official document for different file systems.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html
Default file system in Centos is xfs, use the following command for xfs file system to increase partition size.
sudo xfs_growfs -d /
then "df -h" to check.
On centos and fedora work with fsadm
fsadm resize /dev/vg_name/root
CentOS7 + VM
Ive made it with:
Gparted-live extend the volume
pvresize -v /dev/sda2
lvresize -r -l+100%FREE centos/root
On Centos 7, in answer to the original question where resize2fs fails with "bad magic number" try using fsadm as follows:
fsadm resize /dev/the-device-name-returned-by-df
Then:
df
... to confirm the size changes have worked.
After reading about LVM and being familiar with PV -> VG -> LV, this works for me :
0) #df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 824K 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/fedora-root 15G 2.1G 13G 14% /
tmpfs 1.9G 0 1.9G 0% /tmp
/dev/md126p1 976M 119M 790M 14% /boot
tmpfs 388M 0 388M 0% /run/user/0
1) # vgs
VG #PV #LV #SN Attr VSize VFree
fedora 1 2 0 wz--n- 231.88g 212.96g
2) # vgdisplay
--- Volume group ---
VG Name fedora
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 231.88 GiB
PE Size 4.00 MiB
Total PE 59361
Alloc PE / Size 4844 / 18.92 GiB
Free PE / Size 54517 / 212.96 GiB
VG UUID 9htamV-DveQ-Jiht-Yfth-OZp7-XUDC-tWh5Lv
3) # lvextend -l +100%FREE /dev/mapper/fedora-root
Size of logical volume fedora/root changed from 15.00 GiB (3840 extents) to 227.96 GiB (58357 extents).
Logical volume fedora/root successfully resized.
4) #lvdisplay
5) #fd -h
6) # xfs_growfs /dev/mapper/fedora-root
meta-data=/dev/mapper/fedora-root isize=512 agcount=4, agsize=983040 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0 rmapbt=0
= reflink=0
data = bsize=4096 blocks=3932160, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 3932160 to 59757568
7) #df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 828K 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/fedora-root 228G 2.3G 226G 2% /
tmpfs 1.9G 0 1.9G 0% /tmp
/dev/md126p1 976M 119M 790M 14% /boot
tmpfs 388M 0 388M 0% /run/user/0
Best Regards,
os: rhel7
After gparted, # xfs_growfs /dev/mapper/rhel-root did the trick on a living system.
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 47G 47G 20M 100% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 9.3M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 1014M 205M 810M 21% /boot
tmpfs 379M 8.0K 379M 1% /run/user/42
tmpfs 379M 0 379M 0% /run/user/1000
# lvresize -l +100%FREE /dev/mapper/rhel-root
Size of logical volume rhel/root changed from <47.00 GiB (12031 extents) to <77.00 GiB (19711 extents).
Logical volume rhel/root successfully resized.
# xfs_growfs /dev/mapper/rhel-root
meta-data=/dev/mapper/rhel-root isize=512 agcount=7, agsize=1900032 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=12319744, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=3711, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 12319744 to 20184064
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 77G 47G 31G 62% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 9.3M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 1014M 205M 810M 21% /boot
tmpfs 379M 8.0K 379M 1% /run/user/42
tmpfs 379M 0 379M 0% /run/user/1000
How to resize root partition online :
1) [root#oel7 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/root_vg-root 5.0G 4.5G 548M 90% /
2)
PV /dev/sda2 VG root_vg lvm2 [6.00 GiB / 0 free]
as here it shows that there is no space left on root_vg volume group, so first i need to extend VG
3)
[root#oel7 ~]# vgextend root_vg /dev/sdb5
Volume group "root_vg" successfully extended
4)
[root#oel7 ~]# pvscan
PV /dev/sda2 VG root_vg lvm2 [6.00 GiB / 0 free]
PV /dev/sdb5 VG root_vg lvm2 [2.00 GiB / 2.00 GiB free]
5) Now extend the logical volume
[root#oel7 ~]# lvextend -L +1G /dev/root_vg/root
Size of logical volume root_vg/root changed from 5.00 GiB (1280 extents) to 6.00 GiB (1536 extents).
Logical volume root successfully resized
3) [root#oel7 ~]# resize2fs /dev/root_vg/root
resize2fs 1.42.9 (28-Dec-2013)
resize2fs: Bad magic number in super-block while trying to open /dev/root_vg /root
Couldn't find valid filesystem superblock.
as root partition is not a ext* partiton so , you resize2fs will not work for you.
4) to check the filesystem type of a partition
[root#oel7 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/root_vg-root xfs 6.0G 4.5G 1.6G 75% /
devtmpfs devtmpfs 481M 0 481M 0% /dev
tmpfs tmpfs 491M 80K 491M 1% /dev/shm
tmpfs tmpfs 491M 7.1M 484M 2% /run
tmpfs tmpfs 491M 0 491M 0% /sys/fs /cgroup
/dev/mapper/data_vg-home xfs 3.5G 2.9G 620M 83% /home
/dev/sda1 xfs 497M 132M 365M 27% /boot
/dev/mapper/data_vg01-data_lv001 ext3 4.0G 2.4G 1.5G 62% /sybase
/dev/mapper/data_vg02-backup_lv01 ext3 4.0G 806M 3.0G 22% /backup
above command shows that root is an xfs filesystem , so we are sure that we need to use xfs_growfs command to resize the partition.
6) [root#oel7 ~]# xfs_growfs /dev/root_vg/root
meta-data=/dev/mapper/root_vg-root isize=256 agcount=4, agsize=327680 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=1310720, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 1310720 to 1572864
[root#oel7 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/root_vg-root xfs 6.0G 4.5G 1.6G 75% /
To resize the existing volume mounted
sudo mount -t xfs /dev/sdf /opt/data/
mount: /opt/data: /dev/nvme1n1 already mounted on /opt/data.
sudo xfs_growfs /opt/data/
In my case I could to fix the superblock location with these command:
yum install gdisk
parted -l /dev/mapper/centos-root
growpart /dev/mapper/centos-root 1
xfs_growfs /