The df command shows the disk is full but it is not

Share this post to your friends !
  • 15
    Shares

The df command shows the disk is full but it is not
5 (100%) 1 vote[s]

SystemMen - In this article, I will share with you a case where the df command shows the disk is full but it is not.

Case description

I am using a MariaDB cluster. On the master server number 1, I received a warning that the folder /var/log is full.

I login to the server, checking with the df command, the result that the folder using 99-100%.

[root@db-01 danie]# df -Th
Filesystem                       Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root          ext4       30G  2,6G   26G  10% /
devtmpfs                         devtmpfs  3,9G     0  3,9G   0% /dev
tmpfs                            tmpfs     3,9G     0  3,9G   0% /dev/shm
tmpfs                            tmpfs     3,9G  425M  3,5G  11% /run
tmpfs                            tmpfs     3,9G     0  3,9G   0% /sys/fs/cgroup
/dev/sda1                        xfs       497M  162M  336M  33% /boot
/dev/mapper/centos-tmp           ext4       22G   45M   20G   1% /tmp
/dev/mapper/centos-var_log       ext4       30G   28G  494M  99% /var/log
/dev/mapper/centos-home          ext4      9,8G   37M  9,2G   1% /home
/dev/mapper/centos-var_lib_mysql ext4      197G   24G  164G  13% /var/lib/mysql
tmpfs                            tmpfs     783M     0  783M   0% /run/user/1000

You can see that this folder is 30GB in size and it is using 28GB.

Next, I use the du command to display the current size of the /var/log directory. What surprised me was that it was only using 1.1GB.

[root@db-01 danie]# du -sh /var/log
1,1G   /var/log

So what’s going on?

Find the cause of the df command shows the full is disk

the-df-command-shows-the-disk-is-full-but-it-is-not The df command shows the disk is full but it is not
The df command shows the disk is full but it is not.

With my experience, I have encountered the case of files that have been deleted but it is not actually deleted. These files are labeled deleted but it is still on the disk.

This causes the df command to shows an incorrect result.

I use the lsof command to find deleted files. And I found out that there is a slow log file of about 27-28GB in size labeled deleted.

[root@db-01 danie]# lsof +L1
COMMAND    PID  USER   FD   TYPE DEVICE    SIZE/OFF NLINK    NODE NAME
systemd-j  523  root  txt    REG  253,0      274752     0  660140 /usr/lib/systemd/systemd-journald;5bad8321 (deleted)
systemd-l  776  root  txt    REG  253,0      572320     0  660142 /usr/lib/systemd/systemd-logind;5bad8321 (deleted)
dbus-daem  784  dbus  txt    REG  253,0      442080     0  660381 /usr/bin/dbus-daemon;5bad8321 (deleted)
agetty     799  root  txt    REG  253,0       36864     0  659369 /usr/sbin/agetty;5bad8321 (deleted)
NetworkMa  811  root  txt    REG  253,0     2618600     0  660555 /usr/sbin/NetworkManager;5bad8321 (deleted)
mysqld    9700 mysql   20u   REG  253,5        9083     0      12 /tmp/ibWC769x (deleted)
mysqld    9700 mysql   21u   REG  253,5           0     0      13 /tmp/ibQKzxPO (deleted)
mysqld    9700 mysql   22u   REG  253,5        8551     0      14 /tmp/ibgmSBbm (deleted)
mysqld    9700 mysql   26u   REG  253,5           0     0      15 /tmp/ibqJ15SC (deleted)
mysqld    9700 mysql   30w   REG  253,4 28315029916     0 1048578 /var/log/mariadb/slow.log-20181223 (deleted)

Immediately at this step, I knew why the notification disk was full but not really.

My job now is to delete this file. I have written a script to delete these files.

Specifically in this case. I will do the following steps.

Step 1: Find files with deleted labels.

[root@db-01 danie]# find /proc/*/fd -ls | grep  '(deleted)'
815211450    0 lrwx------   1 mysql    mysql          64 Mar 20 13:52 /proc/9700/fd/20 -> /tmp/ibWC769x\ (deleted)
815211451    0 lrwx------   1 mysql    mysql          64 Mar 20 13:52 /proc/9700/fd/21 -> /tmp/ibQKzxPO\ (deleted)
815211452    0 lrwx------   1 mysql    mysql          64 Mar 20 13:52 /proc/9700/fd/22 -> /tmp/ibgmSBbm\ (deleted)
815211456    0 lrwx------   1 mysql    mysql          64 Mar 20 13:52 /proc/9700/fd/26 -> /tmp/ibqJ15SC\ (deleted)
724035709    0 l-wx------   1 mysql    mysql          64 Mar  5 18:08 /proc/9700/fd/30 -> /var/log/mariadb/slow.log-20181223\ (deleted)

Step 2: Get the file name to delete.

[root@db-01 danie]# find /proc/*/fd -ls | grep  '(deleted)' | awk '{print $11}'
/proc/9700/fd/20
/proc/9700/fd/21
/proc/9700/fd/22
/proc/9700/fd/26
/proc/9700/fd/30

In this case, I need to delete the slow log file with the deleted label. Therefore, , I will only need to delete the file /proc/9700/fd/30.

Step 3: delete files.

rm -f /proc/9700/fd/30

Alternatively, you can use the df command with the -i option to display the disk capacity more accurately.

[root@db-01 danie]# df -i
Filesystem                         Inodes IUsed    IFree IUse% Mounted on
/dev/mapper/centos-root           1966080 43052  1923028    3% /
devtmpfs                           998417   402   998015    1% /dev
tmpfs                             1001256     1  1001255    1% /dev/shm
tmpfs                             1001256   542  1000714    1% /run
tmpfs                             1001256    16  1001240    1% /sys/fs/cgroup
/dev/sda1                          256000   334   255666    1% /boot
/dev/mapper/centos-tmp            1411680    24  1411656    1% /tmp
/dev/mapper/centos-var_log        1966080   108  1965972    1% /var/log
/dev/mapper/centos-home            655360    29   655331    1% /home
/dev/mapper/centos-var_lib_mysql 13107200   449 13106751    1% /var/lib/mysql
tmpfs                             1001256     1  1001255    1% /run/user/1000

You see the above result is completely different, it has displayed the correct capacity of the disk.

What does this parameter -i mean?

-i, --inodes
      list inode information instead of block usage

Conclusion

This is one of the experiences I have experienced. If in the process of working with Linux servers, you encounter it. Hope this article will be useful to you, don’t be too worried if you don’t understand why the disk is always full.

If you liked this article, then please subscribe to our YouTube Channel for more video tutorials. You can also find us on Twitter and Facebook.


Share this post to your friends !
  • 15
    Shares
If you appreciate what we share in this blog, you can support us by:
  1. Stay connected to: Facebook | Twitter | Google Plus | YouTube
  2. Subscribe email to recieve new posts from us: Sign up now.
  3. Start your own blog with SSD VPS - Free Let's Encrypt SSL ($2.5/month).
  4. Become a Supporter - Make a contribution via PayPal.
  5. Support us by purchasing Ribbon Lite Child theme being using on this website.

We are thankful for your support.

«« »»

Got something to say? Join the discussion

Please keep in mind that all comments are subject to our Comment Policy. Your email address will not be published.
This site uses Akismet to reduce spam. Learn how your comment data is processed.