How to remove large number of files on linux?
I had a client that left a broken script running for years and was wondering why his server was filling up.
They had multiple accounts with 500,000 to 1,000,000 files which were insane. I started to remove with a find command but after 24 hrs hours later it was still running and barely put a dent in number of files.
find /home/user/mail/new -type f -exec rm -vf {} \;
I then found someone one talking in a forum about using rsync with delete. So, next, I tried that out and wow, it was like 100x faster.
This is what i used:
rsync -av --delete /root/empty/ /home/user/mail/new
The rsync command removed all the files very fast, next issue is the directory it taking a larger number of bytes than the default 4096, so I also like to rename and remove the directory which the files were in.
drwxr-xr-x 2 user user 39071744 Aug 18 22:23 new/
after I delete and recreate, it should look like this:
drwxr-xr-x 2 user user 4096 Aug 19 09:00 new/
There was still a large number of inodes associated with the directory and I aways like to clear that out in addition to cleaning up all the files from a large directory.
I highly recommended this alternative to use rsync instead of using find and rm to clear out a large number of files in a directory.