Updated in May 2021
Over the last few years me and my wife have been using the network server more frequently from its initial install and setup. We share 3-4 computers across the house and constantly move files onto the NAS. I have installed Nextcloud which I will detail in another post in the future. We have come across an issue that I have noticed with the network share. We moved all of our images and files over and ended up with tons of duplicates and a giant mess. I will show you what I am doing to clean that up. Our Nas is running on a ubuntu server. The following commands should work on most linux NAS servers.
First off, all my files are stored in /mnt/4000/home/*
Get rid of all OS specific files that shouldnt be on the NAS. This will shorten the next step of finding duplicates.
find ./ -iname desktop.ini -delete
Lets install a duplicates scanner:
Installation on Debian / Ubuntu
apt install fdupes
Installation on Fedora
dnf install fdupes
After installing, we can move onto the next step. Run this command to get a list of all the duplicates.
fdupes -r ./ > dupes.txt
Open this file up in a text editor and start hacking away. Delete all the lines of the files you wish to keep. Also delete any blank lines separating them. One example here:
./Shared/Pictures/2016/Krystal Phone Backup 6-21-16/IMG_1622.JPG ./Shared/Pictures/2016/pets/IMG_1622.JPG
I will be getting rid of the phone backup copy because I like to know this is part of pets.
./Shared/Pictures/2016/Krystal Phone Backup 6-21-16/IMG_1428.JPG ./Shared/Pictures/2016/IMG_1428.JPG
This one however ill keep the phone backup copy so that I don’t make a messy folder inside of 2016
Now that we are done editing our list, lets run the command on the list we want to delete.
while IFS= read -r file ; do rm -- "$file" ; done < dupes.txt
You may have to do this a few times to get any you missed.
Now that we have cleaned up all duplicates. We will be moving onto the next step. Finding empty files and folders.
Find all empty files
find ./ -type f -size 0 > emptyfiles.txt
File all empty folders
find ./ -type d -empty > emptyfolders.txt
find ./ -xdev -type f -size +100M > largefiles.txt
Delete any folders or files you want here, Be careful not to delete needed files for programs.
After all of this was done we managed to save 58GB of storage on our NAS.