This morning I arrived to find our backup servers root file system full. This is very strange as the backups all go onto a second btrfs disk which has loads of space left.
The Problem
The root file system was using the full ~40Gb of disk even though the backup runs a single python script using rsync and btrfs snapshots and these backups go onto a second disk not the root filesystem.
I ran apt-get autoremove --purge to see if there was anything left behind but it removed nothing. /tmp and /home were all fine too.
I then started walking the largest directory using du -sh to find the largest directories.
This showed that mlocate was using lots of disk.
Our backup system uses btrfs and creates a snapshot every night so from the mlocate view it was adding our whole backup size every night.
The Solution
I simply had to add our /mnt/btrfs directory to the PRUNEPATHS in /etc/updatedb.conf. Below shows the PRUNEPATHS before and after my change.
After running updatedb again the disk usage dropped to this :
TLDR : /var/lib/mlocate.db was massive due to lots of btrfs snapshots, added backup directory /mnt/btrfs to PRUNEPATHS variable in /etc/updatedb.conf and then ran updatedb.