In a hurry, read the TLDR version below...
Quick summary of the problems I've been experiencing, in case you hadn't seen me ranting on Twitter...
1) Since upgrading to Catalina, I've been unable to complete a single Time Machine backup. Time Machine would continually try to backup, but after days or even weeks, it would eventually fail and I'd have to start it over.
At first I was using an existing Time Machine backup from Mojave. After that failed a few times, I decided to wipe it and start from scratch. Even after starting from scratch, the Time Machine backup would still take forever and then eventually fail. I joked on Twitter... that I was going to post screenshot updates each day showing the progress, but that it would end up being a 6+ month project.
Instead, I filed a FuBAr1 (FeedBack Assistant/radar) and sent Apple some sysdiagnostic and tmdiagnostic files.
2) The second problem I was complaining about was due to my continuously filling SSD that wouldn't give up space no matter how much I deleted from it.
Of course, I was blaming APFS and Time Machine for this, since it's well known now that APFS snapshots are used to keep data available until Time Machine can back it up. I've been using tmutils to delete these snapshots when I needed to free up space, but even after deleting the snapshots, I wasn't getting my free space back2.
I received feedback from my report I'd sent to Apple (Yay, they do work!). They pointed out that the issue was due to a third party app I have installed called Disk Drill by CleverFiles (which I have as part of my Setapp subscription). The app has a "feature" called Guaranteed Recovery that is supposed to help recover files later if you accidentally delete them. It "works" by creating thousands of hard links to what appears to be every file on your system in a hidden directory (/System/Volumes/Data/.cleverfiles/). This means when you delete a file, it's not really deleted because there's a hard link effectively creating a duplicate in the hidden folder. Personally I feel this is a terrible feature since you should have proper backups of your data anyway, so when you delete a file, it should be deleted. (Note: I originally installed Disk Drill in order to attempt to recover images from an SD Card that died, and so had no idea it was doing anything wonky to my main drives).
Over the last month or so, I've been trying to free up space on my main SSD because the OS is constantly complaining that my drive is full. Because I kept getting alerts that I needed more free space, I kept moving/deleting files. Eventually I'd cleared/off-loaded over 500G of data and was still scraping by with about 30G of free space. Now that I'm aware of the issue, I've taken a look into the .cleverfiles hidden folder on my iMac and I see it has over 450G in it. Wow.
First thing, add /System/Volumes/Data/.cleverfiles/ to the Time Machine exception list. This should fix the first issue with Time Machine taking forever to backup. Though, if you are using an existing Time Machine backup, it can still take a long long time for the next backup. My MBP has been working on it's first backup since I added the exception for 2 days still, and it's completed 150G of 177G. I created a brand new backup on my iMac and it completed it's first 650G backup in a day, so I'd recommend just wiping your old Time Machine backup and starting fresh (unless there's something you really need in the old backup, but then, why not restore it first, then wipe etc). Wait until after you read the rest of this post before staring a new backup though.
Next, I went into Disk Drill to disable to the Guaranteed Recovery feature. I don't need it, and think it's a flawed idea. While looking at how to disable it, I see it has a setting for how big the "Guaranteed Recovery Storage" should be. It's set to 8GB by default. Recall my .cleverfiles stats it has >450G. A lot more than 8G. That limit is supposed to be the limit of files hard linked but deleted. But it appears there's a bug in the app because it hasn't been trimming down to the 8G limit (perhaps there's a Catalina issue Disk Drill is hitting that prevents the trimming, and that's why this only started getting really out of hand after I upgraded). I didn't bother digging into this since I'm disabling the feature anyway. After disabling the feature, and "resetting" the storage, I expected the .cleverfiles folder to be removed or at least cleared. It was neither. Perhaps another issue due to Catalina. (Note that Disk Drill does have Full Disk Access permissions on my machine).
I went back into Disk Drill, and it had re-enabled Guaranteed Recovery on each of the drives I had just disabled it on. Another bug? Not sure, no time to dig into it, so instead, I uninstalled the app from my machines. Note that you can't just delete the app, you need to follow the full uninstall instructions here (modified slightly if you have the Setapp version): https://www.cleverfiles.com/help/how-to-uninstall-disk-drill/
I then manually deleted the /System/Volumes/Data/.cleverfiles/ folder.
One issue is that I can't remove a folder: /Library/StagedExtensions/Library/Application Support/CleverFiles due to it being read-only in Catalina. I supposed I could turn off SIP and delete it then, but I try not to do things normal users won't/can't do in order to keep my system consistent for testing.
After deleting .cleverfiles, finishing a full Time Machine backup, and ensuring there are no remaining APFS snapshots, I now have 593G free! Subsequent Time Machine backups are completing quickly again, and I expect my whole machine will start behaving better again (time will tell).
My thanks go out to Apple staff who responded to my report and helped restore (some of) my sanity! Hopefully the developers at CleverFiles can fix the issues they have in Disk Drill.
TLDR: It wasn't Apple's fault. A third party app, Disk Drill by CleverFiles was causing the issue. It was also causing the issue that prevented hard drive space freeing up as I deleted files. The apps not so clever feature was creating hidden hard links to basically every file on the system so you could recover it later in the odd chance you accidentally deleted a file.↩︎
Footnotes:
1: I'm still trying to make this nickname for the new Feedback Assistant reports stick, but I'm probably the only one. 🤪↩︎
2: I had used Daisy Disk to try and locate the missing disk space, but it didn't help find them. It did have a huge chunk (4-500G) marked as hidden, but didn't break down the location.↩︎