I am running a NextCloud instance on a Raspberry Pi in my parents' place, in order to backup my data. My dad ended up liking this solution and moved his data there too. It is running the OS in an SD card, but has an external HDD for the NextCloud data itself.
My problems started when I started getting strange errors from NextCloud. I investigated a bit and found that the root filesystem was corrupt. I saw the writing on the wall, turned it off before things got worse, bought a new SD card, and went over to recover it. Sure enough, the previous SD card had decided to suddenly pass on to the electronic afterlife. Ironically enough, this was the only partition for which I had no backups. Our personal data is, in addition to NextCloud, periodically manually rsync-ed into an external hard drive. I am in addition running another server, which has a proper backup solution in place.
After re-imaging the new card from the old one, running fsck, and restoring a few things that were still corrupted, I got NextCloud to run. At least our data was safe at the time, because it was in a different drive. I took the opportunity and synced my other machine, my dad also took the opportunity and rsync-ed his secondary hard drive, and I went back home and initially thought about calling it a day. However, I thought it would be easy enough to run a script that would take a weekly backup into the external HDD, and if both fail, well, I can then spend a day to set it up from scratch, given that our data itself is safe. Being who I am, I started implementing it at almost midnight.
I set it up to create a new directory with the current date every time, and I added this command in the end, to delete files that hadn't been accessed in over 22 days:
find $BACKUP_DIR -atime +22 -delete
I ran the script - Why is it late to finish, it created the files - oh oops 😱 - to my horror I realised that I had set $BACKUP_DIR to be the root directory of the external hard drive instead of the backup subdirectory! I initially thought I was safe, because I had run a file scan on NextCloud, plus my dad had rsynced his data, everything should count as accessed today. My data was fortunately safe, but I compared my father's data against his secondary HDD and realised that there was, in fact, data loss. Surprisingly, neither the rsync nor NextCloud's scan had counted as accessing the files.
But, fortunately there's the secondary hard drive, right?... Well, it wasn't so simple. First, the drive is NTFS, where you can't manually set a file's created/modified timestamp. As a result, most files appeared newer in the secondary NTFS drive: instead of having the real timestamp of when they were created/modified, they had the timestamp of when they were rsync-ed. I did a few rsync dry runs, saw the issue, and fortunately it was easily solvable with the --ignore-existing flag.
Second, rsync was without --del, so any files my dad had deleted or moved would re-appear. I couldn't just bring them back and pretend nothing happened. Fortunately, he messaged me at 1AM for a different issue, so my eyes lit up and I replied "oh, thank God you're awake, can I upset you now?".
I sent him a diff of both directory listings, in case he could easily figure out what happened, but... he has a whole bunch of files. The diff was enormous and he felt overwhelmed. At this point, the most reasonable solution seemed to be for me to restore everything from the backup and let him re-delete the files. He gave me permission to go ahead with this, and at 3AM he could finally tell his laptop to start re-syncing all these files that now either had been undeleted or appeared to be newer.
There was an unexpected silver lining to the latter. There were some files that apparently he had accidentally deleted and he was wondering where they had gone. After we restored the backup, he found them back.
In the end, I removed the offending find and decided to just log in every month or two and delete any old backups.
TL;DR: --dry-run is your friend when rsync-ing. I accidentally deleted a big part of my father's personal files from his NextCloud account. Managed to recover them in the end, but it was quite a struggle.