Data Hoarder

24 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 10 months ago

MODERATORS

[email protected]

What is your setup for data organisation and backup? (alien.top)

submitted 9 months ago by [email protected] to c/[email protected]

2 comments fedilink hide all child comments

Hello everyone,

I would love to hear how you deal with the organisation of your files as well as the backup.

Do you use software to organise files and to remove duplicates?

How do you store and backup your data? Do you use the 3-2-1 rule?

What is your process, software and hardware?

I myself am still struggling to find a way to do this in a secure and effective way, I would love to hear your experience and advice on this!

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 1 points 9 months ago

Do you use software to organise files and remove duplicates?

Yes and no. I use my two hands + one eye (the other doesn’t work) + find, du, rsync (in comparison mode) and diff. Everything I have is on a FreeBSD VM (docs, pictures) + Ubuntu VM (movies) so native tools work great for me.

How do you store and backup your data?

I have multiple Proxmox hosts that are not in a cluster. All of them and their VMs are being backed up on a physically separate Proxmox Backup Server (PBS) locally every day. In addition to that this system does rsnapshot backups on all of the VMs and hosts I care about. So it has 2 different storage backends (disk pools) - 1 for rsnapshot and 1 for Proxmox Backup Server. Both are 20TB in size and one uses XFS and the other uses EXT4. Every day the PBS backups are synced to a remote PBS instance in Hetzner Germany (my systems are in Bulgaria). The Hetzner system has 40TB in raidz2 (ZFS) and keeps all of the backups for 2 years. My storage VMs are also configured to do snapshots every 15 minutes and keep them for 24h. Yearly I also download my most important data and write it to BD discs and store those in my summer house.

So in short I have the following: Local snapshots every 15 minutes, 2 daily backups on an independent system using different software and storage backends, 1 daily remote backup on a system in another country, 1 yearly offline backup. It’s not perfect but it’s affordable and somewhat secure.