this post was submitted on 29 Nov 2023
2 points (100.0% liked)

Data Hoarder

170 readers
1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 1 year ago
MODERATORS
[email protected]
2
Where do I go from here? Upgrade path from ~60TB and full backups. (alien.top)
submitted 11 months ago by [email protected] to c/[email protected]
0 comments fedilink hide all child comments
 

Here's my setup:

  • Ubuntu Server with ZFS
  • i3-4160, 16GB ECC, SuperMicro X10SL7-F with 6 + 8 (LSI 2308) SATA ports
  • 8x8TB raidz2
  • 4x4TB raidz1
  • Windows 10 Workstation
  • 3700X, 64GB
  • 5x10TB w/StableBit DrivePool
  • 1x14TB
  • 3x3TB
  • 2021 Macbook Pro 16"
  • M1 Max, 64GB
  • 1x12TB, 2x4TB in MacOS RAID Assistant JBOD

So I've got 80TB on the ZFS server, ~70TB on the workstation, and 20TB on the mac. I specifically use 3 different operating systems, 3 different filesystems, and synchronize everything using Syncthing. Data is spread out based on level of redundancy desired (some things have 3 copies, most have 2 copies, and some have 1 copy with a periodic dump of the file tree so I would know what I lost.) Doing this with snapshots on ZFS has made for a very robust system immune to a lot of threats.

The problem is I'm nearing 90% full on all three systems. Do I build another system twice as large as the current largest? Is that the most efficient algorithm for data growth? Would you choose a different multiple? What hardware?

Would you instead just expand the ZFS server somehow with HBAs and another chassis? I wouldn't mind that solution, but I've not found a good case that is just focused on HDDs and fans. Do you know of one?

Thanks for opinions!

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here