this post was submitted on 07 Feb 2024
72 points (93.9% liked)

Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ

54443 readers
351 users here now

⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.

Rules • Full Version

1. Posts must be related to the discussion of digital piracy

2. Don't request invites, trade, sell, or self-promote

3. Don't request or link to specific pirated titles, including DMs

4. Don't submit low-quality posts, be entitled, or harass others



Loot, Pillage, & Plunder

📜 c/Piracy Wiki (Community Edition):


💰 Please help cover server costs.

Ko-Fi Liberapay
Ko-fi Liberapay

founded 1 year ago
MODERATORS
 

Having gradually built up my media collection to near the capacity of my 16TB external HDD, I've reached the point where I'll probably need to build a RAID array to keep the collection in one place. Assuming the RAID array will be at least 32TB, I have a few questions:

  1. From what I've read RAID arrays can help mitigate the risk of individual drives failing if extra space is allotted on the hard drives. Assuming a total capacity of 32TB, how much of that space would be reserved by the RAID array for data loss prevention?

  2. Is there a certain type of hard drive I would have to use? Aside from my 16TB drive, I also have two 2 8TB drives that I'd ideally like to be able to re-use in the RAID array, but have left them in their enclosures for the time being.

  3. If the hard drives in the array have different transfer speeds, does the array as a whole default to the slowest one?

  4. Whether the hard drives I already have are compatible or not, what RAID enclosure and hard drives would you recommend?

top 23 comments
sorted by: hot top controversial new old
[–] [email protected] 32 points 9 months ago* (last edited 9 months ago) (1 children)

The general datahoarder hive mind seems to be moving away from shucking and towards manufacturer refurbished drives, see https://serverpartdeals.com/ for example, especially in RAID where you can lose a drive with impunity and warranty is uncomplicated by shucking.

In RAID and similar strategies the redundancy comes from a parity drive, which protects against the loss of one (or more with some schemes) drive, so if you only have two drives of the same size it's just a mirror (50% of total pool) but you can have a drive die without data loss. With four drives you get to use three of the drives (75%), five you can use four (80%) etc. Classical RAID uses identical sized disks but there are other approaches that allow different sizes e.g. mergerfs + SnapRAID or Unraid, here you lose your largest disk to parity.

RAID is generally faster than the individual drives, e.g. mirroring is nearly twice as fast.

Perhaps go with another 2*8Tb which will get you 24Tb usable and use the 16Tb for offline, preferably offsite backup (remember RAID is not a backup, it protects you from disk failure, but not user error for example accidentally deleting things)

[–] [email protected] 7 points 9 months ago

Just be sure there is some gaps between the serial numbers of those hdds.

[–] [email protected] 15 points 9 months ago (2 children)

One thing I will also suggest is looking into https://home.tdarr.io/. It's software that will re-encode your media to h265, saving you a lot of space. I just set it up on my synology NAS because I am at 80% storage and don't feel like buying another drive yet.

[–] [email protected] 1 points 9 months ago

I've been manually reencoding to h265 l, and while the space savings are significant, the pain is also. Especially if you mess up the audio or subs!

Will definitely look into that!

[–] [email protected] 1 points 8 months ago (1 children)

Are there any drawbacks to h. 265?

I mostly watch Plex on my home network, however I use a Chromecast to watch Plex remote from my location with slow internet. Would this be an issue? Would it make the experience better?

I'm running two external tens and I just bought an 18. I'd love to save some space and I'll have to buy a drive for a long time.

[–] [email protected] 1 points 8 months ago

I am not an expert but I think compatibility is the only significant downside. If your device supports it, you h265 will give you better quality at the same file size.

[–] [email protected] 9 points 9 months ago (1 children)
  1. RAID for uptime, backups for data you care about. RAID(1+) will keep your data online when a disk fails, but backups are the real way to keep data around if shit hits the fan. For a personal media collection, you might be better served with a non resilient RAID0 (total failure if one drive fails) with a backup around to recover from when that happens. If you do e.g. a raid5 you lose 1 disk of capacity in exchange for 1 disk of resiliency, raid6 same but 2 disks. That gives you some safety but there are a lot of instances where those raids don't save you from losing all your data. If you buy 4x 18TB drives, you could have 36TB from the 1st two drives and then backup to the other two drives.

  2. There's no specific type of drive to worry about unless you're doing RAIDs especially with ZFS. Search shingle RAID rebuild for the biggest thing to worry about there.

  3. Almost always, yes. Slow drives throttle the rest.

  4. I've never used them but people say good things about synology most of the time. Everything comes with a cost and it's hard to make any sensible recommendations without knowing your constraints; primarily your budget.

[–] [email protected] 6 points 9 months ago

Want to echo what was said here. RAID is not a backup solution. RAID is for always on capability. Do not use RAID as a way to keep valuable data safe. Just have double the drive space and keep a backup of the data on the second drive.

[–] [email protected] 8 points 9 months ago (1 children)

Could use https://github.com/trapexit/mergerfs if you dont care about losing data. My 3 disks basically show up as one.

[–] [email protected] 3 points 9 months ago

You can also use SnapRaid along side MergerFS to provide some data redundancy. MergerFS will allow you to create a parity drive, without requiring all of your drives to be in your typical RAID pool. This way, if you have several drives die, then you can still access whatever data is available on the remaining drives.

[–] [email protected] 6 points 9 months ago
  1. https://www.synology.com/en-au/support/RAID_calculator or similar is good to easily do these calculations
  2. No, but more RAID configurations than not are limited by the smallest size drive. It's a factor to consider, assuming you can't afford to just buy a bunch of disks. I wound up maintaining two separate NAS devices, one of which gets my old, smaller disks.
  3. Generally yes, though you'd be surprised how little difference disk speed makes once you get enough of them in an array.
  4. I use Synology with various shucked WD externals. I have a bunch of other stuff in my homelab though, so I need the storage to not be it's own project, else I likely would have built something less expensive. I'm sure there will be better suggestions in this thread than mine.
[–] [email protected] 6 points 9 months ago (1 children)

Start googling. Hardware or Software based? How techy are you? RAID should not be considered for backup. Think of it as an individual drive. Still needs a backup. Personally I use a ZFS pool in my Linux homelab that's been rock solid. However the pucker factor would be high if a drive fails.

  1. Depends on how you set them up. There are calculators online that can estimate this for you.

  2. I used NAS drives bc they're expected to be spinning a lot.

  3. No. You will see better performance on average compared to your slowest drive.

  4. I have no relevant experience with specific hardware. For a general base of knowledge figure out what RAID type you want to use. Look at what ZFS or btrfs can do in this space. Same for unraid.

[–] InEnduringGrowStrong 6 points 9 months ago (1 children)

Personally I use a ZFS pool in my Linux homelab that’s been rock solid. However the pucker factor would be high if a drive fails.

I use a ZFS pool for my home stuff too.
To minimize the pucker factor, I have an extra drive of the same capacity and I rotate through them in the mirror pool.
It makes good practice replacing and resilvering drives and since it's a mirror it's also kind of an offline backup.
Also, I had bought the initial drives at the same time and figured rotating though them would minimize the odds of failing around the same time.
I'm much less wary of the whole thing now that I've resilvered the drives several times.

[–] [email protected] 1 points 9 months ago (1 children)

That's a really great idea. I know at least 2 of the drives are from the same batch so it would be good to rotate things in and out. Thanks!

[–] InEnduringGrowStrong 1 points 9 months ago

Yea, if anything it makes good practice.
Swapping hard drives in the larger data pool isn't as daunting anymore.
Although, I gotta be honest, swapping drives in the zpool from which proxmox boots itself is still a bit iffy to me.
I had to set a reminder, because otherwise I won't do it often enough.

[–] [email protected] 2 points 9 months ago

Gonna go with MalReynolds and mention buying refurbished drives over shucking. You are able to get a drive tailored for server use, rather than the crappy drives theyve been sticking in external enclosures lately. In a some cases, youre even getting a unused drive, even though its refurbished.

[–] [email protected] 1 points 9 months ago

I use mergerfs to pool a bunch of varying sized drives and then run snapraid on the whole pool to protect it. Then, I've got Kopia backups running and backup up to a remote repository. This solution has been flexible and I've already been able to recover from 2 drive failures very quickly and easily.

[–] [email protected] 1 points 9 months ago (1 children)

My man, You need a NAS. I can highly recommend TrueNAS scale if you want to go the self building route.

[–] [email protected] 0 points 9 months ago (1 children)

I'm not interested in accessing the hard drives remotely, so a NAS would probably be unnecessary for my use case.

[–] [email protected] 2 points 9 months ago (1 children)

You want the drives in your main rig?

[–] [email protected] 1 points 9 months ago (1 children)

My idea was to just put several external hard drives into a RAID enclosure and connect it to my computer via USB. Wouldn't that function similarly to a typical external hard drive?

[–] [email protected] 1 points 9 months ago

That is correct.

Most people use a NAS so they don't have to start their PC when they want to watch a movie on a TV or mobile, or ro back up their photos and files (automated if possible).

The NAS probably consumes less power, and is always accessible.

If you are alone and watch movies & play music on your PC, then don't bother.

I do it to learn about linux and stuff.

[–] [email protected] 1 points 9 months ago

There is btrfs - a linux filesystem. It keeps hashes for blocks to prevent data rot. And you can configure it into a raid that duplicates data across drives but doesn't require the exact same drives. You can just group any old drives together to create one large volume where everything is duplicated. At least that is how I understand this, I'm new to this too.