Data Hoarder

170 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 1 year ago

MODERATORS

[email protected]

Is a 20 disk raidz3 really a bad idea, for media archiving use cases? (alien.top)

submitted 1 year ago by [email protected] to c/[email protected]

12 comments fedilink hide all child comments

I am archiving a vast amount of media files that are rarely accessed. I'm writing large sequential files, at peaks of about 100MB/s.

I want to maximise storage space primarily; I have 20x 18TB HDDs.

I've been told that large (e.g. 20 disk) vdevs are bad because resilvers will take a very long time, which creates higher risk of pool failure. How bad of an idea is this?

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 1 points 1 year ago (2 children)

Draid mitigates a lot of the problems of very wide raidz

https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html

[–] [email protected] 1 points 1 year ago

Draid was released just a month after I built my last raid set with vdevs. Really hoping there's an in-place migration path someday, assuming nobody finds any bugs in the next couple years.

[–] [email protected] 1 points 1 year ago (1 children)

Draid removes the capability of variable stripe widths, however. Strongly recommend a special metadata device when doing draid with the same level of redundancy as the rest of the pool.

[–] [email protected] 1 points 1 year ago

If you have lots of small files, yes this is bad.

For videos the space lost will just be a rounding loss.

Would be interesting to test for music. A 100k 20mbyte files. You could lose a lot of space if using 1 mbyte+ stripes that have been recommended for a while now.