Selfhosted

1. You don't need zfs cache. Stay away from it. This isn't going to help with what you want to do anyway. Just have enough RAM.

2. You need to backup your stuff. Follow the 3-2-1 rule. RAID is not a backup.

3. Don't use hardware raids, there are many benefits to using software these days.

With that said, let's dig into it. You don't really need NVMe drives tbh. SATA is probably going to be sufficient enough here. With that said, having mirrored drives will be sufficient enough as long as you are backing up your data. This also depends on how much space you will need.

I just finished building out my backup and storage solution and ended up wanting NVMe drives for certain services that run. I just grabbed a few 1 TB drives and mirrors them. Works great and I do get better performance, even with other bottlenecks. This is then replicated to another server for backup and also to cloud backup.

You also haven't said what hardware you are currently using or if you are using any software for the raid. Are you currently using zfs? Unraid? What hardware do you have? You might be able to use a pice slot to install multiple NVMe drives in the same slot. This requires bifurcation though.

I have run photoprism straight from mdadm RAID5 on some ye olde SAS drives with only a reduction in the indexing speed (About 30K photos which took ~2 hours to index with GPU tensorflow).

That being said I'm in a similar boat doing an upgrade and I have some warnings that I have found are helpful:

1. Consumer grade NVMEs are not designed for tons of write ops, so they should optimally only be used in RAID 0/1/10. RAID 5/6 will literally start with a massive parity rip on the drives, and the default timer for RAID checks on Linux is 1 week. Same goes for ZFS and mdadm caching, just proceed with caution (ie 321 backups) if you go that route. Even if you end up doing RAID 5/6, make sure you get quality hardware with decent TBW, as sever grade NVMEs are often triple in TBW rating.
2. ZFS is a load of pain if you're running anything related to Fedora or Redhat, and the performance implications from lots and lots of testing is still arguably inconclusive on a NAS/Home lab setup. Unless you rely on the specific feature set or are making an actual hefty storage node, stock mdadm and LVM will probably fulfill your needs.
3. Btrfs has all the features you need but is a load of trash in performance, highly recommend XFS for file integrity features + built in data dedup, and mdadm/lvm for the rest.

I'm personally going with the NVME scheduled backups to RAID because the caching just doesn't seem worth it when I'm gonna be slamming huge media files around all day along with running VMs and other crap. For context, the 2TB NVME brand I have is only rated for 1200 TBW. That's probably more then enough for a file server, but for my homelab server it would just be caching constantly with whatever workload I'm throwing at it. Would still probably last a few years no issues, but SSD pricing has just been awful these past few years.

On a related note, Photoprism needs to upgrade to Tensorflow 2 so I don't have to compile an antiquated binary for CUDA support.

2 HDDs (mirrored zpool), 1 SATA SSD for cache, 32 GB RAM

First read: 120 MB/s

Read while fully cached (obviously in RAM): 4.7 GB/s

Hah. I see your looking into ZFS caching. Highly recommend. I'm running Ubuntu 24.04 Root on ZFS RAID10. Twelve each data drives and one nvme cache drive. Gotta say it's performing exceptionally. ZFS is a bit tricky, it requires an HBA not a RAID card. You may to to flash the raid card to get it working like I did. After that, I have put together a GitHub for the install on ZFS RAID 10, but you should easily be able to change it to RAIDz2. Fair warning, it wipes the drives entirely.

https://github.com/Reddimes/ubuntu-zfsraid10/

You may be confused about the terminology:

RAID = many disks that help prevent data loss

NVME = Just an SSD

A collection of drives that are joined to a RAID group prevent data loss. An NVME drive is just a drive, which has no data loss prevention. If that drive dies, the data dies.

If those docs say anything about SSD, it's because their code is slow, and it will seem less obvious by using a faster disk.

Both your RAID and NVMe data should be getting backed up daily to 2 different destinations, if it's irreplaceable.

But to answer your question, just place the DB and cache files for Photoprism on the NVMe, and the photos themselves on the RAID.

Spinning disks can perform well if you do it right

https://www.raidz-calculator.com/

Just toss in some high endurance SSD's and you are golden.

RAID gives you greater uptime. That is all. You should also have backups. So how much uptime do you need?

Note: If you want to backup a DBMS, you're going to want to use some system that ensures that the backup is atomic.

If your device permits it, run raid on disc, and use nvme as cache. My Synology does this.

I‘m building this for some time now. I run a raid 1 on mechanical disks for file storage and have an ssd for the logic/database stuff. Have bought a second ssd and will put them in a raid as well bc limited lifespan.