Selfhosted

Somethign I haven't seen mentioned yet is clevis and tang, basically if you have more than one server then they can unlock each other and if they're spatially separated then it is very unlikely they get stolen at the same time.

Though you have to make sure it stops working when a server get stolen, using a mesh VPN works just as well after the server is stolen so either use public IPS and a VPN or use a hidden raspberry pi that is unlikely to be stolen or make the other server stop tang after the first one is stolen.

Checkout this one: https://blog.haschek.at/2020/the-encrypted-homelab.html

It might be what you're looking for.

I would do FDE yeah. My current laptop setup is with systemd-boot and a special initramfs that allows me to unlock it with a yubikey, with fallback to password. Fair warning, this exact configuration is not particularly easy to setup.

There are also modules which enable early network connectivity along with a SSH server, meaning you login and unlock it remotely. I have not tried this.

Debian does not frequently require rebooting under normal circumstances. Kernel updates are not that frequent, and you can usually put it off for a bit if you don't want to deal with it.

I'm using https://github.com/dracut-crypt-ssh/dracut-crypt-ssh on some of my servers. The initrd opens an ssh port where you can login and enter the passphrase. Setting it up is non-trivial, but it works well. Haven't tried it on Debian but there should be something similar.

This has been on my mind, I have yet to do it but the implementation seems trivial.

You can use typical luks full disk encryption with a password. Luks actually has five password slots. Passwords do not have to be actual text, they can be a file or even part of a file.

So my idea is, buy some really cheap, low profile USB flash drives and store some seemingly innocuous data like cat pictures or public domain books, IDK and it doesn't matter what the actual data is. Use full disk encryption and set a regular password, then add a second password that is a file or part of a file that lives on the flash drives, and have it set up to look for that file on boot as an option for unlocking.

Now the disc is fully encrypted but will boot/reboot without interruption as long as the flash drive is installed. You can remove the flash drive when you're feeling paranoid, or even better only install it when you are going to be away for a while. If you leave with the machine having the flash drive but are feeling worried, you can remote into the machine and edit / delete the file or just clear the key slot from Luks.

That's what's been on my mind, anyway. I think the typical suggestion/solution is to just use drop bear and remotely unlock using that, or don't use full disk encryption and selectively encrypt your data instead (partitions or userspace encryption).

I'm not going to proofread this so I hope it makes sense

Unless the crook happens to be extremely nerdy or its law enforcement, already being a Linux formatted partition feels it should be enough for a rando breaking in and stealing a computer.

That being said, something like a PiKVM connected to your server (and Tailscale) could let you enable both UEFI/boot password and propt for LUKS decryption upon boot.

Full disk encryption with a LVM inside a LUKS arrition is pretty easy to setup, arch wiki is very helpful for that ! You can even encrypt most of the bootloader, so the drive is 100% useless if stolen.

Other options are LUKS with Tang and Clevis, or LUKS with SSH and Dropbear.

Sorry, I have no details.

Edit: Tang/Clevis are local software and a network server that provide keys. If stolen, won't boot.

SSH and Dropbear make it so you can login to provide keys.

Since I already use ZFS for my data storage, I just created a private dataset for sensitive data. I also have my services split based on if it's sensitive or not, so the non sensitive stuff comes up automatically and the sensitive stuff waits for me to log in and unlock the dataset.

I use LUKS on my systems. I use mandos and wireguard in intramfs to connect to a mandos server to unlock LUKS during boot.

Good question.

Debian doesn't often require a reboot, but the longer you go, and if you need kernel modules (nvidia is the worst at this) you might need to reboot to keep everything in sync.

My suggestion: raspberry pi, like 1st edition, keep the key very secure, give it a usb serial console. When the server reboots, enter the password that way. It's your emergency console.

Luks FDE, and install dropbear-initramfs, configure ssh authorized_keys and rebuild initramfs. Then you can access initramfs via ssh to type luks password.

You can use FDE and setup a minimal ssh server like dropbear to run at startup. This way, you can supply the password via a keyboard connected to the machine OR via ssh. This gives you a similar workflow to the data partition you mentioned, but encrypts the entire system.

If you want to encrypt only the data partition you can use an approach like https://michael.stapelberg.ch/posts/2023-10-25-my-all-flash-zfs-network-storage-build/#encrypted-zfs to ulock it at boot.

TL;DR: store half of the decryption key on the computer and another half online and write a script that at boot fetches the second half and decrypt the drive. There is a timewindow where a thief could decrypt your data before you remove the key if they connect your computer to the network, but depending on your thread model can be acceptable. you can also decrypt the root portion with a similar approach but you need to store the script in the initramfs and it is not trivial.

Another option I've seen suggested is storing the decryption key on a USB pendrive and connect it with a long extension cord to the server. The assumption is that a thief would unplug all the cables before stealing your server.

I am looking for some recommendations on how to secure the data of my physical servers (against physical theft)…

Lock the case shut. Run a security wire (like for laptops) to the case and mount the other end to something big / immovable.

Truly paranoid: 24/7 video surveillance.

Super truly paranoid: same as above but in a locked room

I'd go with the Full Disk Encryption. You can be sure everything is encrypted that way. Any additional complexity adds ways to mess up and compromise security. Entering the password is a bit cumbersome. But that's part of the deal. I just carry my computer keyboard to my NAS and enter the password each time I need to reboot. Which doesn't happen that often. There also used to be some tutorial somewhere on how to put a Dropbear SSH server into the initrd so you can enter the password over network.

On Dell server hardware with the right cards/licensing, you can remove the need for physical access to the server to input an FDE password by leaning on iDRAC. This provides access to the console remotely during the boot process (and thereafter).

Alternatives exist that supposedly do the same thing, but I've never had to try them. Airconsole, pikvm, blikvm etc.

You can keep this interface unexposed by using wireguard to dial in when you're away, as per your original thinking. Just make sure the endpoint isn't on the server you're rebooting...

I've been there, I used the "encrypted partition to be unlocked after boot via SSH"-option, but it quickly became tedious to have to input the password every time it rebooted. I wanted something that could recover by itself (I.e. start everything up again after a potential crash), so that I could maximize uptime and the investigate the crash later.

So I ended up disabling encryption. What I did instead was to find services with E2EE for my most sensitive stuff. Joplin for my personal notes is currently the only thing I have encrypted. Nextcloud has experimental E2EE, though I'm not really using it as of right now. Everything I deem too sensitive to trust my server with unencrypted, I store on encrypted flash drives.

I think the risk of the server itself being compromised/hacked is bigger than physical theft (at least in my case), and if you take some good precautionary measures, even that risk is pretty small unless you're being directly targeted by a skilled adversary. If the latter is the case, don't store sensitive stuff on something with an IP address.

I wanted to encrypt against physical theft but then I realized that the server weighs 20kg and there's more valuable and lighter stuff to steal. And a thief would never be able to mount the zfs array, they'll just fence my $2000 server for $50 and someone else will sell it piece by piece on ebay. And between people who buys used drives online on ebay almost nobody is going to do a full surface scan of a 12tb drive in the hopes of finding valuable data.

Could you just secure the physical location?

Do you not live in a safe place?