6d ago

Encrypting data on local servers?

I am looking for some recommendations on how to secure the data of my physical servers (against physical theft), that I am about to set up. I am new to selfhosting but have a few years of experience running Linux on a desktop.

My usecase is a simple debian(?) server at home with Paperless ngx and Tailscale for when I am away from home.

The question is how to encrypt the data while still being able to keep the server updated.

Coming from Desktop my first thought was to simply enable FDE on install. But that would mean supplying the password everytime the server needs to reboot for an update. Could someone provide some insights on how often updates to debian require a reboot?

My second thought was to use an encrypted data partition. That way the server could reboot and I could use wireguard to ssh in and open the partition even when I am away from home for a longer time.

I am open to other ideas!

30 comments

Checkout this one: https://blog.haschek.at/2020/the-encrypted-homelab.html
It might be what you're looking for.

I would do FDE yeah. My current laptop setup is with systemd-boot and a special initramfs that allows me to unlock it with a yubikey, with fallback to password. Fair warning, this exact configuration is not particularly easy to setup.
There are also modules which enable early network connectivity along with a SSH server, meaning you login and unlock it remotely. I have not tried this.
Debian does not frequently require rebooting under normal circumstances. Kernel updates are not that frequent, and you can usually put it off for a bit if you don't want to deal with it.

You can use FDE and setup a minimal ssh server like dropbear to run at startup. This way, you can supply the password via a keyboard connected to the machine OR via ssh. This gives you a similar workflow to the data partition you mentioned, but encrypts the entire system.

I'm using https://github.com/dracut-crypt-ssh/dracut-crypt-ssh on some of my servers. The initrd opens an ssh port where you can login and enter the passphrase. Setting it up is non-trivial, but it works well. Haven't tried it on Debian but there should be something similar.
- This is actually really interesting, I might have to try this.
  I currently use a USB stick with a key file on it that I need to plug in on boot. Something like this but it wasn't that easy for me. https://openterprise.it/2022/07/fedora-unlock-luks-full-disk-encrypted-system-using-usb-stick/
  Edit: I wonder if yours will work on my VPS.... I'd love to encrypt that.

Unless the crook happens to be extremely nerdy or its law enforcement, already being a Linux formatted partition feels it should be enough for a rando breaking in and stealing a computer.
That being said, something like a PiKVM connected to your server (and Tailscale) could let you enable both UEFI/boot password and propt for LUKS decryption upon boot.
- I wouldn't rely on the thief not knowing how to read linux partitions. That very well may be the case, but the person they sell your hardware to will know better, considering they are in the market of purchasing used server hardware.
  I self host and my threat model is the thief selling my server to someone who knows what to do with it, but not knowing how to extract encryption keys from the memory of a running server before unpluging it. That being said I haven't figured out encryption yet so watching this thread.
- yea they'll plug in the drive and windows will popup "this drive needs to be formatted" and the rest is history

This has been on my mind, I have yet to do it but the implementation seems trivial.
You can use typical luks full disk encryption with a password. Luks actually has five password slots. Passwords do not have to be actual text, they can be a file or even part of a file.
So my idea is, buy some really cheap, low profile USB flash drives and store some seemingly innocuous data like cat pictures or public domain books, IDK and it doesn't matter what the actual data is. Use full disk encryption and set a regular password, then add a second password that is a file or part of a file that lives on the flash drives, and have it set up to look for that file on boot as an option for unlocking.
Now the disc is fully encrypted but will boot/reboot without interruption as long as the flash drive is installed. You can remove the flash drive when you're feeling paranoid, or even better only install it when you are going to be away for a while. If you leave with the machine having the flash drive but are feeling worried, you can remote into the machine and edit / delete the file or just clear the key slot from Luks.
That's what's been on my mind, anyway. I think the typical suggestion/solution is to just use drop bear and remotely unlock using that, or don't use full disk encryption and selectively encrypt your data instead (partitions or userspace encryption).
I'm not going to proofread this so I hope it makes sense
- I like this, and I suppose it's a shame a Rasp Pi can't be WOL'ed.
  But could another SFF single use/secured device on the same network that doesn't have FDE, also provide that key only if and when you wake it up (manually decrypt the file after ssh'ing into it too?) instead of having a USB drive directly plugged into the main server so, if a nefarious person does have away with the main bounty they're fugged without said second hidden device on the same network?
  ninjaedit: I also at one point in the past did Wireless WOL (wireless NIC's with WOL were prohibitively expensive at the time of me playing) via a Travel router that was acting as an Access Point, simply to wake up forward the magic packet. You could really hide that thing 😇
  
  I just bought some PoE hats for my rpis, and have a managed PoE switch; rumor is, this combo basically translates to rPi WoL.
  (Not meaning to ignore the rest of your comment, but not in a position to respond fully)

Full disk encryption with a LVM inside a LUKS arrition is pretty easy to setup, arch wiki is very helpful for that ! You can even encrypt most of the bootloader, so the drive is 100% useless if stolen.

I use LUKS on my systems. I use mandos and wireguard in intramfs to connect to a mandos server to unlock LUKS during boot.
- What's mandos? I don't find anything useful when searching for it.
  
  https://www.recompile.se/mandos
  I use this to get wireguard in intramfs. I just skip the dropbear related stuff. https://github.com/r-pufky/wireguard-initramfs
  
  https://www.recompile.se/mandos

Other options are LUKS with Tang and Clevis, or LUKS with SSH and Dropbear.
Sorry, I have no details.
Edit: Tang/Clevis are local software and a network server that provide keys. If stolen, won't boot.
SSH and Dropbear make it so you can login to provide keys.

Since I already use ZFS for my data storage, I just created a private dataset for sensitive data. I also have my services split based on if it's sensitive or not, so the non sensitive stuff comes up automatically and the sensitive stuff waits for me to log in and unlock the dataset.
- Not using ZFS but a similar approach: All my data (paperless, and other docker container data) is encrypted with LUKS on a separate disk. The OS is running unencrypted on the SD card (using a Raspberry Pi). This way I can swap out the system and relink the docker container data if needed. Yes, I do need to unlock after a reboot, but since the system is fully up, that's done easily via ssh.
  Still looking into ways to unlock it automatically on certain criteria...

I am looking for some recommendations on how to secure the data of my physical servers (against physical theft)…
Lock the case shut. Run a security wire (like for laptops) to the case and mount the other end to something big / immovable.
Truly paranoid: 24/7 video surveillance.
Super truly paranoid: same as above but in a locked room

Luks FDE, and install dropbear-initramfs, configure ssh authorized_keys and rebuild initramfs. Then you can access initramfs via ssh to type luks password.
- A more detailed guide for dropbear: https://www.cyberciti.biz/security/how-to-unlock-luks-using-dropbear-ssh-keys-remotely-in-linux/
  If I remember correctly, the only outdated bit of information is that the IP configuration doesn't happen anymore in the initramfs configuration but you must pass a parameter at the kernel by editing /etc/default/grub and passing
  GRUB_CMDLINE_LINUX_DEFAULT="ip=192.168.x.x:::::"
  where 192.168.x.x is the IP address that you want at boot

Good question.
Debian doesn't often require a reboot, but the longer you go, and if you need kernel modules (nvidia is the worst at this) you might need to reboot to keep everything in sync.
My suggestion: raspberry pi, like 1st edition, keep the key very secure, give it a usb serial console. When the server reboots, enter the password that way. It's your emergency console.

If you want to encrypt only the data partition you can use an approach like https://michael.stapelberg.ch/posts/2023-10-25-my-all-flash-zfs-network-storage-build/#encrypted-zfs to ulock it at boot.
TL;DR: store half of the decryption key on the computer and another half online and write a script that at boot fetches the second half and decrypt the drive. There is a timewindow where a thief could decrypt your data before you remove the key if they connect your computer to the network, but depending on your thread model can be acceptable. you can also decrypt the root portion with a similar approach but you need to store the script in the initramfs and it is not trivial.
Another option I've seen suggested is storing the decryption key on a USB pendrive and connect it with a long extension cord to the server. The assumption is that a thief would unplug all the cables before stealing your server.

I wanted to encrypt against physical theft but then I realized that the server weighs 20kg and there's more valuable and lighter stuff to steal. And a thief would never be able to mount the zfs array, they'll just fence my $2000 server for $50 and someone else will sell it piece by piece on ebay. And between people who buys used drives online on ebay almost nobody is going to do a full surface scan of a 12tb drive in the hopes of finding valuable data.

Could you just secure the physical location?
Do you not live in a safe place?

I'd go with the Full Disk Encryption. You can be sure everything is encrypted that way. Any additional complexity adds ways to mess up and compromise security. Entering the password is a bit cumbersome. But that's part of the deal. I just carry my computer keyboard to my NAS and enter the password each time I need to reboot. Which doesn't happen that often. There also used to be some tutorial somewhere on how to put a Dropbear SSH server into the initrd so you can enter the password over network.

On Dell server hardware with the right cards/licensing, you can remove the need for physical access to the server to input an FDE password by leaning on iDRAC. This provides access to the console remotely during the boot process (and thereafter).
Alternatives exist that supposedly do the same thing, but I've never had to try them. Airconsole, pikvm, blikvm etc.
You can keep this interface unexposed by using wireguard to dial in when you're away, as per your original thinking. Just make sure the endpoint isn't on the server you're rebooting...

I've been there, I used the "encrypted partition to be unlocked after boot via SSH"-option, but it quickly became tedious to have to input the password every time it rebooted. I wanted something that could recover by itself (I.e. start everything up again after a potential crash), so that I could maximize uptime and the investigate the crash later.
So I ended up disabling encryption. What I did instead was to find services with E2EE for my most sensitive stuff. Joplin for my personal notes is currently the only thing I have encrypted. Nextcloud has experimental E2EE, though I'm not really using it as of right now. Everything I deem too sensitive to trust my server with unencrypted, I store on encrypted flash drives.
I think the risk of the server itself being compromised/hacked is bigger than physical theft (at least in my case), and if you take some good precautionary measures, even that risk is pretty small unless you're being directly targeted by a skilled adversary. If the latter is the case, don't store sensitive stuff on something with an IP address.

30 comments