New to home networking - Security advice?

Depends on your security model, but a common one is to assume any internet facing service is at risk. So you should partition your sensitive data away from the internet facing services. On your home network set up a VLAN, and only attach your internet facing services to one VLAN. Your personal stuff, your sensitive computer, have on a separate VLAN, and they should only transition to the internet and not each other. Or at least the server VLAN should not be able to initiate connections to the personal VLAN.

This won't prevent your server from getting hacked, but it will contain the spread if it does

Networking is super simple - or at least it started out like that. Then we ran out of numbers, and had to invent nat. Then we invented ipv6, which has lots of numbers, but is unfathomably complicated.

I recommend learning about NAT / network address translation. NAT is not a stateful firewall, but acts kinda like one.

You can understand a stateful firewall by understanding the tcp handshake. TCP is hugely important. Don’t worry about fin_wait_2 and that nonsense, just get syn/synack/ack down.

People will brush off udp because it’s easier, but it’s also important.

Once you get NAT/stateful firewalls, I would look into wireguard. That’s the protocol underneath tailscale. Know that it wraps your tcp packets in an encrypted udp datagram. Then find out how tailscale sets up your wireguard connections without port forwarding - or don’t, as webrtc-style signaling is famously impossibly complicated.

Here’s what you should do - spin up all the services you want, but put them behind an nginx reverse proxy. Then put that behind a WAF. Getting those layers aligned will teach you a huge amount of useful stuff.

In general, don’t worry about hackers unless exposing a port to the internet. Then worry. Your router’s stateful firewall will do a good job until you poke holes in it.

If you want a cool side project, listen on port 20 and dump the characters that the web scanners send to you. If they don’t send anything, send a username prompt after the tcp handshake - the robots will give you the login creds that they try against weak boxes :)

Right now I don’t think you need to do anything special. Unless you did something out of the ordinary, Tailscale didn’t put your computers directly on the internet. What it did was create what’s called an overlay network that allows devices connected to that network to talk to each other. It’s private and encrypted and random folks on the internet can’t get on it by default.

Do learn some networking (there are tons of great YouTube channels, books, and podcast that can help), but right now you can afford to do that slowly and not try to rush to complicate your setup before you understand it. There’s so much material out there, but I found this particularly useful to get an overview when I was first learning: https://www.grc.com/sn/sn-309.pdf.

The other comments have already touched upon specific security recommendations and useful learning material. But since you did request an ELI5, I figured I'd throw in some simple advice.

I don't know much about Tailscale, but it looks to be an encrypted VPN into your server. This pipe to your server is secure from actors spying on your public WiFi connection, but would not help if -- for example -- your laptop is compromised and uses the VPN to further attack your server. To that end, the principle of "defense in depth" says that the server itself should have its own firewall, as a secondary or tertiary layer to keep the bad things away.

Your server firewall should default to reject*/block all inbound connections, with explicit exceptions only for services you intend to expose, such as a web server, SSH server, JellyFin, etc. Once an inbound connection is approved through the firewall, the outgoing reply to the client would also be allowed through, as would any follow-up traffic that is part of that same connection. This is connection-tracking, which all stateful firewalls can perform. Debian/Ubuntu use ufw as the default firewall, and it is IMO very easy to configure for common services or port numbers.

The next thing you can do to secure the server itself is to limit your attack space. Don't use password authentication if you can avoid it, and use good, complex passwords where you must. Your SSH server can be configured to silently reject passwords and only accept public-key authentication, and your JellyFin authentication can be generated and stored by a good password manager.

At this point, we could go on about per-application recommendations, but just having a firewall on your server staves off a lot of script-kiddie level of attacks, from outside or even within your LAN.

• The difference between reject and block in the firewall context is that reject causes a reply to be sent back to the client, positively informing them that access is denied and to not try again. The drawback is that this reveals that a firewall is in place, but is also valuable information when debugging a network connection. Whereas block silently discards network traffic, the same result as if the network lost the packet. IMO, block should only ever be used for WAN firewalls -- to not reveal too much info to a potential attacker -- but internally, firewalls within a LAN should use reject, as the benefits outweigh the risk of a network intruder who is already on the LAN.

As for the bonus question, with that much hardware, you could do interesting things such as experimenting with a Kubernetes cluster, or a ZFS filesystem. Or maybe you can do Chia mining with all that disk space, or Folding@Home (and CureCoin). If you're more into just VMs and how they network together, it would be a good test bench to learn about Layer 2 forwarding and Layer 3 routing, if you wanted to understand how IPv6* traffic traverses multiple (virtual) Ethernet links.

• Note: I am an IPv6 fanboy and promote it wherever I can over legacy IP (aka IPv4)

Finally, from the hardware specs you've given, might this be some sort of Dell or HPE server? If so, I would strongly urge you to enable the Lights-out Management (LOM) functionality (Dell calls this iDRAC; HPE calls it iLO), if you haven't already. It may be the single most important tool for any system administrator, which is your role now, since you are in charge of this server. In short, LOM is like having a KVM, plus power control, and the ability to push physical buttons on the server and attach USB drives, all via a slick HTML5 interface.

Good luck and have fun!