Feedback on Network Design and Proxmox VM Isolation
Feedback on Network Design and Proxmox VM Isolation
Network design. I started my homelab / selfhost journey about a year ago. Network design was the topic that scared me most. To challenge myself, and to learn about it, I bought myself a decent firewall box with 4 x 2.5G NICs. I installed OPNsense on it, following various guides. I setup my 3 LAN ports as a network bridge to connect my PC, NAS and server. I set the filtering to be applied between these different NICs, as to learn more about the behavior of the different services. If I want to access anything on my server from my PC, there needs to be a rule allowing it. All other trafic is blocked. This setup works great so far an I'm really happy with it.
Here is where I ran into problems. I installed Proxmox on my server and am in the process of migrating all my services from my NAS over there. I thought that all trafic from a VM in Proxmox would go this route: first VM --> OPNsense --> other VM. Then, I could apply the appropriate firewall rules. This however, doesnt seem to be the case. From what I've learned, VMs in Proxmox can communicate freely with each other by default. I don't want this.
From my research, I found different ideas and opposing solutions. This is where I could use some guidance.
- Use VLANs to segregate the VMs from each other. Each VLAN gets a different subnet.
- Use the Proxmox firewall to prevent communication between VMs. I'd rather avoid this, so I don't have to apply firewall rules twice. I could also install another OPNsense VM and use that, but same thing.
- Give up on filtering traffic between my PC, NAS and server. I trust all those devices, so it wouldn't be the end of the world. I just wanted the most secure setup I could do with my current knowledge.
Is there any way to just force the VM traffic through my OPNsense firewall? I thought this would be easy, but couldn't find anything or just very confusing ideas.
I also have a second question. I followed TechnoTim to setup Treafik and use my local DNS and wildcard certificates. Now, I can reach my services using service.local.example.com, which I think is neat. However, in order to do this, it was suggested to use one docker network called proxy. Each service would be assigned this network and Traefik uses lables to setup the routes. '
Would't this allow all those services to communciate freely? Normally, each container has it's own network and docker uses iptables to isolate them from each other.
Is this still the way to go? I'm a bit overwhelmed by all those options.
Is my setup overkill? I'd love to hear what you guys think! Thank you so much!
By making a bridge in the opensense interfaces you have created a layer2 network. This means that all the devices connected on that network are broadcasting their Mac addresses and are added to the ARP table on the opensense. Since they all are on the same physical network and the same subnet, none of the traffic will ever hit the layer 3 rules on your opensense.
If you want opensense to handle the rules of the traffic you will need to put the devices on different subnets and separate clans. Create a gateway address for every vlan on the opensense and point your devices to the opensense as their gateway.
Ah, I did not know that. So I guess I will create several VLANs with different subnets. This works as I intended it, trafic coming from one VM has to go through OPNsense.
Now I just have to figure out, if I'm being to paranoid. Should I simply group several devices together (eg, 10=Servers, 20=PC, 30=IoT; this is what I see mostly being used) or should I sacrifice usability for a more fine grained segeration (each server gets its own VLAN). Seems overkill, now that I think about it.
You're definitely being paranoid. There's little to gain from giving each machine its own network. You're just over complicating things.
You can achieve your goal better by just using the host firewalls or the one in proxmox. You can easily automate it with ansible or terraform if you want to centrally manage your VMs (and you do).
It sounds like what you're looking to achieve is what's known as zero trust architecture (ZTA). The primary concept is that you never implicitly trust a particular piece of traffic, and always verify it instead.
The most common way I've seen this achieved is exactly what you're talking about - more micro-segmentation of your network.
The design principles are usually centred around what the crown jewels are in your network. For most companies applying ZTA, that's usually their data, especially customer data.
Ideally you create a segment that holds that data, but no processing/compute/applications. You can also create additional segments for more specific use cases if you like, but I've rarely seen this get beyond three primary segments: server; database; data storage (file servers, etc).
In your case, you can either create three separate VLANs on your Proxmox cluster, with your your OPNsense firewall having an interface defined in each, or use the Proxmox firewall. I'd go the former - OPNsense is a lot more capable than the Proxmox firewall, especially if you turn on intrusion detection.
I'm not using any further segmentation beyond my VMs sitting in their own VLAN from my physical, but here's a screenshot of my networking setup on Proxmox. I wrote this reply to another post here on Selfhosted, talking about how my interfaces are setup. In my case, I have OPNsense running as a VM on the same Proxmox cluster. As I said in there, it's a bit of a headfuck getting it done, but very easy to manage once setup.
BTW, ZTA isn't overkill if it's what YOU want to do.
You're teaching yourself some very valuable skills and, and you clearly have a natural talent for thinking both vertically and horizontally about your security. This shit is gold when I interview young techs. One of my favourite interview moments is when I ask about their home setups, and then get to see their passion ignite when they talk about it.
I want to do exactly that on my fw router. I have installed two identical SSDs and the plan is to install Proxmox and run Opnsense on top of it. I also want to segment my WLAN and allocate the 5GHz to phones and laptops and tablets. 2.4GHz to IoT devices and the Guest WLAN for occasional guests. Each one of them should be in a separate VLAN.
My ISP router would have the WLAN disabled and I will run it through a Netgear RX7800 running OpenWRT. The idea is to run Opnsense with intrusion detection and serve as the primary gateway for every device in my network.
Any guide or hint how I can achieve that would be highly appreciated.
VLANs are absolutely the key here. I run 4 SSIDs, each with its own VLAN. You haven't mentioned what switch hardware you're using, but I'm assuming it's VLAN-capable.
The (high-level) way I'd approach this would be to first assign a VLAN for each purpose. In your case, sounds like three VLANs for the different WLAN classes (people; IoT; guest) and at least another for infrastructure (maybe two - I have my Proxmox VMs in their own VLAN, separate to physical infra).
VLANS
Sounds like 5 VLANs. For the purposes of this, I'll assign them thusly:
- vlan10: people, 192.168.10.0/24
- vlan20: physical infrastructure, 192.168.20.0/24
- vlan30: Proxmox/virtual infra, 192.168.30.0/24
- vlan40: IoT, 192.168.40.0/24
- vlan50: guest, 192.168.50.0/24
That'll give you 254 usable IP addresses in each VLAN. I'm assuming that'll be enough. ;)
SWITCH
On your switch, define a couple of trunk ports tagging appropriate VLANs for their purpose:
- One for your Nighthawk, tagging VLANs 10, 20, 40 and 50 (don't need 30 - Proxmox/VMs don't use wireless)
- One for your Proxmox LAN interface, tagging all VLANs (you ultimately want to route all traffic through OPNsense)
If you had additional wired access points for your wireless network, you'd create additional trunk ports for those per item 1. If you have additional Proxmox servers in your cluster, ditto for item 2 above.
WIRELESS
I'm not that familiar with OpenWRT, but I assume you can create some sort of rules that lands clients into VLANs of your choice, and tags the traffic that way. That how it is on my Aruba APs.
For example, anything connecting to the IoT SSID would be tagged with vlan40. Guest with vlan50, and so on.
PROXMOX
- Create a Linux Bridge interface for the LAN interface, bridging the physical interface connected to SWITCH item 2, above
- Create Linux VLAN interfaces on the bridge interface, for each VLAN (per my screenshot example)
You haven't mentioned internet/WAN but, if you're going to use OPNsense as your primary firewall/router in/out of your home network, you'd also create a Linux Bridge interface to the physical interface connecting your internet
OPNSENSE
This is the headfuck stage (at least, it was for me at first). Simply put, you need to attach the Proxmox interfaces to your OPNsense VM, and create VLAN interfaces inside OPNsense, for each VLAN.
I'm not going to attempt to explain it in reduced, comment form - no way I could do it justice. This guide helped me immensely in getting mine working.
If you have any issues after attempting this, just sing out mate, and I'll try and help out. Only ask is that we try and deal with it in comment form here where practical, for when Googlers in the future land here in the Fediverse.
Thank you so much for your kind words, very encouraging. I like to do some research along my tinkering, and I like to challenge myself. I don't even work in the field, but I find it fascinating.
The ZTA is/was basically what I was aiming for. With all those replies, I'm not so sure if it is really needed. I have a NAS with my private files, a nextcloud with the same. The only really critical thing will be my Vaultwarden instance, to which I want to migrate from my current KeePass setup. And this got me thinking, on how to secure things properly.
I mostly found it easy to learn things when it comes to networking, if I disable all trafic and then watch the OPNsense logs. Oh, my PC uses this and this port to print on this interface. Cool, I'll add that. My server needs access to the SMB port on my NAS, added. I followed this logic through, which in total got me around 25-30 firewall rules making heavy use of aliases and a handfull of floating rules.
My goal is to have the control for my networking on my OPNsense box. There, I can easily log in, watch the live log and figure out, what to allow and what not. And it's damn satisfying to see things being blocked. No more unknown probes on my nextcloud instance (or much reduced).
The question I still haven't answered to my satisfaction is, if I build a strict ZTA or fall back to a more relaxed approach like you outlined with your VMs. You seem knowledgable. What would you do, for a basic homelab setup (Nextcloud, Jellyfin, Vaultwarden and such)?
What would you do, for a basic homelab setup (Nextcloud, Jellyfin, Vaultwarden and such)?
I guess my first question is are you intending to open up any of these to be externally available? Once you understand the surface area of a potential attack, you can be a lot more specific about how you protect yourself.
I have just about everything blocked off for external access, and use an always-on Wireguard VPN to access them when I'm not home. That makes my surface area a lot smaller, and easier to protect.
VLANs all the way. I have several VLANs, including:
- Virtual Servers
- Bare metal
- Trusted devices
- IoT devices
- Guest network etc.
EDIT: An alternative would be to replace or supplement Proxmox with Docker/Podman on the bare metal of the server. The container networking would be isolated by default. If you can replace your VM needs with containers, that may get you what you want.
Thanks for your input. Am I understanding right, that all devices in one VLAN can communicate with each other without going through a firewall? Is that best practice? I've read so many different opinions that it's hard to see.
You're correct about vlan.
Think of vlan is a regular switch. Connect stuff, they communicate. Make two vlans in a switch, think of turning your physical switch into two separate switches.
Connect you switches to a router. Don't want to waste two cables from your switch that's cut in half? Do trunk port, with vlan tagged.
Lots of videos will explain better.
Best practice is to separate things of different trust levels into different vlans. You can filter and control the traffic between those vlans with your router.
As previously mentioned, in the Enterprise and business world, best practice is to separate and management from VMS and applications. We call this data plane and control plane. You would restrict access to your proxmox for other hypervisor interface from the VMS and applications. For small home setups and funsies, this gets a little complicated, but if it's your career choice or interest, it's a good thing to explore.
Key terms you can research: data plane, control plane, out of band management, air gapped.
Might want to move the NAS to a separate, unrouted VLAN. Storage only needs local connections and it's good practice not to make it routable.
I would probably put my WLAN on a different subnet/VLAN then the rest of the network, maybe even split it into trusted and guest networks if your AP's support broadcasting multiple SSID's. I don't see anything about NAT/PAT so I assume this isn't going to touch the Internet and just for home brew fuckery.
Both public and local services. I have limited hardware for now, so I'm still using my ISP router as my WLAN AP. Not the best solution, I know, but it works and I can seperate my Home-WLAN from my Guest-WLAN easily.
I want to use an AP at some point in the future, but I'd also need a managed switch as well as the AP itself. Unfortunately, thats not in my budget for now.
If you do go down the VLAN route, make sure to define enforce the networks on the Proxmox and firewall side. If you set the VLAN ID on the client instead, an attacker could change it to a different network.
Not sure how exactly Proxmox works for this, but generally you'd distinguish between tagged and untagged ports.
You'd use untagged ports for client/vm access. Any packet gets the VLAN tag set to what you define.
A tagged port would be used to connect Proxmox to the router. This keeps the VLAN tags in packets intact for the routing you'll need to do.Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
Fewer Letters More Letters AP WiFi Access Point IP Internet Protocol IoT Internet of Things for device controllers NAS Network-Attached Storage NAT Network Address Translation SSD Solid State Drive mass storage SSH Secure Shell for remote terminal access SSO Single Sign-On VPN Virtual Private Network
9 acronyms in this thread; the most compressed thread commented on today has 10 acronyms.
[Thread #427 for this sub, first seen 13th Jan 2024, 22:05] [FAQ] [Full list] [Contact] [Source code]
I'm not knowledgeable on communication between VMs and how to best restrict communication there, but I have tried to make my docker networks more secure.
I went a bit overkill for my reverse proxy and all the docker networks it's connected to. For each service I want to expose through my reverse proxy, I manage a network specifically for that service in my caddy docker compose file. I then refer to that external network in my servjce's docker compose file, so that caddy can access it. For example, caddy is on caddy_net-grafana and on caddy_net-homepage. Grafana and homepage are on those networks respectively. So with this setup, caddy can talk to Grafana and homepage, but Grafana and homepage cannot talk to each other.
It wasn't too bad to setup. I made my own conventions for keeping it manageable and it works for me. I did run into the problem where I had to increase the default subnet pool, as after you create like 30 or 31 networks there aren't any subnets left to give out to new docker networks.
This sounds promising. If I understand correctly, you have a ton of networks declared in your proxy, each for one service. So if I have Traefik as my proxy, I'd create traefik-nextcloud, traefik-jellyfin, traefik-portainer as my networks, make them externally available and assign each service their respective network. Did I get that right?
Yup. It works pretty well for me. Just will likely have to increase the default address pool in daemon.json if you go all out.