Detailed write-up on my VMware vSphere 7 nested homelab networking setup

Detailed write-up on my VMware vSphere 7 nested homelab networking setup

I have been getting quite a few inquiries about my fully nested VMware homelab setup. People seem to be struggling mostly with setting up the networking bits. I describe the overall topology in this blogpost but this blogpost will contain a lot more details on the network configuration. I also made a couple of changes, such as replacing the pfSense appliance with a Vyos appliance and the introduction of NSX-T (although this doesn’t change anything to my physical network configuration). I will use as many screenshots as possible to create a step by step process.

Update Video Tutorial

To answer some of the questions I received about my configuration, I recorded a video tutorial on setting up a nested homelab:

VMware vSphere 7 Nested Homelab Networking Setup Video Tutorial

The physical ESXi host

I’m using a Supermicro SYS-5028D-TN4T server. It has 4 physical NICs and an IPMI port but since my lab is running fully nested, I use only a single NIC. I even shared the IPMI on the NIC so I can save a port on my switch.

Routing between physical and nested with a Vyos appliance

My home network is just a simple flat networking running in 192.168.2.0/24 with my ISP router being the 192.168.2.254 default gateway to the internet.

I use a Vyos appliance to route between my physical homelab and all the VLANs I use in my homelab environment. I configured two NICs on the appliance. One NIC is connected to my physical home network (the uplink) and the other NIC is connected to my homelab environment:

If you look at the interface configuration you will see interface eth0 configured with 192.168.2.145/24 and eth1 is subdivided in multiple vif interfaces. These are subinterfaces for all my routable VLANs:

My ESXi management network for my homelab is VLAN 1611 with subnet 172.16.11.0/24 for example. That VLAN is routed by a VIF 1611 on the Vyos appliance with IP address 172.16.11.254/24. That’s the default gateway for VLAN 1611. The Vyos router holds a 0.0.0.0/0 route to my ISP router 192.168.2.254/24 by the way.

How to enable VLANs on ESXi

For the nested environment to support the use of VLANs, we need to take a couple of steps on the physical ESXi host network configuration. If we look at the physical NICs we see the single attached 1000Mbps interface. The other 3 NICs are disconnected:

First of all, we need two vSwitches:

The first switch has a single uplink (the only configured physical NIC on the Supermicro). This vSwitch holds two portgroups:

  1. The management vmkernel port so I can reach the physical ESXi host from my home network. This management interface is configured with the IP address 192.168.2.35/24 and default gateway 192.168.2.145/24.
  2. A Virtual Machine portgroup to hold all the VMs that run inside my home network 192.168.2.0/24. These are my Windows AD / Jumphost, a Linux jumphost and the uplink interface of the Vyos appliance:

Really straightforward. The second vSwitch is where the VLAN magic happens. This is a vSwitch without any uplinks. The Vyos appliance connects the first vSwitch with the uplink to the physical network 192.168.2.0/24 to the VLANs on this isolated vSwitch. The isolated vSwitch only has two Virtual Machine portgroups:

  1. Nested-ESXi
  2. ESXi-Mgmt

The magic of VLAN 4095

The Nested-ESXi switch is configured with VLAN 4095. This basically turns it into a VLAN trunk port. This means it now supports guest VLAN tagging. I can now create a management interface on a nested ESXi server and tag it with VLAN 1611 on the nested ESXi host itself. The ESXi-Mgmt is a basic portgroup with VLAN 1611 because I want to run some VMware appliances directly on the physical ESXi host and not on one of my nested ESXi hosts (such as vCenter Server, NSX-T Manager and the NSX-T Edge). They need access to the ESXi Management network VLAN 1611 172.16.11.0/24 so that’s why I needed to create that VLAN backed portgroup on the physical host as well. I can imagine this is where it becomes a bit fuzzy.

Additional screenshots

I will walk you through the setup in a couple of steps / screenshots.

The Nested-ESXi vSwitch:

Pay special attention to the MTU 9000 and security settings configuration (allow promiscuous, forged transmits and MAC changes)

Next up is the Nested-ESXi portgroup configuration:

Configuring VLAN 4095 on this port group is essential to enable VLAN support in a nested homelab. Also ensure you enable the required security settings to support nested virtualization:

The ESXi-Mgmt port-group to support VMs running on the physical ESXi host in VLAN 1611:

My Windows jumphost has IP address 192.168.2.36/24 and is using the Vyos router with 192.168.2.145/24 as its default gateway. The 0.0.0.0/0 route to 192.168.2.254 (my ISP router) enables internet access:

Nested vCenter and ESXi

I deployed vCenter Server on the physical ESXi host and deployed four nested ESXi hosts with 32GB RAM and 16 CPUs in a single cluster. Heavily oversubscribed of course but this does allow me to run nested applianced with a high CPU and RAM count demand. Logging into vCenter Server and taking a closer look to the network setup there:

vCenter holds one Distributed Switch with two uplinks (the other two uplinks are for NSX-T) and a number of portgroups:

  • ESXi Management VLAN 1611
  • vMotion VLAN 1612
  • vSAN VLAN 1613
  • Overlay VLAN 1615
  • Uplink 01 VLAN 50 (VLAN between NSX-T and the Vyos for north/south traffic)
  • Uplink 02 VLAN 60 (VLAN between NSX-T and the Vyos for north/south traffic)

I can configure MTU 9000 wherever I like because the underlying vSwitch on the physical ESXi host is also running MTU 9000. Some screenshots:

The ESXi Management portgroup with VLAN 1611:

The network topology from the perspective of a nested ESXi host:

The configuration of the VMkernel interfaces on a nested ESXi host (don’t forget to set the proper MTU here too):

What about NSX-T?

And that’s it for now :-). I’m currently rebuilding my nested homelab configuration with vSphere 7.0 with Kubernetes and NSX-T 3.0. This is taking a bit longer due to a lack of time but I will be sure to blog about those adventures too. Make sure you stay tuned!

If you have any questions about this nested setup, you know how to reach me (best on Twitter).


16 thoughts on “Detailed write-up on my VMware vSphere 7 nested homelab networking setup”

  • 1
    vlab athomas3 on June 1, 2020 Reply

    Honestly there are lots of gaps in this.
    It gives no one a clue how you deployed your esx’s inside that 1611 VLAN as per this post
    From that jumpbox you created its not possible to deploy this vCenter.
    Can you elaborate the networking section The magic of VLAN 4095 section with more pictures.
    I’ve tried this exactly as you did and i see no result.

    • 2
      vlab athomas3 on June 1, 2020 Reply

      n

    • 3
      jkusters on August 28, 2020 Reply

      I missed your question, sorry.

      I’m not sure which part you are missing. You just configure the management vmkernel with VLAN1611 (via the management console for example). The jumpbox is in my home netwerk and has a routed connection to my management VLAN. The vyos router takes care of this with the uplink connected to my home network and vif interfaces for my VLANs.

  • 4
    forbzie on June 16, 2020 Reply

    Hi,

    I have my homelab setup exactly the same way so far although i use pfsense for the VLAN’s.
    When will you be posting the install & configure part of NSX T version 3 ?
    I’m keen to understand how you setup the transport zones, node profiles and gateways etc….
    thanks

  • 5
    Shetu on August 28, 2020 Reply

    How can I use this method for pfsense vm?

  • 7
    jkusters on September 2, 2020 Reply

    I’ve added a video tutorial to this blogpost to provide some additional details…

  • 8
    pjhines on December 14, 2020 Reply

    I am trying to setup this lab but I guess my confusion is from the video and screen shots. The video has VLAN 6,8,10,11,12,and 13 but your screenshot from the router shows VLANS 50,60.70.80.1611,1612,1613,1614,1615 and 1621. So which configuration is correct for the VYOS router? I am really looking forward to having a fully functional nested lab.

  • 9
    Jeffrey Kusters on December 15, 2020 Reply

    I created the video on an different setup so there might be some inconsistency. Sorry about that. The VIF interfaces on the vyos need to align with the portgroups in vSphere. Every VLAN that needs to be routable (ie reach other networks) needs a VIF interface on vyos. If 1611 is for ESXi management, you need a VIF 1611 interface on the vyos router. OK?

  • 10
    David A Kropp on December 23, 2020 Reply

    This is fantastic, could we get a step by step walkthrough posted?

  • 11
    Christophe Lombard on March 13, 2021 Reply

    Hi Jeffrey,
    Thank you for the video. I have a question, for the TEP IP Pool does it have to be in the same subnet as VTEP VLAN?

  • 12
    AlPi on March 18, 2021 Reply

    Nice writeup Jeffrey!
    Quick question: You mention in your video that your LAB Windows AD controller is also your jumpbox.

    Don’t you run into issues by placing your LAB domain controller in your HOME network ?
    Was wondering how you prohibit your LAB domain controller “spilling” over into your home network where you might also have AD running.

    Cheers

  • 13
    Kai on March 26, 2021 Reply

    Hi Jeddrey

    Thanks for the excellent post. One thing I did not get how my jump box could reach vcenter in Esxi-MGMT PG with no connection to VyOS.
    And it’s the same for the connection between
    Nested-Esxi-PG(vlan4095) to Esxi-MGMT-PG(vlan1611). I don’t recall vswitch has valn routing abilities between different PGs. I might add a third NIC to VyOS and connect that to Esxi-MGMT-PG. What do you think of that.? am I missing something here. Basically, i am not getting how Esxi-MGMT-PG connects to Nested-Esxi-PG and my JumpBox in the Home network.

    Thanks in advance.

    • 14
      Jeffrey Kusters on March 29, 2021 Reply

      The VyOS router routes between my home network and my management VLAN (vlan 10). My jumpbox is connected to my home network by the way and not to the management network. So, its just a routed connection between my home network and my management network. I’m not sure I understand your issue…

  • 15
    gr8matt on May 5, 2021 Reply

    Jeffrey, I cannot thank you enough for your time for these detailed write-ups. I have been working in a lab for the last couple weeks and spend a lot of time studying your deployment! Thanks so much for taking the time to post this and help out the community! Your time is appreciated!

    • 16
      Jeffrey Kusters on May 6, 2021 Reply

      You’re welcome and thank you for taking the time to leave this note. Much appreciated!

Leave a Reply