How to Build a VMware Homelab – Step-By-Step Tutorial Series: 2. ESXi Nested Homelab Install
Having a VMware homelab is key to learning products like vSphere and NSX. In previous blog posts and videos on YouTube, I provided partial walkthroughs of my ESXi Nested Homelab environment running vSphere and NSX. I realize these do not fully explain how to build and configure everything, and there are some inconsistencies. I’m going to wipe my entire homelab and rebuild it step-by-step, and I will try to cover everything in 10 blogposts and 10 YouTube videos. Consider this an ultimate beginner’s tutorial on setting up a fully nested VMware homelab. These blog posts and videos are targeted at people new to VMware vSphere or NSX, so I will try to explain everything fully, and as simple as possible.
1. Installing VMware ESXi using the USB drive
First, we must install ESXi on our physical server. We are going to use the ESXi installation USB drive we created in part 1. Simply plug it in your physical server and make sure to boot from it. For my SuperMicro Superserver, I’m using the IPMI interface to access the remote console. Pressing F11 brings me to the boot menu where I can select the USB drive.
Installing ESXi is pretty straightforward. Just follow the ESXi installation wizard:
In my previous homelab build, I booted from a USB drive and used my two SDD drives 100% as a VMFS datastore. As I explained in the previous part, booting from removable media is no longer a recommended practice. This time, I decided to install ESXi on my NVMe Samsung SSD 960 Pro drive. The spare capacity will be used as a VMFS datastore.
The ESXi installation wizard warns me about the VMFS datastore currently present on the NVMe drive. Installing ESXi to this disk will destroy the VMFS datastore and all data/VMs on it. Be 100% sure you are installing ESXi to the correct disk. I bet every VI Admin has some kind of horror story about accidentally wiping a VMFS datastore by selecting the wrong disk. Not me, though. No, that never happened to me. Really … it didn’t. Again, be 100% sure and press OK.
Choose the keyboard layout:
Set your root password:
Are you really, really, really sure you selected the correct disk?
And that’s it. ESXi is being installed.
Remove the USB drive and reboot the ESXi server:
Press F2 to configure the ESXi Server:
2. Initial ESXi Configuration on the ESXi Console
The next step is to configure the Management network. I also like to enable the ESXi shell and SSH. In production environments, this is typically not recommended because it is a potential security risk, but in a homelab environment, I think this is perfectly acceptable and handy.
My physical ESXi server is connected to my home network with subnet 192.168.2.0/254. This is just my simple and flat home network with a single subnet and a default gateway (192.168.2.254/24) on my ISP router. I’m setting a static IP address:
Once we get our supporting infrastructure services (i.e. a domain controller with DNS) up and running, this will change, but for now my DNS server is my Pi-hole Raspberry Pi with IP address 192.168.2.7:
Set the DNS suffix:
To enable the ESXi shell and SSH we go to Troubleshooting options:
Enable ESXi Shell and SSH:
You can now use your web browser to connect to the Web UI. For me, the URL is https://192.168.2.35/ui/.
3. ESXi Nested Homelab Network Configuration
First, we create a new vSwitch without any uplinks/physical adapters connected. This is a 100% internal vSwitch. It will be used to support the ESXi nested homelab. I call it Nested-ESXi, and we need to set some specific settings to support nested virtualization:
The MTU size needs to be 9000 bytes so we can use jumbo frames for vMotion, vSAN and – most important – support the Geneve overlay protocol used by VMware NSX.
To support nested virtualization, we need to change the security settings to accept ‘Promiscuous mode’, ‘MAC address changes’ and ‘Forged transmits’.
Next, we need to create two Virtual Machine port groups:
- Nested-ESXi-Trunk
- Nested-ESXi-Management
The Nested-ESXi-Trunk port group needs to be configured with VLAN 4095. Using this VLAN ID sets the port group into VLAN trunking mode. See https://kb.vmware.com/s/article/1004252 for more details. This is the crucial step in our nested homelab build that will allow us to use VLANs inside of our nested homelab environment. The port group will transport all VLANs, and we can tag the VLANs on our guest operating systems/VMs.
Eventually, we are going to run management workloads, such as vCenter Server, NSX Manager, and so on in our ESXi nested homelab. I use VLAN 10 as my management network. If we are going to run these appliances on the physical server, we need to create a Virtual Machine port group with VLAN 10. That’s the Nested-ESXi-Management port group. Make sure to put the correct VLAN ID in:
Your Nested-ESXi vSwitch should now look like this. No physical adapters/uplinks. One Virtual Machine Port Group with VLAN ID 10 for Nested-ESXi-Management and one Virtual Machine Port Group with VLAN ID 4095, MTU 9000 and all security parameters set to ‘Accept’.
The other vSwitch (vSwitch0) is used to connect the physical ESXi server to my home network. It uses one physical uplink (vmnic0) and has the ESXi management vmkernel interface connected with IP address 192.168.2.35/24. It also holds a Virtual Machine Port Group to connect VMs directly to my home network. My Domain Controller, for example, also functions as a Windows jumphost, so it is connected to my physical home network (192.168.2.0/24). vSwitch0 should look like this:
4. Deploying Nested ESXi servers
With the network fully configured, we can now deploy our nested ESXi servers. vCommunity rockstar William Lam hosts a ton of valuable information on nested virtualization on his blog. Specifically, he also offers Nested ESXi Virtual Appliances. These are preconfigured OVA templates you can download and use to deploy nested ESXi servers easily. I grabbed the latest version:
To deploy the Nested ESXi server, simply click Create/Register VM and select Deploy a virtual machine from an OVF or OVA file.
Follow this wizard and select the Nested ESXi OVA file you just downloaded and configure the settings specific to your environment:
Once deployed, edit the settings of the nested ESXi server to modify the allocated resources. I’m allocating 2 CPUs and 32GB RAM per Nested ESXi server. I also increased the capacity of the Hard Disks. Hard Disk 1 is used as the system disk for ESXi. Hard Disks 2 and 3 will be used for vSAN, with Hard Disk 2 being the performance/flash drive and Hard Disk 3 being the capacity drive. When we have vCenter up and running, I will probably Storage vMotion the capacity drive to my SSD850 drive because I have a bit more capacity there.
Also, ensure your Network Adapters are connected to the Nested-ESX-Trunk port group:
After booting your first Nested ESXi server, you can configure it just like you configured your Physical ESXi server. You can use the Remote Console of the Web UI to access the ESXi console. I’m using VLAN 10 with subnet 10.0.10.0/24 as my Nested-ESXi-Management network. The ESXi management vmkernel interface will connect to this VLAN. Because we used VLAN ID 4095 on the Nested-ESXi-Trunk port group, we can do VLAN tagging on ESXi:
Configure the rest of the network settings and enable the ESXi shell and SSH if you like:
Now, repeat this process for your other Nested ESXi servers. I’m going to run a 3-node vSAN cluster, so I deployed two additional Nested ESXi servers.
4. Supporting infrastructure services on a Windows Domain Controller
We don’t have any supporting infrastructure services available yet, such as DNS or Active Directory. I deployed a Windows Domain Controller on my physical ESXi server, connected to my Home-Network Virtual Machine Port Group. This Windows Server will also function as a jump host between my physical home network and my nested homelab environment.
I really want to focus on the VMware side of things, so I’m not providing a detailed overview of how to deploy a Windows Server with AD, DNS and RDP enabled.
Closing words
If you followed along, you will have a Windows Server connected to your home network, and 3 ESXi servers connected to VLAN 10. There is no connectivity available yet between these two networks because we have no routing in place yet. That will be the topic for the next part of this series. We will deploy a Vyos virtual appliance with an uplink interface into the physical home network and a downlink interface connected to the Nested-ESX-Trunk port. We can then configure virtual interfaces, or VIFs, for every VLAN we need in our nested homelab. I will also include a network diagram to help you better understand the topology we are building.
See you next time!