VMware NVME PCIE Driver Missing on ESXi 7.0 after upgrade from 6.7 U3 on SuperMicro 5028D-TN4T (Updated)

Update April 15th 2020:

William Lam reached out and asked me to provide the vm-support log bundle. Apparently, I incorrectly used the esxcli command to upgrade my host. Thanks to William, I found out a bunch of other VIBs were also not properly upgraded yet. If you are doing a full upgrade, make sure you specify the proper profile name using:

esxcli software profile update -d </path/to/depot.zip> -p <profile name>

(You can list the profiles included in the depot with esxcli software sources profile list -d </path/to/depot.zip>)

I ran the proper upgrade command and the following VIBs were also updated:

Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMW_bootbank_brcmnvmefc_12.4.293.2-3vmw.700.1.0.15843807, VMW_bootbank_i40iwn_1.1.2.5-1vmw.700.1.0.15843807, VMW_bootbank_nvmerdma_1.0.0.0-1vmw.700.1.0.15843807, VMW_bootbank_qedrntv_3.12.1.2-12vmw.700.1.0.15843807, VMware_bootbank_crx_7.0.0-1.0.15843807, VMware_bootbank_loadesx_7.0.0-1.0.15843807, VMware_bootbank_lsuv2-nvme-pcie-plugin_1.0.0-1vmw.700.1.0.15843807, VMware_bootbank_lsuv2-oem-dell-plugin_1.0.0-1vmw.700.1.0.15843807, VMware_bootbank_lsuv2-oem-hp-plugin_1.0.0-1vmw.700.1.0.15843807, VMware_bootbank_lsuv2-oem-lenovo-plugin_1.0.0-1vmw.700.1.0.15843807, VMware_bootbank_vdfs_7.0.0-1.0.15843807

—————————————————————–

I decided to upgrade my SuperMicro 5028D-TN4T homelab to vSphere 7.0. I’m running a fully nested lab on a single, standalone ESXi host. After upgrading the physical host from ESXi 6.7 Update 3 to ESXi 7.0, the Samsung 960 Pro NVME disk I use for physical storage was missing. A quick look at the loaded drivers told me the VMware nvme-pcie driver was missing:

My homelab boots from an external USB drive, so I decided to perform a clean install on a new USB stick. The driver seemed to load fine on a fresh install of ESXi 7.0. Instead of troubleshooting this issue, I decided to take a quick shortcut and try to install the VIB manually on the upgraded system. The steps are pretty straightforward and are mostly documented here for my own reference…

I unzipped the “VMware-ESXi-7.0.0-15843807-depot.zip” offline upgrade bundle on my local machine and copied the driver “VMW_bootbank_nvme-pcie_1.2.2.13-1vmw.700.1.0.15843807.vib” located in the /vib20/nvme-pcie folder to /tmp of my ESXi 7.0 host using Filezilla:

Next step was SSH’ing into the host and placing it in Maintenance Mode:

esxcli system maintenanceMode set -e true

I installed the VIB using esxcli and rebooted the host.

esxcli software vib install –v /tmp/VMW_bootbank_nvme-pcie_1.2.2.13-1vmw.700.1.0.15843807.vib

! Remember to add the -v option to install a VIB

I rebooted the host and validated the proper installation of the VIB:

And finally the proper listing of the adapter:

Last step is to take the host out of Maintenance Mode:

esxcli system maintenanceMode set -e false

No idea why the driver wasn’t loading. On the positive side: I also updated my BIOS and firmware in the process to rule out any issues there. Next up is using William Lam’s awesome automation work to deploy vSphere 7.0 with Kubernetes! Thank you once again William!