How to setup Nested Virtualization in Microsoft Azure
At the Microsoft Build Conference this year, Microsoft announced Nested Virtualization for Azure Virtual Machines, and last week Microsoft announced the availability of these Azure VMs, which support Nested Virtualization. Nested Virtualization basically allows you to run a Hypervisor in side a Virtual Machine running on a Hypervisor, which means you can run Hyper-V within a Hyper-V Virtual Machine or within a Azure Virtual Machine, kind a like Inception for Virtual Machines.
You can use Nested Virtualization since Windows Server 2016 or the same release of Windows 10, for more details on this, check out my blog post: Nested Virtualization in Windows Server 2016 and Windows 10
With the release of the Azure Dv3 and Ev3 VM sizes:
- D2-64 v3 instances are the latest generation of General Purpose Instances. D2-64 v3 instances are based on the 2.3 GHz Intel XEON ® E5-2673 v4 (Broadwell) processor and can achieve 3.5GHz with Intel Turbo Boost Technology 2.0. D2-64 v3 instances offer the combination of CPU, memory, and local disk for most production workloads.
- E2-64 v3 instances are the latest generation of Memory Optimized Instances. E2-64 v3 instances are based on the 2.3 GHz Intel XEON ® E5-2673 v4 (Broadwell) processor and can achieve 3.5GHz with Intel Turbo Boost Technology 2.0. E2-64 v3 instances are ideal for memory-intensive enterprise applications.
With the upgrade to new Intel Broadwell processors, Microsoft enabled Nested Virtualization, which will allows a couple of different scenarios, when you create a Virtual Machine running Windows Server 2016.
- You can run Hyper-V Containers (Windows Containers with additional isolation) inside an Azure VM. With future releases we will also be able to run Linux Containers in Hyper-V Containers running on a Windows Server OS.
- You can quickly spin up and shut down new demo and test environments, and you only pay when you use them (pas-per-use)
How to Setup Nested Virtualization in Azure
Deploy Azure VM
To setup Nested Virtualization inside an Azure Virtual Machine, you first need to create a new Virtual Machines using one of the new instance sizes like Ev3 or Dv3 and Windows Server 2016.I also recommend to install all the latest Windows Server patches to the system.
Optional: Optimize Azure VM Storage
This step is optional, but if you want to better performance and more storage for your Nested Virtual Machines to run on, this makes sense.
In my case I attached 2 additional data disks to the Azure VM. Of course you can choose more or different sizes. Now you can see 2 new data disk inside your Azure Virtual Machine. Do not format them, because we gonna create a new storage spaces pool and a simple virtual disk, so we get the performance form both disks at the same time. In the past this was called disk striping.
With that you can create a new Storage Spaces Storage Pool and a new Virtual Disk inside the VM using the storage layout “Simple” which basically configures it as striping.
I also formatted the disk and set the drive letter to V:, this will be the volume where I will place my nested virtual machines.
Install Hyper-V inside the Azure VM
The next step would be to install the Hyper-V role in your Azure Virtual Machine. You can use PowerShell to do this since this is a regular Windows Server 2016.This command will install Hyper-V and restart the virtual machine.
1 | Install-WindowsFeature -Name Hyper-V -IncludeManagementTools -Restart |
After the installation you have Hyper-V installed and enabled inside your Azure Virtual Machine, now you need to configure the networking for the Hyper-V virtual machines. For this we will use NAT networking.
Configure Networking for the Nested Environment
To allow the nested virtual machine to access the internet, we need to setup Hyper-V networking in the right why. For this we use the Hyper-V internal VM Switch and NAT networking. I described this here: Set up a Hyper-V Virtual Switch using a NAT Network
Create a new Hyper-V Virtual Switch
First create a internal Hyper-V VM Switch
1 | New-VMSwitch -SwitchName "NATSwitch" -SwitchType Internal |
Configure the NAT Gateway IP Address
The Internal Hyper-V VM Switch creates a virtual network adapter on the host (Azure Virtual Machine), this network adapter will be used for the NAT Gateway. Configure the NAT gateway IP Address using New-NetIPAddress cmdlet.
1 | New-NetIPAddress –IPAddress 172.21.21.1 -PrefixLength 24 -InterfaceAlias "vEthernet (NATSwitch)" |
Configure the NAT rule
After that you have finally created your NAT network and you can now use that network to connect your virtual machines and use IP Address from 172.21.21.2-172.21.21.254.
1 | New-NetNat –Name MyNATnetwork –InternalIPInterfaceAddressPrefix 172.21.21.0/24 |
Now you can use these IP Addresses to assign this to the nested virtual machines. You can also setup a DHCP server in one of the nested VMs to assign IP addresses automatically to new VMs.
Optional: Create NAT forwards inside Nested Virtual Machines
To forward specific ports from the Host to the guest VMs you can use the following commands.
This example creates a mapping between port 80 of the host to port 80 of a Virtual Machine with an IP address of 172.21.21.2.
1 | Add-NetNatStaticMapping -NatName "MyNATnetwork" -Protocol TCP -ExternalIPAddress 0.0.0.0 -InternalIPAddress 172.21.21.2 -InternalPort 80 -ExternalPort 80 |
This example creates a mapping between port 82 of the Virtual Machine host to port 80 of a Virtual Machine with an IP address of 172.21.21.3.
1 | Add-NetNatStaticMapping -NatName "MyNATnetwork" -Protocol TCP -ExternalIPAddress 0.0.0.0 -InternalIPAddress 172.16.0.3 -InternalPort 80 -ExternalPort 82 |
Optional: Configure default Virtual Machine path
Since I have created an extra volume for my nested virtual machines, I configure this as the default path for Virtual Machines and Virtual Hard Disks.
1 | Set-VMHost -VirtualHardDiskPath V:\VMs -VirtualMachinePath V:\VMs |
Create Nested Virtual Machines inside the Azure VM
Now you can basically start to create Virtual Machines inside the Azure VM. You can for example use an existing VHD/VHDX or create a new VM using an ISO file as you would do on a hardware Hyper-V host.
Some crazy stuff to do
There is a lot more you could do, not all of it makes sense for everyone, but it could help in some cases.
- Running Azure Stack Development Kit – Yes Microsoft released the Azure Stack Development Kit, you could use a large enough Azure virtual machine and run it in there.
- Configure Hyper-V Replica and replicate Hyper-V VMs to your Azure VM running Hyper-V.
- Nested a Nested Virtual Machine in a Azure VM – You could enable nesting on a VM running inside the Azure VM so you could do a VM inside a VM inside a VM. Just follow my blog post to created a nested Virtual Machine: Nested Virtualization in Windows Server 2016 and Windows 10
In my opinion Nested Virtualization is mostly help full if you run Hyper-V Containers, but it also works great, if you want to run some Virtual Machines inside a Azure VM, for example to run a lab or test something.
