Tag: Live Migration

Azure Live Migration

Azure uses Live Migration for VMs

If you have worked with Azure in the past, you might have been aware that Azure didn’t have live migration for VMs hosted in Azure for a long time. This had an impact for customers in terms of VM up-time during host maintenance. You basically got emails, that the host your VMs were running is going into maintenance during a specific time, and you will have a possible outage. Microsoft Hyper-V, which is the Hypervisor in Azure, had Live Migration for a long time. Today, Microsoft revealed that they are using Live Migration in Azure since early 2018 to move virtual machines in cases of rack maintenance and software and BIOS updates, as well as hardware faults.

But Microsoft didn’t stop there, they made even better using Machine Learning. Predictive ML helps Microsoft to detect proactively failure and do failure predictions. And in case a hardware failure is predicted, Microsoft can move the virtual machines from that host without downtime, using live migration.

To further push the envelope on live migration, we knew we needed to look at the proactive use of these capabilities, based on good predictive signals. Using our deep fleet telemetry, we enabled machine learning (ML)-based failure predictions and tied them to automatic live migration for several hardware failure cases, including disk failures, IO latency, and CPU frequency anomalies.


We partnered with Microsoft Research (MSR) on building our ML models that predict failures with a high degree of accuracy before they occur. As a result, we’re able to live migrate workloads off “at-risk” machines before they ever show any signs of failing. This means VMs running on Azure can be more reliable than the underlying hardware.

Microsoft talks in a blog post more about Live Migration in Azure and goes more in details about the challenges and how live migration in Azure works. It is great to see Microsoft adding features to improve VM resiliency with features like live migration and machine learning technology.

Windows Server 2012 R2

What’s new in Windows Server 2012 R2 Hyper-V

Today Microsoft announced the new features which are coming in Windows Server 2012 R2 which will be the next version of Windows Server at Microsoft TechEd North America. By the way just to show you how great Windows Server 2012 was and how great it scaled, Windows Azure uses the same Hyper-V virtualization service built-into Windows Server 2012 and this means complete virtual machine compatibility between on premise Hyper-V and Windows Azure IaaS. This blog post shows what’s new in Windows Server 2012 R2 Hyper-V.

Here the next version names:

  • Windows Server 2012 R2
  • System Center 2012 R2
  • Windows 8.1

Now Microsoft announced a lot of new features especially for Hyper-V, and here are some of them:

  • Shared VHDX – a VHDX can now be shared between two Virtual Machine by using the virtual SCSI controller. This is created if you need shared storage for guest clustering inside virtual machines instead of using iSCSI or virtual fiber channel.
  • Live Migration Compression – Live Migration traffic will be compressed by the Hyper-V host before it’s sent over the wire. Which does reduce Live Migration time dramatically, up to 50% faster.
  • Live Migration over SMB Direct (RDMA) – Live Migration can use leverage SMB 3.0 and this means it can also make use of SMB Direct or RDMA which allows you to do live migration even faster.
  • Storage Quality of Service (QoS) – Limit storage IOPS per virtual machine
  • Live Virtual Machine Cloning / Exporting – You can now live clone a virtual machines without downtime and also export a running virtual machine.
  • Linux Guest OS support enhancements – Support for live backups of linux virtual machines and dynamic memory support for Linux guests.
  • Hyper-V Replica 2.0 – Hyper-V replica can now replicate not just two one other host, this replica can also replicate to a third Hyper-V host and the replication time was changed to three different settings (every 30 secs, every 5 minutes or every 15 minutes). Hyper-V Replica also got some background scalability and performance improvements.
  • Windows Azure Compatibility – As I already mentioned Windows Azure is running Windows Server 2012 Hyper-V which means that Hyper-V virtual machines on-premise are also capable to run in Windows Azure
  • Online resizing of VHDX – You can expand and shrink VHDX files during the virtual machine is running.
  • Automatic Guest Activation – zero touch activation of virtual machines. Virtual machines automaticly get activated if the Hyper-V hosts is an activated Datacenter edition.
  • VM Connect using RDP or enhanced VM interaction – This uses Remote Desktop over the VMBus, which allows you to use full remote desktop capabilities (Shared clipboard, audio redirection, folder redirection, smartcards, USB pass-through enhanced login and more…)
  • Generation 2 virtual machines – Gen2 VMs are legacy free and based on UEFI. So this means no more emulated devices, boot from virtual SCSI controllers or synthetic network adapters (PXE boot >100MBit) and enables UEFI secure boot as a standard. Supported guest operating systems: 64-bit versions of Windows 8, Windows Server 2012, Windows 8.1 and Windows Server 2012 R2.
  • Zero-downtime upgrade (Cross version live Migration) – Live migrate virtual machines from Windows Server 2012 to Windows Server 2012 R2 (this also includes shared-nothing live migration).
  • Hyper-V Recovery Manager – I already mentioned the new service called Hyper-V Recovery Manager in Windows Azure which allows you to run a orchestrator failover of your virtual machines using Hyper-V Replica.
  • Deduplication – Deduplication of VDI Virtual Machines

There are a lot of other cool features in Windows Server 2012 R2 which add other great value to Hyper-V and your Private cloud. I will cover them in some other blog posts in the next days.

Windows Server 2012 Hyper-V: Configure VM Processor Compatibility via PowerShell

Windows Server 2012 Logo

If you have a mixed Hyper-V environment with Hyper-V hosts which have different CPU generations you may cannot live migration Virtual Machines from one Hyper-V host to another without setting the VM CPU Compatibility setting.

Note: I saw a lot of different environments where different generation of Intel CPUs were used, and most of the time Live Migration did still work. I would recommend to set this setting only if you really need it.

Hyper-V VM Processor Compatibility

If you need to do this for multiple Virtual Machines you may better use Windows PowerShell to do this.

To do this you can use the new Hyper-V PowerShell module and the Get-VMProcessor and Set-VMProcessor cmdlet. With the Get-VMProcessor cmdlet you can get information about the virtual CPU of VMs. You can list this for all virtual machines running on one host by using the following command

Get-VMProcessor *

Hyper-V Get-VMProcessor

To change this setting for all virtual machines running on the Hyper-V host you can use the following PowerShell command. This does only working with virtual machines which are not running.

Get-VMProcessor * | Set-VMProcessor -CompatibilityForMigrationEnabled 1

Hyper-V Set-VMProcessor

If you want to shut down all the VMs first you can use the following command.

Get-VM | Stop-VM

If you just want to do this for a single VM you can use the following commands.

Get-VM TMG | Stop-VM
Get-VM TMG | Set-VMProcessor -CompatibilityForMigrationEnabled 1

Hyper-V Set-VMProcessor Single VM

To do a full automation you could do it the following way:

Get-VM * | Stop-VM
Get-VM * | Set-VMProcessor -CompatibilityForMigrationEnabled 1
Get-VM * | Start-VM

As you can see you can also find the “CompatibilityForOlderOperatingSystemsEnabled” property to enable Processor Compatibility for legacy Operating Systems in Windows Server 2012 Hyper-V for example Windows NT 4.0 Server or Windows 2000.

Microsoft Surface Pro meets Hyper-V

This happens if you build your datacenter based on the Microsoft Surface Pro. I just saw a tweets from Jeff Woolsey and Ben Armstrong about how you install Windows Server 2012 Hyper-V on the Surface Pro and are running some Live Migrations of virtual machines.

Checkout the blog post or the short video:

Windows Server 2012 Hyper-V Converged Fabric

Windows Server 2012 RC Logo

In Windows Server 2008 R2 we had some really simple configurations and best practices for Hyper-V and network configurations. The problem with this was, that this configurations were not really flexible. This had two main reasons, first NIC teaming wasn’t officially supported by Microsoft and secondly there was no possibility to create virtual network interfaces without third party solution.

Here is a example of a Hyper-V 2008 R2 host design which was used in a cluster setup.

Traditional Design

traditional Hyper-V Host

Each dedicated Hyper-V network such as CSV/Cluster communication or the Live Migration network used a own physical network interface. The different network interfaces could also be teamed with third party software from HP, Broadcom or Intel. This design is still a good design in Windows Server 2012 but there are other configurations which are a lot more flexible.

Microsoft MVP Adian Finn and Hans Vredevoort did a already some early work with Windows Server 2012 Converged Fabric and you should definitely read their blog posts.

In Windows Server 2012 you can get much more out of your network configuration. First of all NIC Teaming is now integrated and supported in Windows Server 2012 and another cool feature is the use of virtual network adapters in the Management OS (Host OS or Parent Partition). This allows you to create for example one of the following designs.

Virtual Switch and Dedicated Management Interfaces

Hyper-V Converged Fabric

This scenario has two teamed 10GbE adapter for Cluster and VM traffic.

Virtual Switch and Dedicated Teamed Management Interfaces

Hyper-V Converged Fabric

The same scenario with a teamed management interface.

Dedicated Virtual Switch for Management and VM Traffic

Hyper-V Converged Fabric

One Virtual Switch for Management and Cluster traffic and a dedicated switch for VM traffic.

One Virtual Switch for everything

Hyper-V Converged Fabric

This is may favorite design at the moment. Two 10GbE adapter as one team for Virtual Machine, Cluster traffic and management. It is a very flexible design and allows the two 10GbE adapters to be used very dynamic.

This design solutions will also be very interesting if you us SMB 3.0 as a storage for Hyper-V Virtual Machines.

FileServer and Hyper-V Cluster


There are at the moment not a lot of official information which designs will be unsupported and which will be supported. You can find some information about supported designs in the TechEd North America session WSV329 Architecting Private Clouds Using Windows Server 2012 by Yigal Edery and Joshua Adams.


Now after you have seen these designs you may want to create such a configuration and want to know how you can do this. Not everything can be done via GUI you have to use your Windows PowerShell skills. In this scenario I use the design with four 10GbE network adapters 2 for iSCSI and to for my network connections.

  • Install the Hyper-V Role
  • Create NIC Teams
  • Create a Hyper-V Virtual Switch
  • Add new Virtual Network Adapters to the Management OS
  • Set VLANs of the Virtual Network Adapters
  • Set QoS Policies of the Virtual Network Adapters
  • Configure IP Addresses of the Virtual Network Adapters

Install Hyper-V Role

Before you can use the features of the Virtual Switch and can start create Virtual Network Adapters on the Management OS (Parent Partition) you have to install the Hyper-V role. You can do this via Server Manager or via Windows PowerShell.

 Add-WindowsFeature Hyper-V -Restart

Create NIC Teams

Now most of the time you will create a NIC Teaming for fault tolerance and load balancing. A team can be created over the Server Manager or PowerShell. Of course I prefer the Windows PowerShell. For a Team which will not only be used for Hyper-V Virtual Machines but also for Management OS traffic I use the TransportPorts as load balancing algorithm. If you use this team only for Virtual Machine traffic there is a algorithm called Hyper-V-Port. The Teaming Mode of course depends on your configuration.

 New-NetLbfoTeam -Name Team01 -TeamMembers NIC1,NIC2 -LoadBalancingAlgorithm HyperVPort -TeamingMode SwitchIndependent

NIC Teaming


Create the Virtual Switch

After the team is created you have to create a new Virtual Switch. We also define the DefaultFlowMinimumBandwidthWeight to be set to 20.

 New-VMSwitch -Name VMNET -NetAdapterName Team01 -AllowManagementOS $False -MinimumBandwidthMode Weight
Set-VMSwitch "VMNET" -DefaultFlowMinimumBandwidthWeight 3.

VM Switch


After you have created the Hyper-V Virtual Switch or VM Switch you will find this switch also in the Hyper-V Manager.

Hyper-V Virtual Switch

 Create Virtual Network Adapters for the Management OS

After you have created your Hyper-V Virtual Switch you can now start adding VM Network Adapters to this Virtual Switch. We also configure the VLAN ID and the QoS policy settings.

 Add-VMNetworkAdapter -ManagementOS -Name "Management" -SwitchName "VMNET"
Add-VMNetworkAdapter -ManagementOS -Name "LiveMigration" -SwitchName "VMNET"
Add-VMNetworkAdapter -ManagementOS -Name "CSV" -SwitchName "VMNET"
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "Management" -Access -VlanId 185
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "CSV" -Access -VlanId 195
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "LiveMigration" -Access -VlanId 196
Set-VMNetworkAdapter -ManagementOS -Name "LiveMigration" -MinimumBandwidthWeight 20
Set-VMNetworkAdapter -ManagementOS -Name "CSV" -MinimumBandwidthWeight 10
Set-VMNetworkAdapter -ManagementOS -Name "Management" -MinimumBandwidthWeight 10

VMNetworkAdapter ManagementOS


Your new configuration will now look like this:

Network Connections

As you can see the name of the new Hyper-V Virtual Ethernet Adapter is vEthernet (NetworkAdapaterName). This will be important for automation tasks or configuring IP addresses via Windows PowerShell.

Set IP Addresses

Some months ago I wrote two blog posts, the first was how to configure you Hyper-V host network adapters like a boss and the second one was how to replace the netsh command with Windows PowerShell. Now using Windows PowerShell to configure IP addresses will save you a lot of time.

# Set IP Address Management
New-NetIPAddress -InterfaceAlias "vEthernet (Management)" -IPAddress -PrefixLength "24" -DefaultGateway
Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Management)" -ServerAddresses,
# Set LM and CSV
New-NetIPAddress -InterfaceAlias "vEthernet (LiveMigration)" -IPAddress -PrefixLength "24"
New-NetIPAddress -InterfaceAlias "vEthernet (CSV)" -IPAddress -PrefixLength "24"
New-NetIPAddress -InterfaceAlias "iSCSI01" -IPAddress -PrefixLength "24"
New-NetIPAddress -InterfaceAlias "iSCSI02" -IPAddress -PrefixLength "24"


There is still a lot more about Windows Server 2012 Hyper-V Converged Fabric in the future, but I hope this post will give you a quick insight into some new features of Windows Server 2012 and Hyper-V.

Hyper-V 2012 – Hey I Just Met You And This Is Crazy

Windows Server 2012 RC Logo

Okay I admit it, the title is more a reference to a song than a true fact, because my first contact with the latest Hyper-V release was last September.

Last September Microsoft showed the newest release of Hyper-V at the build conference. Back then I wrote a blog post about the new version of Microsoft Hypervisor Hyper-V called “Hyper-V: Version 3 kills them all“.

Now Microsoft released the Windows Server 2012 and Microsoft Hyper-V 2012 Release Candidate last week. They made a lot of changes since September 2011 and I tried to show this in another blog post (What’s new in Windows Server 2012 Release Candidate Hyper-V: Scale). With these changes I decided to upgrade my blog post from September 2011 with the latest changes made with Windows Server 2012 Release Candidate.

Windows Server 2012 the Cloud OS

Windows Server 2012 Management

First let’s start with Windows Server 2012 as the base of Microsoft cloud strategy. Windows Server 2012 is probably the most significant release of the Windows Server platform ever. Microsoft’s focus in Windows Server 2012 was to make it easy for all to build public, private or hybrid cloud solutions. Microsoft has used the experience and learning from their own Cloud services like Hotmail, Messenger, Office 365, Bing, Windows Azure, and Xbox Live. There are a lot of improvements to manageability, security, scalability, extensibility, predictability and reliability which will also improve the possibilities with Hyper-V. The Power of Many, The Simplicity of One – In technical terms Microsoft made a lot of improvements how you can manage a lot of servers and services, Storage, Networking and PowerShell. Of course there is a lot more, but this are the parts I think are the most important. And here are some keywords to the improvements in Windows Server 2012:

  • Storage improvements – SMB 3.0, SMB transparent Failover, data de-duplication, Storage Spaces, online filesystem repairs, 64TB NTFS volumes, ReFS volumes, etc.
  • NIC Teaming
  • NIC Naming and CDN (consistent device naming)
  • PowerShell v3 – You can now just do everything in PowerShell and even more with 23000 PowerShell cmdlets.
  • Server Dashboard – The new Dashboard lets you manage all servers, or even better, all Services from one place.
  • Multi-tenant – everything seems to be made for that
  • Performance Counters
  • IP address management (IPAM)

Hyper-V Host improvements

Windows Server 2012 Hyper-V Manager

The Hyper-V Host gets a lot of improvements in terms of features and scale.

  • up to 320 logical CPUs
  • supports up to 4 TB RAM
  • no more vCPU:pCPU ration limit
  • up to 2048 Virtual CPUs per Virtual Machine

Hyper-V Virtual Machine improvements

Windows Server 2012 Hyper-V Virtual Machine

Microsoft did a lot to extend the existing Virtual Machine hardware to support even high workload Virtual Machines. Most of the time you were talking with VMware consultants, they tried to say that Hyper-V is not made for Enterprise workloads. Now with the release of Hyper-V 2012 scale should not be an argument anymore.

Hyper-V Networking improvements

Windows Server 2012 Hyper-V Networking

Hyper-V got a lot of improvements in terms of networking. Microsoft realized that networking features are really important if you start to create private and public cloud scenarios and now even create a mix of public and private cloud scenarios without creating a lot of work for the IT teams to reconfigure Virtual Machines.

  • QoS and flexible bandwidth allocation
  • Single-root I/O virtualization or SR-IOV (Direct Access to the physical Network adapter)
  • Network Virtualization
  • PVLAN support
  • Dynamic Virtual Machine Queue (D-VMQ)
  • Receive Side Coalescing (RSC)
  • DHCP Guard
  • Router Guard
  • Port mirroring
  • Port ACLs
  • Trunk mode Allows directing traffic from a group of VLANs to a specific VM
  • IPsec Task offload
  • Integrated Network Adapter Teaming
  • Better Network Adapter Naming and Consistent Network Device Naming
  • Hyper-V Extensible Switch (for example Cisco Nexus 1000v)
  • Data Center Bridging (DCB) – eliminates loss due to queue overflow and to be able to allocate bandwidth on links
  • Network Metering

Hyper-V Clustering improvements

Hyper-V gets also a lot of Cluster improvements. Microsoft is working on Cloud solutions which will give great availability to low cost. For example Hyper-V Replica or Shared-Nothing Live Migration, which allows you to move a Virtual Machine from one Hyper-V host to another host over the Ethernet without the need for a shared storage or a Cluster.

  • supporting up to 4000 VMs per cluster
  • supporting up to 64 Cluster nodes
  • improved Cluster Manager Console
  • Application Monitoring – Application health detection inside the virtual machine
  • New Placement policies – Virtual Machine Priority and enhanced placement
  • Storage Live Migration
  • VM Failover Prioritization
  • Cluster Wide Task Scheduling
  • Hyper-V Replica supporting clustering – replicate a Virtual Machine from one Cluster to another Cluster or Standalone Hyper-V Host
  • No need for Block Storage – you can use SMB Shares
  • Support for Storage Spaces
  • Automated Node Draining – like Maintenance mode in SCVMM
  • Cluster Aware Updating (CAU)
  • Cluster Shared Volume Improvements – BitLocker support, a lot of performance improvements, Self-Healing
  • CSV Block Cache
  • CSV 2.0 (No Redirected I/O for Backup
  • CSV 2.0 Block Level I/O redirection
  • CSV enabled volumes now appear as “CSVFS”
  • No Active Directory dependencies
  • Live Migration Queuing
  • Migrate multiple Virtual Machine at the same time
  • Anti-Affinity VM Rules
  • Dynamic Quorum
  • Guest Clustering via ISCSI, SMB or Fibre Channel

Hyper-V Storage improvements


A I mentioned earlier Microsoft made a lot of improvements in terms of storage in Windows Server 2012 and Hyper-V can take advantage of those which are quiet impressive. For example with the new features in SMB 3.0 you can now use SMB file shares to store your Virtual Machines.

  • New Virtual Disk format (VHDX supports up to 64 TB Virtual Disks)
  • Offloaded Data Transfer – ODX (Open Diagnostic Data Exchange)
  • Live merging of VHDs and Snapshots
  • RDMA
  • SMB 3.0 – Transparent Failover
  • SMB 3.0 Direct
  • SMB 3.0 Multichannel
  • Native 4 KB sector disks support
  • Data De-duplication
  • Virtual Fibre Channel inside the Virtual Machines
  • VM boot from SAN
  • Storage Spaces (Pool Disks or LUNs)
  • New File system ReFS

Hyper-V Management Improvements

Hyper-V Powershell

As everywhere in Windows Server 2012 PowerShell is the key. And the new Server Manager Dashboard Microsoft enables to create Server Groups to manage multiple servers from a single console.

  • PowerShell cmdlets for Hyper-V
  • PowerShell Workflows – commands and tasks across servers
  • Hyper-V Extensible Switch – lets vendors to create “plugins”.
  • Server Manager Dashboard – lets you manage multiple Hyper-V hosts from a single console.
  • System Center Virtual Machine Manager 2012 SP1 – not a part of Windows Server 2012 but will add great management solutions.
  • Improved VM Import
  • Local Hyper-V Administrator Group
  • Client Hyper-V

Hyper-V Live Migration and Disaster Recovery

Windows Server 2012 Hyper-V Replica

Now I think this is maybe the greatest new feature. You can migrate Virtual Machines from one Hyper-V Host to another without Shared Storage or Cluster configuration. This feature is called Shared-Nothing Live Migration. Microsoft also included a new feature called Hyper-V Replica which includes the option to replicate Virtual Machine to another host which can be hosted in the same datacenter, secondary datacenter or even in the cloud.

  • Improved Live Migration
  • Unlimited Simultaneous live migrations
  • Live Storage Migration
  • Shared-Nothing Live Migration – Live Migration to another Hosts (Not clustered) over Ethernet
  • Hyper-V Replica – Replicated Virtual Machines to another Hyper-V host on-premise or public cloud over LAN or WAN connections.

You can get more information and the download link about Windows Server 2012 Release Candidate and Hyper-V Server 2012 Release Candidate.

The Windows Server Team and especially the Hyper-V Team did a great job, and I am sure Hyper-V will gain significant market share in the future.

d-on-d – Datacenter On Demand


This week I could present some of the new features in Windows Server 8 Hyper-V on an event organized by Microsoft Switzerland in a very cool location. I am talking about D-ON-D (Datacenter on demand) which is run by Kybernetika.

D-ON-D offers you to rent servers and rooms for events, workshops, test project or demos.

If you want to know more about what D-ON-D offers check out the websites: