Hyper-V over SMB: SMB Direct (RDMA)

Another important part of SMB 3.0 and Hyper-V over SMB is the performance. In the past you could use iSCSI, Fiber Channel or FCoE (Fiber Channel over Ethernet). Now SMB 3.0 has a lot of performance improvements to make the Hyper-V over SMB scenario even work. But if you need even more performance you can use new feature which came with Windows Server 2012 and is of course also present in Windows Server 2012 R2 called SMB Direct, which supports the use of network adapters that have Remote Direct Memory Access (RDMA) capability.  Network adapters with RDMA offer some great enhancements such as very low latency, increased throughput and low CPU utilization since the functionality is offloaded to the network card.

Advantages

• Increased throughput: Leverages the full throughput of high speed networks where the network adapters coordinate the transfer of large amounts of data at line speed.
• Low latency: Provides extremely fast responses to network requests, and, as a result, makes remote file storage feel as if it is directly attached block storage.
• Low CPU utilization: Uses fewer CPU cycles when transferring data over the network, which leaves more power available to server applications.

(Source TechNet)

Technology and Requirements

At the moment there are different versions of network adapters with RDMA capabilities, currently these are iWARP, InfiniBand or RoCE.

• iWARP, is a simple solution which does not really need any more configuraiton
• InfiniBand,
• RoCE (RDMA over Converged Ethernet), which needs also Switches to be configured in the right way for bandwidth management (DCB/PFC)

On the software side you need Windows Server 2012 or Windows Server 2012 R2 with SMB 3.0. SMB Direct is not supported in previous versions of SMB and Windows Server.

Setup of SMB Direct

Well SMB Direct or RDMA if oyu will is enabled by default, so Windows Server will make use of it when ever possible. But there are some things you have to make sure.

• Which type of RDMA am I using, is it iWARP, InfiniBand or RoCE. Some of them maybe require additional configuration on the network. If you are using RoCE RDMA seems to work without configuration but you can run into performance issues as my fellow Microsoft MVP Didier van Hoye descripes in his blog post.
• Install the latest NIC drivers
• Install the latest firmware
• Enable SMB Multichannel if you disabled it. SMB Direct will be also disabled when you disable Multichannel.
• In a Failover Cluster make sure that the RDMA NICs are also marked as client access adapters.
• SMB Direct doesn’t work with NIC Teaming or Virtual Switches
• On the file server you should also tune performance by disabling hyper threading, Disabling processor C States and setting the power profile to full power.

Verify SMB configuration

Verify if RDMA is enabled, first cmdlet checks if it’s enabled on the server it self, second one checks if it’s enabled on the network adapters and the third checks if the hardware is RDMA capable.

 
Get-NetOffloadGlobalSetting | Select NetworkDirect
Get-NetAdapterRDMA
Get-NetAdapterHardwareInfo

Verify that SMB Multichannel is enabled, which confirms the NICs are being properly recognized by SMB and that their RDMA capability is being properly identified.

On the client:

 
Get-SmbClientConfiguration | Select EnableMultichannel
Get-SmbClientNetworkInterface

On the server:

 
Get-SmbServerConfiguration | Select EnableMultichannel
Get-SmbServerNetworkInterface
netstat.exe -xan | ? {$_ -match "445"}

And as already mentioned in the SMB Mutlichannel blog post, you can verify the SMB connections:

 
Get-SmbConnection
Get-SmbMultichannelConnection
netstat.exe -xan | ? {$_ -match "445"}

And of course you have some great performance counters.

If you run some copy jobs you can see the amazing performance (if your storage is fast enough). Here you can also see a print screen with Mellanox ConnectX-3 Ethnernet adapters which are using RoCE in Windows Server 2012. You can see that you don’t see any TCP traffic in the Task Manager on the RDMA NICs.

Get more information about SMB Direct on Jose Barretos blog SMB3.info or on Microsoft TechNet.