Tag: RoCE

Cisco UCS Hardware

Cisco UCS supports RoCE for Microsoft SMB Direct

As you may know we use SMB as the storage protocol for several Hyper-V deployments using Scale-Out File Server and Storage Spaces which adds a lot value to your Hyper-V deployments. To boost performance Microsoft is using RDMA or SMB Direct to accelerate Storage network performance.

RDMA over Converged Ethernet (RoCE) allows direct memory access over an Ethernet network. RoCE is a link layer protocol, and hence, it allows communication between any two hosts in the same Ethernet broadcast domain. RoCE delivers superior performance compared to traditional network socket implementations because of lower latency, lower CPU utilization and higher utilization of network bandwidth. Windows Server 2012 and later versions use RDMA for accelerating and improving the performance of SMB file sharing traffic and Live Migration. If you need to know more about RDMA or SMB Direct checkout my blog post: Hyper-V over SMB: SMB Direct

With Cisco UCS Manager Release 2.2(4), Cisco finally supports RoCE for SMB Direct. It sends additional configuration information to the adapter while creating or modifying an Ethernet adapter policy.

Guidelines and Limitations for SMB Direct with RoCE

  • SMB Direct with RoCE is supported only on Windows Server 2012 R2.
  • SMB Direct with RoCE is supported only with Cisco UCS VIC 1340 and 1380 adapters.
  • Cisco UCS Manager does not support more than 4 RoCE-enabled vNICs per adapter.
  • Cisco UCS Manager does not support RoCE with NVGRE, VXLAN, NetFlow, VMQ, or usNIC.
  • You can not use Windows Server NIC Teaming together with RMDA enabled adapters in Windows Server 2012 and Windows Server 2012 R2 or you will lose RDMA feature on these adapters.
  • Maximum number of queue pairs per adapter is 8192.
  • Maximum number of memory regions per adapter is 524288.
  • If you do not disable RoCE before downgrading Cisco UCS Manager from Release 2.2(4), downgrade will fail.

Checkout my post about Hyper-V over SMB:



Windows Server 2012 SMB Direct RDMA Copy

Hyper-V over SMB: SMB Direct (RDMA)

Another important part of SMB 3.0 and Hyper-V over SMB is the performance. In the past you could use iSCSI, Fiber Channel or FCoE (Fiber Channel over Ethernet). Now SMB 3.0 has a lot of performance improvements to make the Hyper-V over SMB scenario even work. But if you need even more performance you can use new feature which came with Windows Server 2012 and is of course also present in Windows Server 2012 R2 called SMB Direct, which supports the use of network adapters that have Remote Direct Memory Access (RDMA) capability.  Network adapters with RDMA offer some great enhancements such as very low latency, increased throughput and low CPU utilization since the functionality is offloaded to the network card.

Advantages

  • Increased throughput: Leverages the full throughput of high speed networks where the network adapters coordinate the transfer of large amounts of data at line speed.
  • Low latency: Provides extremely fast responses to network requests, and, as a result, makes remote file storage feel as if it is directly attached block storage.
  • Low CPU utilization: Uses fewer CPU cycles when transferring data over the network, which leaves more power available to server applications.

(Source TechNet)

Technology and Requirements

At the moment there are different versions of network adapters with RDMA capabilities, currently these are iWARP, InfiniBand or RoCE.

  • iWARP, is a simple solution which does not really need any more configuraiton
  • InfiniBand,
  • RoCE (RDMA over Converged Ethernet), which needs also Switches to be configured in the right way for bandwidth management (DCB/PFC)

On the software side you need Windows Server 2012 or Windows Server 2012 R2 with SMB 3.0. SMB Direct is not supported in previous versions of SMB and Windows Server.

Setup of SMB Direct

Well SMB Direct or RDMA if oyu will is enabled by default, so Windows Server will make use of it when ever possible. But there are some things you have to make sure.

  • Which type of RDMA am I using, is it iWARP, InfiniBand or RoCE. Some of them maybe require additional configuration on the network. If you are using RoCE RDMA seems to work without configuration but you can run into performance issues as my fellow Microsoft MVP Didier van Hoye descripes in his blog post.
  • Install the latest NIC drivers
  • Install the latest firmware
  • Enable SMB Multichannel if you disabled it. SMB Direct will be also disabled when you disable Multichannel.
  • In a Failover Cluster make sure that the RDMA NICs are also marked as client access adapters.
  • SMB Direct doesn’t work with NIC Teaming or Virtual Switches
  • On the file server you should also tune performance by disabling hyper threading, Disabling processor C States and setting the power profile to full power.

Verify SMB configuration

Verify if RDMA is enabled, first cmdlet checks if it’s enabled on the server it self, second one checks if it’s enabled on the network adapters and the third checks if the hardware is RDMA capable.

 
Get-NetOffloadGlobalSetting | Select NetworkDirect
Get-NetAdapterRDMA
Get-NetAdapterHardwareInfo

Verify that SMB Multichannel is enabled, which confirms the NICs are being properly recognized by SMB and that their RDMA capability is being properly identified.

On the client:

 
Get-SmbClientConfiguration | Select EnableMultichannel
Get-SmbClientNetworkInterface

On the server:

 
Get-SmbServerConfiguration | Select EnableMultichannel
Get-SmbServerNetworkInterface
netstat.exe -xan | ? {$_ -match "445"}

And as already mentioned in the SMB Mutlichannel blog post, you can verify the SMB connections:

 
Get-SmbConnection
Get-SmbMultichannelConnection
netstat.exe -xan | ? {$_ -match "445"}

And of course you have some great performance counters.

SMB Direct Performance Counters

If you run some copy jobs you can see the amazing performance (if your storage is fast enough). Here you can also see a print screen with Mellanox ConnectX-3 Ethnernet adapters which are using RoCE in Windows Server 2012. You can see that you don’t see any TCP traffic in the Task Manager on the RDMA NICs.

Windows Server 2012 SMB Direct RDMA Copy

Get more information about SMB Direct on Jose Barretos blog SMB3.info or on Microsoft TechNet.