Tag: RDMA

Cisco UCS Hardware

Cisco UCS supports RoCE for Microsoft SMB Direct

As you may know we use SMB as the storage protocol for several Hyper-V deployments using Scale-Out File Server and Storage Spaces which adds a lot value to your Hyper-V deployments. To boost performance Microsoft is using RDMA or SMB Direct to accelerate Storage network performance.

RDMA over Converged Ethernet (RoCE) allows direct memory access over an Ethernet network. RoCE is a link layer protocol, and hence, it allows communication between any two hosts in the same Ethernet broadcast domain. RoCE delivers superior performance compared to traditional network socket implementations because of lower latency, lower CPU utilization and higher utilization of network bandwidth. Windows Server 2012 and later versions use RDMA for accelerating and improving the performance of SMB file sharing traffic and Live Migration. If you need to know more about RDMA or SMB Direct checkout my blog post: Hyper-V over SMB: SMB Direct

With Cisco UCS Manager Release 2.2(4), Cisco finally supports RoCE for SMB Direct. It sends additional configuration information to the adapter while creating or modifying an Ethernet adapter policy.

Guidelines and Limitations for SMB Direct with RoCE

  • SMB Direct with RoCE is supported only on Windows Server 2012 R2.
  • SMB Direct with RoCE is supported only with Cisco UCS VIC 1340 and 1380 adapters.
  • Cisco UCS Manager does not support more than 4 RoCE-enabled vNICs per adapter.
  • Cisco UCS Manager does not support RoCE with NVGRE, VXLAN, NetFlow, VMQ, or usNIC.
  • You can not use Windows Server NIC Teaming together with RMDA enabled adapters in Windows Server 2012 and Windows Server 2012 R2 or you will lose RDMA feature on these adapters.
  • Maximum number of queue pairs per adapter is 8192.
  • Maximum number of memory regions per adapter is 524288.
  • If you do not disable RoCE before downgrading Cisco UCS Manager from Release 2.2(4), downgrade will fail.

Checkout my post about Hyper-V over SMB:



Windows Server 2012 R2 Private CLoud Storage and Virtualization

Windows Server 2012 R2 Private Cloud Virtualization and Storage Poster and Mini-Posters

Yesterday Microsoft released the Windows Server 2012 R2 Private Cloud Virtualization and Storage Poster and Mini-Posters. This includes overviews over Hyper-V, Failover Clustering, Scale-Out File Server, Storage Spaces and much more. These posters provide a visual reference for understanding key private cloud storage and virtualization technologies in Windows Server 2012 R2. They focus on understanding storage architecture, virtual hard disks, cluster shared volumes, scale-out file servers, storage spaces, data deduplication, Hyper-V, Failover Clustering, and virtual hard disk sharing.

Bedsides the overview poster, Microsoft Includes the following Mini-Posters:

  • Virtual Hard Disk and Cluster Shared Volumes Mini Poster
  • Virtual Hard Disk Sharing Mini Poster
  • Understanding Storage Architecture Mini Poster
  • Storage Spaces and Deduplication Mini Poster
  • Scale-Out and SMB Mini Poster
  • Hyper-V and Failover Clustering Mini Poster

You can get the posters from the Microsoft download page.



ConnectX-3 Pro NVGRE Offloading RDMA

Hyper-V Network Virtualization: NVGRE Offloading

At the moment I spend a lot of time working with Hyper-V Network Virtualization in Hyper-V, System Center Virtual Machine Manager and with the new Network Virtualization Gateway. I am also creating some architecture design references for hosting providers which are going to use Hyper-V Network Virtualization and SMB as storage. If you are going for any kind of Network Virtualization (Hyper-V Network Virtualization or VXLAN) you want to make sure you can offload NVGRE traffic to the network adapter.

Well the great news here is that the Mellanox ConnectX-3 Pro not only offers RDMA (RoCE), which is used for SMB Direct, the adapter also offers hardware offloads for NVGRE and VXLAN encapsulated traffic. This is great and should improve the performance of Network Virtualization dramatically.

NVGRE Offloading

More information on the Mellanox ConnectX-3 Pro:

ConnectX-3 Pro 10/40/56GbE adapter cards with hardware offload engines to Overlay Networks (“Tunneling”), provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in public and private clouds, enterprise data centers, and high performance computing.

Public and private cloud clustered databases, parallel processing, transactional services, and high-performance embedded I/O applications will achieve significant performance improvements resulting in reduced completion time and lower cost per operation.

Mellanox ConnectX-3 Pro

Benefits:

  • 10/40/56Gb/s connectivity for servers and storage
  • World-class cluster, network, and storage performance
  • Cutting edge performance in virtualized overlay networks (VXLAN and NVGRE)
  • Guaranteed bandwidth and low-latency services
  • I/O consolidation
  • Virtualization acceleration
  • Power efficient
  • Scales to tens-of-thousands of nodes

Key features:

  • 1us MPI ping latency
  • Up to 40/56GbE per port
  • Single- and Dual-Port options available
  • PCI Express 3.0 (up to 8GT/s)
  • CPU offload of transport operations
  • Application offload
  • Precision Clock Synchronization
  • HW Offloads for NVGRE and VXLAN encapsulated traffic
  • End-to-end QoS and congestion control
  • Hardware-based I/O virtualization
  • RoHS-R6

Virtualized Overlay Networks — Infrastructure as a Service (IaaS) cloud demands that data centers host and serve multiple tenants, each with their own isolated network domain over a shared network infrastructure. To achieve maximum efficiency, data center operators are creating overlay networks that carry traffic from individual Virtual Machines (VMs) in encapsulated formats such as NVGRE and VXLAN over a logical “tunnel,” thereby decoupling the workload’s location from its network address. Overlay Network architecture introduces an additional layer of packet processing at the hypervisor level, adding and removing protocol headers for the encapsulated traffic. The new encapsulation prevents many of the traditional “offloading” capabilities (e.g. checksum, TSO) from being performed at the NIC. ConnectX-3 Pro effectively addresses the increasing demand for an overlay network, enabling superior performance by introducing advanced NVGRE and VXLAN hardware offload engines that enable the traditional offloads to be performed on the encapsulated traffic. With ConnectX-3 Pro, data center operators can decouple the overlay network layer from the physical NIC performance, thus achieving native performance in the new network architecture.

I/O Virtualization — ConnectX-3 Pro SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VMs) within the server. I/O virtualization with ConnectX-3 Pro gives data center managers better server utilization while reducing cost, power, and cable complexity.

RDMA over Converged Ethernet — ConnectX-3 Pro utilizing IBTA RoCE technology delivers similar low-latency and high- performance over Ethernet networks. Leveraging Data Center Bridging capabilities, RoCE provides efficient low latency RDMA services over Layer 2 Ethernet. With link-level interoperability in existing Ethernet infrastructure, Network Administrators can leverage existing data center fabric management solutions.

Sockets Acceleration — Applications utilizing TCP/UDP/IP transport can achieve industry- leading throughput over 10/40/56GbE. The hardware-based stateless offload engines in ConnectX-3 Pro reduce the CPU overhead of IP packet transport. Socket acceleration software further increases performance for latency sensitive applications.

Storage Acceleration — A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can leverage Ethernet or RDMA for high-performance storage access.

Software Support – All Mellanox adapter cards are supported by Windows, Linux distributions, VMware, FreeBSD, Ubuntu, and Citrix XenServer. ConnectX-3 Pro adapters support OpenFabrics-based RDMA protocols and software and are compatible with configuration and management tools from OEMs and operating system vendors.

 



Windows Server 2012 SMB Direct RDMA Copy

Hyper-V over SMB: SMB Direct (RDMA)

Another important part of SMB 3.0 and Hyper-V over SMB is the performance. In the past you could use iSCSI, Fiber Channel or FCoE (Fiber Channel over Ethernet). Now SMB 3.0 has a lot of performance improvements to make the Hyper-V over SMB scenario even work. But if you need even more performance you can use new feature which came with Windows Server 2012 and is of course also present in Windows Server 2012 R2 called SMB Direct, which supports the use of network adapters that have Remote Direct Memory Access (RDMA) capability.  Network adapters with RDMA offer some great enhancements such as very low latency, increased throughput and low CPU utilization since the functionality is offloaded to the network card.

Advantages

  • Increased throughput: Leverages the full throughput of high speed networks where the network adapters coordinate the transfer of large amounts of data at line speed.
  • Low latency: Provides extremely fast responses to network requests, and, as a result, makes remote file storage feel as if it is directly attached block storage.
  • Low CPU utilization: Uses fewer CPU cycles when transferring data over the network, which leaves more power available to server applications.

(Source TechNet)

Technology and Requirements

At the moment there are different versions of network adapters with RDMA capabilities, currently these are iWARP, InfiniBand or RoCE.

  • iWARP, is a simple solution which does not really need any more configuraiton
  • InfiniBand,
  • RoCE (RDMA over Converged Ethernet), which needs also Switches to be configured in the right way for bandwidth management (DCB/PFC)

On the software side you need Windows Server 2012 or Windows Server 2012 R2 with SMB 3.0. SMB Direct is not supported in previous versions of SMB and Windows Server.

Setup of SMB Direct

Well SMB Direct or RDMA if oyu will is enabled by default, so Windows Server will make use of it when ever possible. But there are some things you have to make sure.

  • Which type of RDMA am I using, is it iWARP, InfiniBand or RoCE. Some of them maybe require additional configuration on the network. If you are using RoCE RDMA seems to work without configuration but you can run into performance issues as my fellow Microsoft MVP Didier van Hoye descripes in his blog post.
  • Install the latest NIC drivers
  • Install the latest firmware
  • Enable SMB Multichannel if you disabled it. SMB Direct will be also disabled when you disable Multichannel.
  • In a Failover Cluster make sure that the RDMA NICs are also marked as client access adapters.
  • SMB Direct doesn’t work with NIC Teaming or Virtual Switches
  • On the file server you should also tune performance by disabling hyper threading, Disabling processor C States and setting the power profile to full power.

Verify SMB configuration

Verify if RDMA is enabled, first cmdlet checks if it’s enabled on the server it self, second one checks if it’s enabled on the network adapters and the third checks if the hardware is RDMA capable.

 
Get-NetOffloadGlobalSetting | Select NetworkDirect
Get-NetAdapterRDMA
Get-NetAdapterHardwareInfo

Verify that SMB Multichannel is enabled, which confirms the NICs are being properly recognized by SMB and that their RDMA capability is being properly identified.

On the client:

 
Get-SmbClientConfiguration | Select EnableMultichannel
Get-SmbClientNetworkInterface

On the server:

 
Get-SmbServerConfiguration | Select EnableMultichannel
Get-SmbServerNetworkInterface
netstat.exe -xan | ? {$_ -match "445"}

And as already mentioned in the SMB Mutlichannel blog post, you can verify the SMB connections:

 
Get-SmbConnection
Get-SmbMultichannelConnection
netstat.exe -xan | ? {$_ -match "445"}

And of course you have some great performance counters.

SMB Direct Performance Counters

If you run some copy jobs you can see the amazing performance (if your storage is fast enough). Here you can also see a print screen with Mellanox ConnectX-3 Ethnernet adapters which are using RoCE in Windows Server 2012. You can see that you don’t see any TCP traffic in the Task Manager on the RDMA NICs.

Windows Server 2012 SMB Direct RDMA Copy

Get more information about SMB Direct on Jose Barretos blog SMB3.info or on Microsoft TechNet.



Hyper-V over SMB Multichannel

Hyper-V over SMB: SMB Multichannel

As already mentioned in my first post, SMB 3.0 comes with a lot of different supporting features which are increasing the functionality in terms of performance, security, availability and backup. Here are some quick notes about some of the features which make the whole Hyper-V over SMB scenario work, this time SMB Multichannel.

SMB Multichannel was designed to solve two problems, first make the path from the Hyper-V host (SMB Client) to the File Server or Scale-Out File Server (SMB Server) redundant and get more performance by using multiple network paths. If you are using iSCSI or Fiber Channel you use MPIO (Multipath-IO) to use multiple available paths to the storage. For normal SMB traffic you may be used NIC Teaming to achieve that. SMB Multichannel is a much easier solution which offers great performance. SMB Multichannel will automatically make use of different network adapters which are configured with different IP subnets.

In my tests and the environment I have build for customers, I have seen great performance with SMB Mutlichannel. It works “better” as NIC Teaming because most of the time you just get one active network interface except you use LACP and stuff like which requires the configuration of network switches with cheap switches you may lose redundancy. The same with MPIO, MPIO most of the time works great but you can sometimes not get the performance you should get in an active/active configuration. With SMB Multichannel I can simply bundle two or even more NICs together and Multichannel will make use of all of them.

Hyper-V over SMB Multichannel

Btw. SMB Mutlichannel is also a must have if you are using RDMA NICs, because of the redundancy you only get with SMB Multichannel.

SMB Mutlichannel and NIC Teaming

SMB Mutlichannel does also work in combination with Windows Server NIC Teaming. But the feature SMB Direct (RDMA) does not allow you to use NIC Teaming or the Hyper-V Virtual Switch, because you would loose the RDMA functionality.

How do you setup SMB Multichannel?

Well the setup of SMB Multichannel is quite easy because it’s enabled by default, but there are some things you should now about SMB Multichannel for designing your environment or just to troubleshoot an installation.

Verify SMB Multichannel

Verify that SMB Multichannel is active on the client

Get-SmbClientConfiguration | Select EnableMultichannel

Verify that SMB Multichannel is active on the server

Get-SmbServerConfiguration | Select EnableMultichannel

You can also disable or enable Mutlichannel if you need to.

Set-SmbServerConfiguration -EnableMultiChannel $false
Set-SmbClientConfiguration -EnableMultiChannel $false
 
Set-SmbServerConfiguration -EnableMultiChannel $true
Set-SmbClientConfiguration -EnableMultiChannel $true

To see if all the right connections are used you can use the following commands on the client to see all the SMB connections and verifiy that all SMB Mutlichannel connections are open and using the write protocol version.

Get-SmbConnection
Get-SmbMultichannelConnection

SMB Mutlichannel Constraint

Another thing you have to know about SMB Mutlichannel is how you limit the SMB connection to specific network interfaces. For example, you have a Hyper-V hosts which has an 1 gigabit network adapter for management and stuff and you have two 10Gbit (or greater) RDMA interfaces which should be used for the connection to the storage, you want to make sure that the Hyper-V hosts only uses the RDMA network interface to connected to the storage. With a simple PowerShell cmdlet you can limit the Hyper-V host to only access a file shares with the RDMA interfaces.

  • Fileshare where the Virtual Machines are stored: \\SMB01\VMs01
  • Name of the 1 GBit network interface for Management and stuff: Management
  • Name of the RDMA interfaces: RDMA01 and RDMA02
 
New-SmbMultichannelConstraint -InterfaceAlias RDMA01, RDMA02 -ServerName SMB01

Make sure you run this on every Hyper-V host, not on the SMB file server.

Well there is a little bit more behind SMB Multichannel but  this should give you a great jumpstart into this feature. If you want to know more about SMB Multichannel checkout Jose Barretos (Microsoft Corp.) blog post on The basics of SMB Multichannel.