Low Bandwidth throughput on vNIC on an HPE 631SFP28 or 631FLR adapters

HPE Network Adapters – Memory Leak Issue Can Occur On An HPE Ethernet 10/25Gb 2-port 631SFP28 Adapter Or HPE Ethernet 10/25Gb 2-port 631FLR-SFP28 Adapter Configured On ProLiant Servers Running VMware ESXi 6.5 / 6.7

On any HPE ProLiant server running VMware ESXi 6.5 or VMware ESXi 6.7, and configured with an HPE Ethernet 10/25Gb 2-port 631SFP28 Adapter or HPE Ethernet 10/25Gb 2-port 631FLR-SFP28 Adapter with NVM Image version 214.0.203000 (or earlier), a memory leak issue can occur in the network adapter firmware.

This occurs when the vmkernel dynamically allocates and frees RX queues. This occurs due to an anomaly in the network adapter firmware where memory is not being freed appropriately, leading to memory exhaustion that causes the network adapter to stop functioning. When this occurs, the link is up but there is no traffic passing through the network adapter.

Some of the symptoms are very low bandwidth on vSAN performance tests and failures, vSAN network failures, stability problems etc. It gets worse when more servers are added to cluster which increases the load on the NIC adapter.

SCOPE

Any HPE ProLiant server running VMware ESXi 6.5 or VMware ESXi 6.7, and configured with either of the following network adapters with NVM Image version 214.0.203000 (or earlier):

  • HPE Ethernet 10/25Gb 2-port 631SFP28 Adapter
  • HPE Ethernet 10/25Gb 2-port 631FLR-SFP28 Adapter 

RESOLUTION

To fix this issue, download and install the NVM Image version 214.0.224000 (later), available at the following URL:

HPE Broadcom NetXtreme-E Online Firmware Upgrade Utility for VMware Version 5.8.4

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_15ea6b5229744421991d1aeb59#tab3

Reference:

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=emr_na-a00090850en_us