14 Replies Latest reply on Dec 7, 2018 7:33 AM by eraymond00

    Intel X710 with i40en-1.7.11 driver on VMware 6.7

    eraymond00

      Hi,

       

      I'm conducting network performance testing on VMware 6.7 using an Intel X710 quad port 10G NIC and seeing some unusual behavior. It started while testing our application, but also will happen when testing using a Centos 7.3 VM running an ethernet bridge. This X710 on the ESXi host has the latest i40en 1.7.11 driver and 6.01 firmware installed.

       

      We've set up a simple topology where the VM is connected to two of the X710 ports on the host through two standard vSwitches.

       

      Screen Shot 2018-11-07 at 10.31.46 AM.png

       

      When forwarding TCP traffic across the ethernet bridge, after a few hundred HTTP transactions, we start to see more and more ARP requests and replies from a server IP/MAC address to the broadcast MAC, trying to resolve a client IP/MAC address. This type of traffic will increase while the HTTP traffic starts to decrease, until its eventually entirely ARP req/replies seen on the wire. Once the HTTP traffic test is stopped, the ARP traffic will continue for another minute or two. Without touching anything after the test completes, if we restart it again, the same pattern of HTTP traffic devolving into entirely ARP req/replies happens all over again.

       

      We've tried to narrow down the specifics of this test and we've come up with the following observations:

       

      - this will only happen with the Fotrville X710 cards, if we swap it for a Niantic 82599, this problem doesn't happen

       

      - this will only happen with more than 100 or so IP addr src/dst pairs. If few than 100, the ARP req/replies are not seen.

       

      - this problem does not seem to happen if using distributed vSwitches on the ESXi host, only the standard vSwitches exhibit this behavior.

       

      - from the Force10 switch connecting the traffic generator to the ESXi host, the outgoing interface claims to have sent only a few hundred broadcast packets, where as the vmnic stats on the ESXi host claims to have received millions of broadcast packets. This makes it seem as though they are originating on the i40en driver or the standard vSwitch code.

       

      At this point, it's difficult to tell where this problem is originating, either in the i40en driver or in the VMware standard vSwitch. Has anyone else observed anything similar to this behavior or might have a suggestion as to where to look further?

       

      Thanks,

      EJ Raymond

        • 1. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hello Eraymond00,

          Thank you for posting in Intel Wired Ethernet Communities.  Please let us know if the X710 is being used alone, or an integrated adapter is being used with the X710.  Please provide the markings from the product label of one of the adapters as well.  Let us know if you have any other questions.

          Best regards,
          Daniel D
          Intel Customer Support

          • 2. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
            eraymond00

            I can reproduce this problem on a Silicom Quad 10G X710 (PE310G4I71L-SRD) and a Dell X710-DA4 Quad port 10G SFP PCI-E (DDJKY). These cards are not being used alone with in a Dell PowerEdge R710 (24x CPU Intel Xeon Gold 5118) w/ 128G RAM. Both cards are running the latest firmware from their respective suppliers (Intel = 6.01, Dell = 6.00). Attached is a picture taken of both cards.

             

            The half height adapter is the Intel X710:

            IMAG0101.jpg

             

            [root@SVVA2:~] esxcli network nic get -n vmnic12

               Advertised Auto Negotiation: true

               Advertised Link Modes: Auto, 1000BaseSR/Full, 10000BaseSR/Full

               Auto Negotiation: false

               Cable Type: FIBRE

               Current Message Level: 0

               Driver Info:

                     Bus Info: 0000:d8:00:0

                     Driver: i40en

                     Firmware Version: 6.01 0x8000351b 255.65535.255

                     Version: 1.7.11

               Link Detected: true

               Link Status: Up

               Name: vmnic12

               PHYAddress: 0

               Pause Autonegotiate: false

               Pause RX: false

               Pause TX: false

               Supported Ports: FIBRE

               Supports Auto Negotiation: true

               Supports Pause: true

               Supports Wakeon: true

               Transceiver:

               Virtual Address: 00:50:56:52:35:26

               Wakeon: MagicPacket(tm)

             

            and the full height adapter is the Dell X710:

             

            IMAG0098.jpg

             

            [root@SVVA2:~] esxcli network nic get -n vmnic4

               Advertised Auto Negotiation: true

               Advertised Link Modes: Auto, 1000BaseSR/Full, 10000BaseSR/Full

               Auto Negotiation: true

               Cable Type: FIBRE

               Current Message Level: 0

               Driver Info:

                     Bus Info: 0000:3b:00:0

                     Driver: i40en

                     Firmware Version: 6.00 0x80003752 18.5.17

                     Version: 1.7.11

               Link Detected: true

               Link Status: Up

               Name: vmnic4

               PHYAddress: 0

               Pause Autonegotiate: false

               Pause RX: false

               Pause TX: false

               Supported Ports: FIBRE

               Supports Auto Negotiation: true

               Supports Pause: true

               Supports Wakeon: true

               Transceiver:

               Virtual Address: 00:50:56:53:14:e0

               Wakeon: MagicPacket(tm)

            • 3. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
              eraymond00

              Correction, the last msg should have read "these adapters are being used standalone."

              • 4. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hello Eraymond00,

                 

                Thank you for the information.  I only see an image of the Silicom and Dell adapter in the post, is there a retail Intel adapter involved as well?  Would you be able to provide some details of the application being tested, and the software used to monitor the network so we can rule it out?  Are the ARP requests all incoming or outgoing?  Let us know if any other questions arise.

                 

                Best regards,
                Daniel D
                Intel Customer Support

                • 5. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                  eraymond00

                  - These are the only two types of X710 I've tested with. I don't have access to a retail Intel X710 at the moment. 

                   

                  - There is no application involved, I've just created an ethernet bridge between two interfaces of the X710 and I'm sending HTTP over the bridge.

                   

                  - The software used to monitor the network is the 'pktcap' utility on the ESXi host, part of the ESXi host-level utilities. I've confirmed using 'tshark' on the bridge interfaces as well.

                   

                  - The ARP requests/replies are passing in both directions, with requests originating in both directions as well.

                  • 6. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                    Intel Corporation
                    This message was posted on behalf of Intel Corporation

                    Hello Eraymond00,

                    Thank you for the reply.  Please check with Dell and Silicom if there are any known issues, and confirm the SFP modules and cables are compatible with these adapters.  Let us know if they are able to resolve the issue or if you still need assistance.

                    Best regards,
                    Daniel D
                    Intel Customer Support

                    • 7. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                      eraymond00

                      Daniel,

                       

                      We've already reached out to Dell and Silicom and have updated to their latest firmwares. The SFP and cables are compatible, since the problem seems only reproducible using the standard vswitch and not the distributed vswitch. It's also only reproducible when the Fortville is used for both interfaces. If a Fortville is used for one and a Niantic used for the other, this problem is not seen.

                       

                      As they've indicated to us, this is not a known issue for Dell or Silicom. Since we cannot identify the root cause of this problem and it may be either an issue with the VMware switching software or the ESXi i40en driver, we're in the process of trying to work with VMware as well as hoping for assistance from Intel.

                       

                      Thanks,

                      EJ Raymond

                      • 8. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                        Intel Corporation
                        This message was posted on behalf of Intel Corporation

                        Hello Eraymond00,

                        Thank you for checking with Dell and Silicom.  We will investigate this issue with the information provided, and update you as soon as possible.  Let us know if you have any other questions.

                        Best regards,
                        Daniel D
                        Intel Customer Support

                        • 9. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          Hello Eraymond00,

                          Thank you for your patience.  Do you have access to two X710 adapters of alike branding. This could be an incompatibility with the different firmware on Dell and Silicom.  Please continue to work with VMware, as the issue is not present in a non-distributed vSwitch.  Let us know if they are able to provide a solution. 

                          Best regards,
                          Daniel D
                          Intel Customer Support

                          • 10. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                            Intel Corporation
                            This message was posted on behalf of Intel Corporation

                            Hello Eraymond00,

                            Please let us know if you require any other assistance after contacting VMware.  Please also try the scenario with adapters of the same brand if possible.  Let us know if you have any other questions.

                            Best regards,
                            Daniel D
                            Intel Customer Support

                            • 11. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                              eraymond00

                              Daniel,

                               

                              In this topology, I can configure the PNICs to one of the four QUAD 10G ports of the same PNIC, which should eliminate firmware incompatibilities. With the same testing, I can recreate the ARP storm. The problem seems to be isolated to having X710 PNICs as the uplinks to the vswitches connecting the eth bridge. Any other endpoint used with these uplinks does not have an effect on the recreation.

                               

                              As an update, we've narrowed the problem down more to the specific usage of the 'netqueue loadbalancer' feature of the PNICs on the host. This determines the number of Tx/Rx rings used at the vmKernel level. This feature provides performance optimizations by using dynamic failover of one Rx queue to the next in cases of overload. To enable/disable the usage of this feature, the user can issue the following command at the ESXi 6.7 command line:

                               

                              esxcli network nic queue loadbalancer state set -e 0 -n vmnicX

                               

                              We've noticed that during the recreation of the test, the subsequent Rx queues are being utilized when the ARP storm occurs. We can also seem to make the ARP storm come and go during a single test, by enabling/disabling this feature at runtime. Does this conflict with another feature that may be enabled by default? Please let us know if there is any other driver/PNIC configurations to try.

                               

                              Thanks,

                              EJ Raymond

                              • 12. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                                Intel Corporation
                                This message was posted on behalf of Intel Corporation

                                Hello Eraymond00,

                                Thank you for the update.  We will look into this information regarding the queue loadbalancer.  We will update you soon with any recommendations.  Let us know if you have any questions.  Please continue to work with VMware as well for a possible solution.

                                Best regards,
                                Daniel D
                                Intel Customer Support

                                • 13. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Hello Eraymond00,

                                  Please provide an output with the other features supported by the NIC, so we can see what would possibly cause a conflict.  This should be esxcli network nic software list, and also esxcli network nic queue loadbalancer list.  Let us know if you have any other questions.

                                  Best regards,
                                  Daniel D
                                  Intel Customer Support

                                  • 14. Re: Intel X710 with i40en-1.7.11 driver on VMware 6.7
                                    eraymond00

                                    This is a dump of our PNIC configuration. We assume that these settings are set to their default values:

                                     

                                    [root@SVVA2:~] esxcfg-nics -l

                                    Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description

                                    vmnic0  0000:18:00.0 ntg3        Up   1000Mbps   Full   20:04:0f:e9:c8:0c 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

                                    vmnic1  0000:18:00.1 ntg3        Down 0Mbps      Half   20:04:0f:e9:c8:0d 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

                                    vmnic10 0000:86:00.0 ixgben      Up   10000Mbps  Full   00:1b:21:87:c5:68 1500   Intel(R) 82599 10 Gigabit Dual Port Network Connection

                                    vmnic11 0000:86:00.1 ixgben      Up   10000Mbps  Full   00:1b:21:87:c5:69 1500   Intel(R) 82599 10 Gigabit Dual Port Network Connection

                                    vmnic12 0000:d8:00.0 i40e        Up   10000Mbps  Full   00:e0:ed:73:6c:e4 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

                                    vmnic13 0000:d8:00.1 i40e        Up   10000Mbps  Full   00:e0:ed:73:6c:e5 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

                                    vmnic14 0000:d8:00.2 i40e        Up   10000Mbps  Full   00:e0:ed:73:6c:e6 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

                                    vmnic15 0000:d8:00.3 i40e        Up   10000Mbps  Full   00:e0:ed:73:6c:e7 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

                                    vmnic2  0000:19:00.0 ntg3        Up   1000Mbps   Full   20:04:0f:e9:c8:0e 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

                                    vmnic3  0000:19:00.1 ntg3        Up   1000Mbps   Full   20:04:0f:e9:c8:0f 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

                                    vmnic4  0000:3b:00.0 i40e        Down 0Mbps      Half   f8:f2:1e:1b:1e:c0 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

                                    vmnic5  0000:3b:00.1 i40e        Down 0Mbps      Half   f8:f2:1e:1b:1e:c2 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

                                    vmnic6  0000:3b:00.2 i40e        Up   10000Mbps  Full   f8:f2:1e:1b:1e:c4 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

                                    vmnic7  0000:3b:00.3 i40e        Up   10000Mbps  Full   f8:f2:1e:1b:1e:c6 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

                                    vmnic8  0000:af:00.0 ixgben      Up   10000Mbps  Full   00:1b:21:87:c3:ec 1500   Intel(R) 82599 10 Gigabit Dual Port Network Connection

                                    vmnic9  0000:af:00.1 ixgben      Up   10000Mbps  Full   00:1b:21:87:c3:ed 1500   Intel(R) 82599 10 Gigabit Dual Port Network Connection

                                    vusb0   Pseudo       cdce        Up   100Mbps    Full   d0:94:66:7d:59:a9 1500   DellTM iDRAC Virtual NIC USB Device

                                     

                                    [root@SVVA2:~] esxcli network nic software list

                                    NIC      IPv4 CSO  IPv4 TSO  Scatter Gather  Offset Based Offload  VXLAN Encap  Geneve Offload  IPv6 TSO  IPv6 TSO Ext  IPv6 CSO  IPv6 CSO Ext  High DMA  Scatter Gather MP  VLAN Tagging  VLAN Untagging

                                    -------  --------  --------  --------------  --------------------  -----------  --------------  --------  ------------  --------  ------------  --------  -----------------  ------------  --------------

                                    vmnic0   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic1   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic10  off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic11  off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic12  off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic13  off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic14  off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic15  off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic2   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic3   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic4   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic5   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic6   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic7   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic8   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vmnic9   off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                    vusb0    off       off       off             off                   off          off             off       off           off       off           off       off                off           off

                                     

                                    [root@SVVA2:~] esxcli network nic queue loadbalancer list

                                    NIC      RxQPair  RxQNoFeature  PreEmptibleQ  RxQLatency  RxDynamicLB  DynamicQPool  MacLearnLB  RSS  LRO  GeneveOAM

                                    -------  -------  ------------  ------------  ----------  -----------  ------------  ----------  ---  ---  ---------

                                    vmnic0   UA       ND            UA            UA          NA           UA            NA          UA   UA   UA

                                    vmnic1   UA       ND            UA            UA          NA           UA            NA          UA   UA   UA

                                    vmnic10  UD       ND            UA            UA          NA           SA            NA          UA   UA   UA

                                    vmnic11  UA       ND            UA            UA          NA           SA            NA          UA   UA   UA

                                    vmnic12  SA       ND            SA            SA          NA           SA            NA          UA   UA   UA

                                    vmnic13  SA       ND            SA            SA          NA           SA            NA          UA   UA   UA

                                    vmnic14  SA       ND            SA            SA          NA           SA            NA          UA   UA   UA

                                    vmnic15  SA       ND            SA            SA          NA           SA            NA          UA   UA   UA

                                    vmnic2   UA       ND            UA            UA          NA           UA            NA          UA   UA   UA

                                    vmnic3   UA       ND            UA            UA          NA           UA            NA          UA   UA   UA

                                    vmnic4   UA       ND            UA            UA          NA           UA            NA          UA   UA   UA

                                    vmnic5   UA       ND            UA            UA          NA           UA            NA          UA   UA   UA

                                    vmnic6   SA       ND            SA            SA          NA           SA            NA          UA   UA   UA

                                    vmnic7   SA       ND            SA            SA          NA           SA            NA          UA   UA   UA

                                    vmnic8   UA       ND            UA            UA          NA           SA            NA          UA   UA   UA

                                    vmnic9   UA       ND            UA            UA          NA           SA            NA          UA   UA   UA

                                    vusb0    UA       ND            UA            UA          NA           UA            NA          UA   UA   UA

                                     

                                    [root@SVVA2:~] esxcli network nic queue loadbalancer state list

                                    NIC      Enabled

                                    -------  -------

                                    vmnic0      true

                                    vmnic1      true

                                    vmnic10     true

                                    vmnic11     true

                                    vmnic12     true

                                    vmnic13     true

                                    vmnic14     true

                                    vmnic15     true

                                    vmnic2      true

                                    vmnic3      true

                                    vmnic4      true

                                    vmnic5      true

                                    vmnic6      true

                                    vmnic7      true

                                    vmnic8      true

                                    vmnic9      true

                                    vusb0       true