10 Replies Latest reply on Dec 6, 2018 7:58 AM by jspuij

    Error in ixl VF driver during link state change.

    jspuij

      I'm running a FreeBSD (pfSense) VM using SR-IOV on a KVM linux host. The linux host has it's dual port NIC bonded (balance-xor, layer3+4) and FreeBSD VF are bonded as well (LAGG, mode balanced).

      Pulling (or reinserting) one of the plugs of the bond results in an error about 50% of the time:

       

      ixlv0: PF initiated reset!

      ixlv0: ASQ VF Error detected

      ixlv0: ASQ Critical Error detected

      ixlv0: WARNING: Resetting!

      ixlv0: Unable to send opcode DISABLE_QUEUES to PF, status I40E_ERR_QUEUE_EMPTY, aq error OK

      ixlv0: ixlv_reset: VF reset failed

      ixlv0: ixlv_reset: VF reset failed

       

      Sometimes the state change is successful however:

       

      ixlv1: PF initiated reset!

      ixlv1: link state changed to DOWN

       

      Note that this is the other VF, but this seems to be totally random. Once one VF fails, the other one stops working as well. Only restarting the virtual machine wil restore network connectivity. A linux VM does not exhibit this behavior and changes state reliably every time (with reduced throughput for the bond of course). Host bond functions OK as well.

       

      Configuration of the host bond and VF:

       

      # ip link

      ...

      2: enp26s0f0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond1 state DOWN mode DEFAULT group default qlen 1000

          link/ether ac:1f:6b:09:ef:c2 brd ff:ff:ff:ff:ff:ff

          vf 0 MAC ac:1f:6b:09:ee:10, spoof checking off, link-state auto, trust on

          vf 1 MAC ac:1f:6b:09:ee:11, spoof checking off, link-state auto, trust on

          vf 2 MAC ac:1f:6b:09:ee:12, spoof checking off, link-state auto, trust on

          vf 3 MAC ac:1f:6b:09:ee:13, spoof checking off, link-state auto, trust on

      3: enp26s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000

          link/ether ac:1f:6b:09:ef:c2 brd ff:ff:ff:ff:ff:ff

          vf 0 MAC ac:1f:6b:09:ee:10, spoof checking off, link-state auto, trust on

          vf 1 MAC ac:1f:6b:09:ee:11, spoof checking off, link-state auto, trust on

          vf 2 MAC ac:1f:6b:09:ee:12, spoof checking off, link-state auto, trust on

          vf 3 MAC ac:1f:6b:09:ee:13, spoof checking off, link-state auto, trust on

      ...

      17: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000

          link/ether ac:1f:6b:09:ef:c2 brd ff:ff:ff:ff:ff:ff

       

      Is this a bug in the FreeBSD driver or am I forgetting something?

        • 1. Re: Error in ixl VF driver during link state change.
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hello jspuij,

          Thank you for posting on the Intel ® communities.

          I would need to have more information in order to best assist you, please let me know the model of the Intel product you have. 


          Regards,
          David V
           
          Intel Customer Support Technician
          Under Contract to Intel Corporation

          • 2. Re: Error in ixl VF driver during link state change.
            Intel Corporation
            This message was posted on behalf of Intel Corporation

            Hello jspuij,

            I am following up with your case and see that we have not heard back from you.

            Please let me know the model of the Intel product you have. 

            If you need more assistance do not hesitate to reply.


            Regards,
            David V
             
            Intel Customer Support Technician
            Under Contract to Intel Corporation

            • 3. Re: Error in ixl VF driver during link state change.
              jspuij

              I'm sorry, the notification e-mail ended up in my spam folder. We have the following adapter (dual port 10Gb):

               

              1a:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

              1a:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

               

              Actually I found the solution to the issue. Disabling the built-in LLDP firmware agent fixes the issue with FreeBSD guests.

              Still I believe it's a bug in the freebsd guest driver.

               

              There are other issues with the virtual functions preventing me using them (we were running bonding on the interfaces, and multiple servers running a cluster with virtual IP's, and those virtual IP's mac addresses were just not picked up by the host mac filter), so we ended up using virtIO network devices instead.

              • 4. Re: Error in ixl VF driver during link state change.
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hello jspuij,

                Thank you for your response.

                The appropriate support team will be getting back to you as soon as possible to assist you with this. 


                Regards,
                David V
                 
                Intel Customer Support Technician
                Under Contract to Intel Corporation

                • 5. Re: Error in ixl VF driver during link state change.
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  Hello Jspuij,

                  Thank you for the reply.  Gald to see you were able to find a solution.  Please let us know if you want us to assist in further investigation, or If you prefer to use VirtIO as a work around.  If you want to continue investigating this issue please provide the following:

                  • Linux version on the host.
                  • Version of FreeBSD.
                  • Is this an integrated or PCIe controller?  Markings from the adapter product label.
                  • Ethtool -i <interface> from Linux host.
                  • Have you tried other bond types?
                  Let us know if you have any other questions.

                  Best regards,
                  Daniel D
                  Intel Customer Support
                  • 6. Re: Error in ixl VF driver during link state change.
                    jspuij

                    Hello Daniel,

                     

                    Thank you for following up on the issue. Yes we would like to use the virtual functions, as this frees the host from a lot of unnecessary interrupts and buffer copies. To answer your questions:

                     

                    • uname -a on the host: Linux 4.15.18-7-pve #1 SMP PVE 4.15.18-27 (Wed, 10 Oct 2018 10:50:11 +0200) x86_64 GNU/Linux (Proxmox VE)
                    • Freebsd VM: pfsense  2.4.4-RELEASE (amd64) built on Thu Sep 20 09:03:12 EDT 2018 FreeBSD 11.2-RELEASE-p3
                    • Network adapter: Supermicro AOC-URN2-i4GXS (Intel® X710 + i350 on 1 PCIe riser card)
                    • ethtool -i enp26s0f0:

                     

                    driver: i40e

                    version: 2.7.12

                    firmware-version: 6.01 0x800036eb 1.1861.0

                    expansion-rom-version:

                    bus-info: 0000:1a:00.0

                    supports-statistics: yes

                    supports-test: yes

                    supports-eeprom-access: yes

                    supports-register-dump: yes

                    supports-priv-flags: yes

                     

                    • We are currently running balance-xor on the host with xmit-hash-policy layer3+4. We have used LACP on the host, with the guest running balance-xor (to prevent LACP guest traffic from interfering, later we moved the VMs to their own VLANs) but it did not make any difference.

                     

                    Settings script for the bonded adapters:

                     

                    #!/bin/bash

                    echo $2 > /sys/class/net/$1/device/sriov_numvfs

                    ethtool --set-priv-flags $1 disable-fw-lldp on

                    counter=0

                    while [ $counter -lt $2 ]

                    do

                      ip link set $1 vf $counter spoofchk off

                      ip link set $1 vf $counter trust on

                      ((counter++))

                    done

                     

                    Where $2 is the number of virtual functions and $1 is the device name (enp26s0f0 and enp26s0f1). The virtual functions were bonded on the guest VM using LAGG from FreeBSD with the same bonding mode. The LAGG would fully function with it's real IP and mac address. Adding a  CARP virtual IP to the LAGG bond would make it impossible to send traffic to that IP from the host to VM or any other machine on the VLAN to the VM.

                     

                    Funny thing is: We could get the CARP virtual IP to work by adding it's MAC to the mac list on the host

                     

                    echo add 00:00:5e:00:01:02 > /sys/class/net/enp26s0f0/device/sriov/0/mac_list

                    echo add 00:00:5e:00:01:02 > /sys/class/net/enp26s0f1/device/sriov/0/mac_list

                     

                    This however defeats the purpose of the virtual IP within the cluster of VMs. It should failover automatically. I tried enabling promiscuous mode on both host adapters and VM adapters but that did not seem to make any difference.

                     

                    Thanks for looking into this.

                    • 7. Re: Error in ixl VF driver during link state change.
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Hello Jspuij,

                      Have you checked with Supermicro that the drivers and operating systems are compatible with this device?  Since this is an OEM device, we recommend checking with the OEM first for any considerations.  Proxmox and FreeBSD may also provide advice.  Please let us know if you still need assistance after contacting them.

                      Best regards,
                      Daniel D
                      Intel Customer Support

                      • 8. Re: Error in ixl VF driver during link state change.
                        jspuij

                        Supermicro only supports SLES and RHEL. I've installed SLES to see whether it has the same issue. It does. Both SLES and RHEL drivers from supermicro are redirects to the latest intel driver on the intel website. (with a wrong version number on the supermicro site). I've used both the kernel drivers on Linux and FreeBSD and the ones provided by intel and it did not matter.

                         

                        I'll keep using VirtIO for know as these servers are going into the datacenter.

                        • 9. Re: Error in ixl VF driver during link state change.
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          Hello Jspuij,

                          Thank you for the update.  We will look into the issue, and update you soon.  Will you still be able to run some tests on these servers with the IXL driver if necessary, or do you have a test server?  Let us know if you have any other questions.

                          Best regards,
                          Daniel D
                          Intel Customer Support

                          • 10. Re: Error in ixl VF driver during link state change.
                            jspuij

                            Thank you, I'll be able to do tests as we have test servers.