Getting Intel SR-IOV Virtual Function counters

Question

After some research on Intel SR-IOV, I've realized that there is no convenient way to get the VF counters while VF interface is being attached to a VM (for example to a QEMU/KVM VM with a Libvirt manager). Furthermore, I didn't find even an inconvenient way to do it, because as soon as VM is being created on a HOST and it is bound in configuration to some VF interface(this is the main use case of SRI-OV VFs), the interface is automatically being detached from the HOST and attached to the VM (I think that it is done on purpose in order to allow passthrough to bypass the kernel or something), and there is actually no way that I found yet to find the counters of the VF used by the attached VM on the HOST(I'm using Ubuntu 16.04 and I've tried looking for counters on VF ports, with ethtool -S on the PF interface but with no avail), those counters may be actually very helpful in various situations. It would be really helpful if someone who has some experience in dealing with SRIOV could shade some light on this issue and tell me I'm wrong and that there is a simple way.
Thanks!

Update:. Actually I found that ethtool -S might be useful, it let you see the TX/RX queue counters of the nic, and I found that VF function driver ixgbevf uses exactly one different pair of TX/RX queues for each VF. And they are being assigned in increasing order, so it might be possible to do a mapping. But unfortunately it's an ugly and driver dependent solution.

score 2 · Answer 1 · edited Dec 07 '16 at 11:23

2

Unfortunately you're not wrong, the whole idea of SR-IOV is that it allows the VM to interact directly with the NIC without the hypervisor or host in the middle. It isn't seeing the packets to generate counters,you won't get anything using ethtool or the normal ways. You maybe able to get something from the root of the NIC but it depends on the drivers your using etc.

edited Dec 07 '16 at 11:23

Chopper3

101,808

answered Dec 07 '16 at 11:21

Nath

1,362

sanjeev Tewari · Answer 2 · 2024-01-09T11:25:49.083

I have tested this solution on Ubuntu, running on a SRIOV enabled compute in openstack installation. Steps to get counters for individual VF of a PF. The physical NIC (25G) is running on INTEL i40 driver for XXV710 device.

#lspci |grep -i ethernet
 3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
.....
# ethtool -i eth2
driver: i40e
version: 2.7.29
firmware-version: 7.20 0x80009f02 1.2829.0
expansion-rom-version: 
bus-info: 0000:3b:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Step 1-

find for sriov dir in /sys/devices:

# find  /sys/devices/ -type d -name sriov
/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.1/sriov

Step 2 : Go to this dir and do ls -lrt. You will see the VF numbers. These numbers are same as seen in output of "ip link show ' command. Do the corelation.

# find  /sys/devices/ -type d -name sriov  |xargs ls -lrt
/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.1/sriov:
total 0
drwxr-xr-x 3 root root 0 Jan  9 21:40 7
drwxr-xr-x 3 root root 0 Jan  9 21:40 15
drwxr-xr-x 3 root root 0 Jan  9 21:40 5
drwxr-xr-x 3 root root 0 Jan  9 21:40 13
drwxr-xr-x 3 root root 0 Jan  9 21:40 3
drwxr-xr-x 3 root root 0 Jan  9 21:40 11
drwxr-xr-x 3 root root 0 Jan  9 21:40 1
drwxr-xr-x 3 root root 0 Jan  9 21:40 8
drwxr-xr-x 3 root root 0 Jan  9 21:40 6
drwxr-xr-x 3 root root 0 Jan  9 21:40 14
drwxr-xr-x 3 root root 0 Jan  9 21:40 4
drwxr-xr-x 3 root root 0 Jan  9 21:40 12
drwxr-xr-x 3 root root 0 Jan  9 21:40 2
drwxr-xr-x 3 root root 0 Jan  9 21:40 10
drwxr-xr-x 3 root root 0 Jan  9 21:40 0
drwxr-xr-x 3 root root 0 Jan  9 21:40 9

output of ip link show

## ip link show eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 3cfdfea85df9 state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:a8:5d:f9 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 11 MAC fa:16:3e:19:8c:05, vlan 1929, spoof checking off, link-state auto, trust on
    vf 12 MAC fa:16:3e:43:88:4f, vlan 1927, spoof checking off, link-state auto, trust on
    vf 13 MAC fa:16:3e:4f:b3:d0, vlan 1925, spoof checking off, link-state auto, trust on
    vf 14 MAC fa:16:3e:56:b6:4a, vlan 1923, spoof checking off, link-state auto, trust on
    vf 15 MAC 02:f1:f1:04:01:03, vlan 1938, spoof checking off, link-state auto, trust on

you can do 1 to 1 mapping and identify which is the vf you are interested in. Assume its vf 15.

Step 3- go to dir /.

# cd 15/stats

Step 4 - do ls and cat all files , except tx_spoofed

ls -lrt
total 0
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_errors
-r--r--r-- 1 root root 4096 Jan  9 21:40 rx_packets
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_dropped
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_spoofed
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_packets
-r--r--r-- 1 root root 4096 Jan  9 21:40 rx_bytes
-r--r--r-- 1 root root 4096 Jan  9 21:40 rx_dropped
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_bytes
tail  *
==> rx_bytes <==
108786154191
==> rx_dropped <==
0
==> rx_packets <==
975217621
==> tx_bytes <==
433665695588
==> tx_dropped <==
0
==> tx_errors <==
0
==> tx_packets <==
590997020

# tail  *
==> rx_bytes <==
108786397727
==> rx_dropped <==
0
==> rx_packets <==
975220967
==> tx_bytes <==
433665771018
==> tx_dropped <==
0
==> tx_errors <==
0
==> tx_packets <==
590998121

These are your counters.

BR// Sanjeev Tewari

score 0 · Answer 3 · answered Feb 12 '24 at 05:03

You can see VF counters from the host with: ip -s link show <pnic>

For example, I do ip -s link show ens1f0 and see this output corresponding with VF 22:

vf 22     link/ether 56:b8:7a:ed:41:e1 brd ff:ff:ff:ff:ff:ff, vlan 30, spoof checking off, link-state auto, trust on
    RX: bytes  packets  mcast   bcast   dropped 
         30540      206       4       3        0 
    TX: bytes  packets   dropped 
         29082      213        0

To get these values as JSON:

ip -j -s link | jq '.[] | select(.ifname == "ens1f0") | .vfinfo_list[] | select(.vf==22)'

Getting Intel SR-IOV Virtual Function counters

3 Answers3

tail *