0

Recently, I was developing vDPA drivers for our NIC. When testing virtio-net vDPA with multi VFs, I found that kernel vDPA framework allocated same DMA addresses for multi VFs, then different VF operate the same DMA address, such as updating the used index, cause kernel virtqueues works abnormally.

The steps are as follows:

  1. enable the NIC sriov, create 4 VFs, we can see that 4 vdpa management devices are create successful.
*[root@localhost ~]# echo 4 > /sys/class/net/enp1s0np0/device/sriov_numvfs
[root@localhost ~]# vdpa mgmtdev show
pci/0000:01:08.0:
supported_classes net
pci/0000:01:08.1:
supported_classes net
pci/0000:01:08.2:
supported_classes net
pci/0000:01:08.3:
supported_classes net*
  1. add vdpa device and enable virtio vdpa module. create 4 vdpa device and the driver are binded successful.
*[root@localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.0 name vdpa0
[root@localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.1 name vdpa1
[root@localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.2 name vdpa2
[root@localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.3 name vdpa3
[root@localhost ~]# modprobe virtio_vdpa
[root@localhost ~]# ls -l /sys/bus/vdpa/drivers/virtio_vdpa/
total 0
--w-------. 1 root root 4096 Nov 21 16:55 bind
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 module -> ../../../../module/virtio_vdpa
--w-------. 1 root root 4096 Nov 21 16:55 uevent
--w-------. 1 root root 4096 Nov 21 16:55 unbind
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 vdpa0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.0/vdpa0
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 vdpa1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.1/vdpa1
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 vdpa2 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.2/vdpa2
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 vdpa3 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.3/vdpa3*
  1. check the kernel messages output, we can see the virtio-net device and dma addresses.
*[Mon Nov 21 16:55:42 2022] virtio_net virtio0: devname virtio0 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio0: devname virtio0 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio1: devname virtio1 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio1: devname virtio1 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio2: devname virtio2 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio2: devname virtio2 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio0 enp1s0v0: renamed from eth5
[Mon Nov 21 16:55:42 2022] virtio_net virtio3: devname virtio3 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio3: devname virtio3 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] virtio_net virtio1 enp1s0v1: renamed from eth6
[Mon Nov 21 16:55:42 2022] virtio_net virtio2 enp1s0v2: renamed from eth7
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio3 enp1s0v3: renamed from eth5*

It seems that kernel vDPA framework assigns the same virtqueue dma address to four different vf. This application scenario refers to the vDPA description of Red Hat: https://www.redhat.com/en/blog/vdpa-kernel-framework-part-3-usage-vms-and-containers

My kernel version is 5.15.15, kernel vDPA options are all enabled when compile the kernel.

[root@localhost linux-5.15.15]# cat .config | grep VDPA
`CONFIG_VIRTIO_VDPA=m
CONFIG_VDPA=y
CONFIG_VDPA_SIM=m
CONFIG_VDPA_SIM_NET=m
CONFIG_VDPA_SIM_BLOCK=m
#CONFIG_VDPA_USER is not set
CONFIG_MLX5_VDPA=y
CONFIG_MLX5_VDPA_NET=m
CONFIG_VP_VDPA=m
CONFIG_VHOST_VDPA=m`
[root@localhost linux-5.15.15]#

So, is there anything i missed or misunderstood about the kernel vDPA framework, if someone can give some advice, that will be great help. thanks a lot.

I've tested with vhost-vdpa, start VMs with QEMU, and multi VFs works well.

kyle
  • 1

1 Answers1

0

I’ve investigated the virtio specification, and found that the virtio 1.1 specification has updated and add a new feature bit VIRTIO_F_SR_IOV. I guess that there will be some adaptation in the kernel vDPA framework to support multi vf. But currently I haven't seen the implementation in the 5.19 kernel yet. However, this confirms that the kernel framework is not yet supported.

The requirement for VIRTIO_F_SR_IOV as follows( see virtio 1.1 specification chapter 6.1/6.2):

If VIRTIO_F_SR_IOV has been negotiated, a driver MAY enable virtual functions through the device's PCI SR-IOV capability structure. A driver MUST NOT negotiate VIRTIO_F_SR_IOV if the device does not have a PCI SR-IOV capability structure or is not a PCI device. A driver MUST negotiate VIRTIO_F_SR_IOV and complete the feature negotiation (including checking the FEATURES_OK device status bit) before enabling virtual functions through the device's PCI SR-IOV capability structure. After once successfully negotiating VIRTIO_F_SR_IOV, the driver MAY enable virtual functions through the device's PCI SR-IOV capability structure even if the device or the system has been fully or partially reset, and even without re-negotiating VIRTIO_F_SR_IOV after the reset.

A device SHOULD offer VIRTIO_F_SR_IOV if it is a PCI device and presents a PCI SR-IOV capability structure, otherwise it MUST NOT offer VIRTIO_F_SR_IOV

see changes:

https://www.oasis-open.org/committees/ballot.php?id=3218

https://github.com/oasis-tcs/virtio-spec/issues/11

we are looking forward the future kernel version will support this feature.

kyle
  • 1