how to configure OVS-DPDK in the host side

  • Intel 82599ES 10G NIC

DPDK support: yes
http://dpdk.org/doc/nics

for Mellanox 100G NIC, you should use ofed which requires dpdk and ovs-dpdk to be built from source


  • install dpdk and openvswitch with dpdk
apt install openvswitch-switch-dpdk

  • enable dpdk and update default openvswitch to openvswitch-dpdk
systemctl enable dpdk
service dpdk start
update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk

  • bind NICs to dpdk pmd in /etc/dpdk/interfaces
pci    0000:04:00.0    igb_uio
pci    0000:04:00.1    igb_uio

find PCI addresses of your NIC using “lspci |grep -i ethernet”


  • configure dpdk options in ovs-dpdk
ovs-vsctl set Open_vSwitch . "other_config:dpdk-init=true"
ovs-vsctl set Open_vSwitch . "other_config:pmd-cpu-mask=0xc"
ovs-vsctl set Open_vSwitch . "other_config:dpdk-socket-mem=2048,0"
ovs-vsctl set Open_vSwitch . \
   "other_config:dpdk-extra=--vhost-owner libvirt-qemu:kvm --vhost-perm 0666" ovs-vsctl get Open_vSwitch . other_config

i.e. how to remove DPDK options, for example
ovs-vsctl remove Open_vSwitch . other_config pmd-cpu-mask

in case of quad-CPU like R930

ovs-vsctl set Open_vSwitch . "other_config:dpdk-init=true"
ovs-vsctl set Open_vSwitch . "other_config:pmd-cpu-mask=0x66666666"
ovs-vsctl set Open_vSwitch . "other_config:dpdk-socket-mem=0,2048,2048,0"
ovs-vsctl set Open_vSwitch . \
   "other_config:dpdk-extra=--vhost-owner libvirt-qemu:kvm --vhost-perm 0666"
ovs-vsctl get Open_vSwitch . other_config

  • restart dpdk and openvswitch
service dpdk restart
service openvswitch-switch restart
ps ax |grep dpdk
ps ax |grep ovs

  • configure ovs-dpdk
ovs-vsctl add-br br-lan0 -- set bridge br-lan0 datapath_type=netdev
ovs-vsctl add-br br-lan1 -- set bridge br-lan1 datapath_type=netdev
ovs-vsctl add-port br-lan0 dpdk0 -- set Interface dpdk0 type=dpdk \
                                    options:dpdk-devargs=0000:04:00.0
ovs-vsctl add-port br-lan1 dpdk1 -- set Interface dpdk1 type=dpdk \
                                    options:dpdk-devargs=0000:04:00.1
ovs-vsctl add-port br-wan dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
                                    type=dpdkvhostuser
ovs-vsctl add-port br-wan dpdkvhostuser1 -- set Interface dpdkvhostuser1 \
                                    type=dpdkvhostuser

you can omit configuring vhost-user interface since openstack automatically configures vhost-user front-end when it launches an instance.
ovs-vsctl add-port br-wan vhost-user-1 — set Interface vhost-user-1 type=dpdkvhostuser


  • configure vhostuser interface in VM
virsh
list --all
edit domName
<devices>
    <interface type='vhostuser'>
      <source type='unix' path='/var/run/openvswitch/dpdk0' mode='client'/>
      <model type='virtio'/>
    </interface>
</devices>
<devices>
    <interface type='vhostuser'>
      <source type='unix' path='/var/run/openvswitch/dpdk1' mode='client'/>
      <model type='virtio'/>
    </interface>
</devices>

if you use vhostuserclient mode,

  <interface type='vhostuser'>
    <source type='unix' path='/var/run/openvswitch/dpdk0' mode='server'/>
    <model type='virtio'/>
  </interface>

https://libvirt.org/formatdomain.html


  • sample XML for ovs-dpdk

look at hugepages and numa tags

virsh # dumpxml Xenial-cloud-img
<domain type='kvm'>
  <name>Xenial-cloud-img</name>
  <uuid>4c6042ca-dcce-487a-9493-959764a8e2c9</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>2048</shares>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-zesty'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
    <numa>
      <cell id='0' cpus='0-3' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/mslee/work/CloudRouter/test/xenial-server-cloudimg-amd64-disk1-kairoson-certificate.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:6f:38:be'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8e:10'/>
      <source type='unix' path='/var/run/openvswitch/dpdk0' mode='client'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8e:11'/>
      <source type='unix' path='/var/run/openvswitch/dpdk1' mode='client'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

  • increasing the number of queues

if you want to increase the number of rx/tx queues per a DPDK physical port, set the number of rx queues for DPDK physical interface. The rx queues are assigned to pmd threads on the same NUMA node in a round-robin fashion.

ovs-vsctl set Interface dpdk0 options:n_rxq=<integer>

  • increasing the size of queue

if you want to increase the queue size, set the number of rx/tx descriptors that a physical port will be initialized with.

ovs-vsctl set Interface dpdk0 options:n_rxq_desc=<integer>
ovs-vsctl set Interface dpdk0 options:n_txq_desc=<integer>

Generally, smaller queue sizes can have a positive impact for latency at the expense of throughput. The opposite is often true for larger queue sizes.
http://docs.openvswitch.org/en/latest/intro/install/dpdk/

Note: increasing the number of rx descriptors eg. to 4096 may have a negative impact on performance due to the fact that non-vectorised DPDK rx functions may be used. This is dependent on the driver in use, but is true for the commonly used i40e and ixgbe DPDK drivers.


  • Rx Mergeable Buffers

Rx mergeable buffers is a virtio feature that allows chaining of multiple virtio descriptors to handle large packet sizes. Large packets are handled by reserving and chaining multiple free descriptors together. Mergeable buffer support is negotiated between the virtio driver and virtio device and is supported by the DPDK vhost library. This behavior is supported and enabled by default, however in the case where the user knows that rx mergeable buffers are not needed i.e. jumbo frames are not needed, it can be forced off by adding mrg_rxbuf=off to the QEMU command line options. By not reserving multiple chains of descriptors it will make more individual virtio descriptors available for rx to the guest using dpdkvhost ports and this can improve performance.

# if you don't use jumbo frame, set mrg_rxbuf=off to improve performance.
  <interface type='network'>
    <source network='default'/>
    <target dev='vnet1'/>
    <model type='virtio'/>
    <driver name='vhost' txmode='iothread' ioeventfd='on' event_idx='off' \
            queues='5' rx_queue_size='256' tx_queue_size='256'>
      <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off'
            mrg_rxbuf='off'/>
      <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/>
    </driver>
  </interface>

refer to ‘Setting NIC driver-specific options’ section for more detail.
https://libvirt.org/formatdomain.html


  • sample VM
<domain type='kvm'>
  <name>xenial-ovs-dpdk</name>
  <uuid>9c86a660-a74f-4183-be39-e0c65f964865</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>2048</shares>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-zesty'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
    <numa>
      <cell id='0' cpus='0-3' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/mslee/work/CloudRouter/test/xenial-server-cloudimg-amd64-disk1-kairoson-certificate-clone-1.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:4e:e1:08'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:00:f1:9e:06'/>
      <source type='unix' path='/var/run/openvswitch/dpdkvhostuser0' mode='client'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:00:02:f5:27'/>
      <source type='unix' path='/var/run/openvswitch/dpdkvhostuser1' mode='client'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Leave a Reply

Your email address will not be published. Required fields are marked *