DPDK系列之七：OVS-DPDK的在CentOS安装和测试

一、前言

根据文章《DPDK系列之六：qemu-kvm网络后端的加速技术》，OVS-DPDK可以给虚拟机的部署密度和虚拟机网络性能的提升带来帮助。本文将介绍如何编译和安装DPDK以及如何使用DPDK提升网络性能：

• OS：CentOS 7.4.1708
• OVS：2.8.1
• DPDK：17.08.1

转载自https://blog.csdn.net/cloudvtech

二、操作系统和DPDK的安装

2.1 系统配置

硬件

1 NUMA node with 4 CPU cores
8 GB memory
3 NICs
40 GB disk

操作系统

cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core)
[[e-mail] ~]# uname -a
Linux ovs-dpdk 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

2.2 安装基础依赖软件包

yum makecache
yum -y update
yum install -y epel-release
yum install -y net-tools tcpdump telnet wget zip unzip vim
yum install -y gcc gcc-c++  kernel-devel kernel-headers kernel.x86_64 net-tools
yum install -y numactl-devel.x86_64 numactl-libs.x86_64
yum install -y libpcap.x86_64 libpcap-devel.x86_64
yum install -y pciutils
yum install -y autoconf automake libtool

2.3 编译DPDK

wget http://fast.dpdk.org/rel/dpdk-17.08.1.tar.xz
tar -xvf dpdk-17.08.1.tar 
cd dpdk-stable-17.08.1/

patch lib/librte_eal/linuxapp/igb_uio/igb_uio.c

export RTE_SDK=/root/dpdk-stable-17.08.1
export RTE_TARGET=x86_64-native-linuxapp-gcc

make config T=x86_64-native-linuxapp-gcc
sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config
add in config/common_linuxapp : CONFIG_RTE_BUILD_COMBINE_LIBS=y

make
cd $RTE_SDK
make install T=x86_64-native-linuxapp-gcc

绑定网卡

ifconfig ens38 down
modprobe uio_pci_generic
modprobe uio
insmod /root/dpdk-stable-17.08.1/build/kmod/igb_uio.ko 
/root/dpdk-stable-17.08.1/usertools/dpdk-devbind.py --bind=igb_uio  0000:02:06.0

[[e-mail] openvswitch-2.8.1]#  /root/dpdk-stable-17.08.1/usertools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:02:06.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' drv=igb_uio unused=vfio-pci,uio_pci_generic

Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:02:05.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens37 drv=e1000 unused=igb_uio,vfio-pci,uio_pci_generic *Active*

转载自https://blog.csdn.net/cloudvtech

三、编译和运行OVS

3.1 编译OVS-DPDK

wget http://openvswitch.org/releases/openvswitch-2.8.1.tar.gz
tar -zxvf openvswitch-2.8.1.tar.gz 
cd openvswitch-2.8.1/
./boot.sh 
./configure --with-dpdk=/root/dpdk-stable-17.08.1/x86_64-native-linuxapp-gcc/ CFLAGS="-g -Ofast”
make 'CFLAGS=-g -Ofast -march=native' -j10
make install

3.2 配置OVS-DPDK

ovsdb-tool create /usr/local/etc/openvswitch/conf.db /root/openvswitch-2.8.1/vswitchd/vswitch.ovsschema
ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x03
export DB_SOCK=/usr/local/var/run/openvswitch/db.sock

3.3 运行OVS-DPDK

[[e-mail] openvswitch-2.8.1]# /root/openvswitch-2.8.1/utilities/ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start
Starting ovs-vswitchd Zone 0: name:<rte_eth_dev_data>, phys:0xa1fb6cc0, len:0x34000, virt:0x7f2b74bb6cc0, socket_id:0, flags:0 
                                                                                                     [ OK ]
Enabling remote OVSDB managers                                                                       [ OK ]

转载自https://blog.csdn.net/cloudvtech

四、创建基于DPDK的OVS端口

4.1 创建一个基于宿主机网卡的DPDK端口

ovs-vsctl add-port br0 myportnameone -- set Interface myportnameone     type=dpdk options:dpdk-devargs=0000:02:06.0

4.2 添加另外一个基于宿主机网卡的端口

ifconfig ens37 down
/root/dpdk-stable-17.08.1/usertools/dpdk-devbind.py --bind=igb_uio  0000:02:05.0
/root/dpdk-stable-17.08.1/usertools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:02:05.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' drv=igb_uio unused=vfio-pci,uio_pci_generic
0000:02:06.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' drv=igb_uio unused=vfio-pci,uio_pci_generic

Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio,vfio-pci,uio_pci_generic *Active*

ovs-vsctl add-port br0 myportnametwo -- set Interface myportnametwo type=dpdk options:dpdk-devargs=0000:02:05.0

对应的日志信息：

ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
2018-01-01T16:04:45.327Z|00044|memory|INFO|46340 kB peak resident set size after 115.7 seconds
2018-01-01T16:04:45.334Z|00045|ofproto_dpif|INFO|[e-mail]: Datapath supports recirculation
2018-01-01T16:04:45.334Z|00046|ofproto_dpif|INFO|[e-mail]: VLAN header stack length probed as 1
2018-01-01T16:04:45.334Z|00047|ofproto_dpif|INFO|[e-mail]: MPLS label stack length probed as 3
2018-01-01T16:04:45.334Z|00048|ofproto_dpif|INFO|[e-mail]: Datapath supports truncate action
2018-01-01T16:04:45.334Z|00049|ofproto_dpif|INFO|[e-mail]: Datapath supports unique flow ids
2018-01-01T16:04:45.334Z|00050|ofproto_dpif|INFO|[e-mail]: Datapath supports clone action
2018-01-01T16:04:45.334Z|00051|ofproto_dpif|INFO|[e-mail]: Max sample nesting level probed as 10
2018-01-01T16:04:45.334Z|00052|ofproto_dpif|INFO|[e-mail]: Datapath supports eventmask in conntrack action
2018-01-01T16:04:45.334Z|00053|ofproto_dpif|INFO|[e-mail]: Datapath supports ct_state
2018-01-01T16:04:45.334Z|00054|ofproto_dpif|INFO|[e-mail]: Datapath supports ct_zone
2018-01-01T16:04:45.334Z|00055|ofproto_dpif|INFO|[e-mail]: Datapath supports ct_mark
2018-01-01T16:04:45.334Z|00056|ofproto_dpif|INFO|[e-mail]: Datapath supports ct_label
2018-01-01T16:04:45.334Z|00057|ofproto_dpif|INFO|[e-mail]: Datapath supports ct_state_nat
2018-01-01T16:04:45.334Z|00058|ofproto_dpif|INFO|[e-mail]: Datapath supports ct_orig_tuple
2018-01-01T16:04:45.334Z|00059|ofproto_dpif|INFO|[e-mail]: Datapath supports ct_orig_tuple6
2018-01-01T16:04:45.344Z|00060|bridge|INFO|bridge br0: added interface br0 on port 65534
2018-01-01T16:04:45.345Z|00061|bridge|INFO|bridge br0: using datapath ID 00007e7fb388574d
2018-01-01T16:04:45.345Z|00062|connmgr|INFO|br0: added service controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
2018-01-01T16:04:55.351Z|00063|memory|INFO|peak resident set size grew 100% in last 10.0 seconds, from 46340 kB to 92756 kB
2018-01-01T16:04:55.351Z|00064|memory|INFO|handlers:2 ports:1 revalidators:2 rules:5

ovs-vsctl add-port br0 myportnametwo -- set Interface myportnametwo type=dpdk options:dpdk-devargs=0000:02:05.0
2018-01-01T16:05:44.914Z|00065|dpif_netdev|INFO|PMD thread on numa_id: 0, core id:  0 created.
2018-01-01T16:05:44.914Z|00066|dpif_netdev|INFO|There are 1 pmd threads on numa node 0
2018-01-01T16:05:45.052Z|00067|netdev_dpdk|INFO|Port 0: 00:0c:29:b9:82:0f
2018-01-01T16:05:45.053Z|00068|bridge|INFO|bridge br0: added interface myportnametwo on port 1
2018-01-01T16:05:45.055Z|00069|bridge|INFO|bridge br0: using datapath ID 0000000c29b9820f
2018-01-01T16:05:45.061Z|00070|netdev_dpdk|WARN|Failed to enable flow control on device 0

ovs-vsctl add-port br0 myportnameone -- set Interface myportnameone type=dpdk options:dpdk-devargs=0000:02:06.0
2018-01-01T16:07:08.295Z|00071|netdev_dpdk|INFO|Port 1: 00:0c:29:b9:82:19
2018-01-01T16:07:08.297Z|00072|bridge|INFO|bridge br0: added interface myportnameone on port 2

4.3 查看OVS信息

内存页信息

转载自https://blog.csdn.net/cloudvtech