vdev_probe() fails when starting `pktgen` app in a docker container - docker

Question

I am trying to start a Pktgen app in docker container using openvswitch virtual ports. The structure of my ideal system is shown as follows:
(1) Build dpdk-stable-17.11.4 and ovs-2.9.0, and create 4 virtual ports as follows:
Bridge "ovs-br0"
Port "vhost-user2"
Interface "vhost-user2"
type: dpdkvhostuser
Port "ovs-br0"
Interface "ovs-br0"
type: internal
Port "vhost-user3"
Interface "vhost-user3"
type: dpdkvhostuser
Port "vhost-user0"
Interface "vhost-user0"
type: dpdkvhostuser
Port "vhost-user1"
Interface "vhost-user1"
type: dpdkvhostuser
When I check the log of creating virtual ports, I find an error:
2019-01-27T15:59:23.346Z|00041|bridge|INFO|bridge ovs-br0: added interface ovs-br0 on port 65534
2019-01-27T15:59:23.375Z|00042|bridge|INFO|bridge ovs-br0: using datapath ID 000066b38a29f447
2019-01-27T15:59:23.375Z|00043|connmgr|INFO|ovs-br0: added service controller "punix:/usr/local/var/run/openvswitch/ovs-br0.mgmt"
2019-01-27T15:59:23.411Z|00044|dpdk|INFO|VHOST_CONFIG: vhost-user server: socket created, fd: 44
2019-01-27T15:59:23.411Z|00045|netdev_dpdk|INFO|Socket /usr/local/var/run/openvswitch/vhost-user0 created for vhost-user port vhost-user0
2019-01-27T15:59:23.411Z|00046|dpdk|INFO|VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/vhost-user0
2019-01-27T15:59:23.411Z|00047|netdev_dpdk|WARN|dpdkvhostuser ports are considered deprecated; please migrate to dpdkvhostuserclient ports.
2019-01-27T15:59:23.411Z|00048|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 2 created.
2019-01-27T15:59:23.411Z|00049|dpif_netdev|INFO|There are 1 pmd threads on numa node 0
2019-01-27T15:59:23.412Z|00050|dpdk|ERR|RING: Cannot reserve memory for tailq
2019-01-27T15:59:23.424Z|00051|dpdk|ERR|RING: Cannot reserve memory for tailq
2019-01-27T15:59:23.434Z|00052|dpdk|ERR|RING: Cannot reserve memory for tailq
2019-01-27T15:59:23.443Z|00053|dpdk|ERR|RING: Cannot reserve memory for tailq
2019-01-27T15:59:23.458Z|00054|dpif_netdev|INFO|Core 2 on numa node 0 assigned port 'vhost-user0' rx queue 0 (measured processing cycles 0).
2019-01-27T15:59:23.458Z|00055|bridge|INFO|bridge ovs-br0: added interface vhost-user0 on port 1
2019-01-27T15:59:23.478Z|00056|dpdk|INFO|VHOST_CONFIG: vhost-user server: socket created, fd: 49
2019-01-27T15:59:23.478Z|00057|netdev_dpdk|INFO|Socket /usr/local/var/run/openvswitch/vhost-user1 created for vhost-user port vhost-user1
2019-01-27T15:59:23.478Z|00058|dpdk|INFO|VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/vhost-user1
2019-01-27T15:59:23.478Z|00059|dpif_netdev|INFO|Core 2 on numa node 0 assigned port 'vhost-user0' rx queue 0 (measured processing cycles 0).
2019-01-27T15:59:23.479Z|00060|dpif_netdev|INFO|Core 2 on numa node 0 assigned port 'vhost-user1' rx queue 0 (measured processing cycles 0).
It says that RING: Cannot reserve memory for tailq. I have no idea about this error. I have already allocated some hugepages in DPDK.
(2) Build pktgen-3.5.0
(3) Write Dockerfile and start docker. The Dockerfile:
FROM ubuntu:latest
RUN apt update -y
RUN apt-get install -y numactl libpcap-dev
WORKDIR /root/dpdk
COPY dpdk-stable-17.11.4 /root/dpdk/.
COPY pktgen-3.5.0 /root/pktgen/.
RUN ln -s /root/pktgen/app/x86_64-native-linuxapp-gcc/pktgen /usr/bin/pktgen
ENV PATH "$PATH:/root/dpdk/x86_64-native-linuxapp-gcc/app/"
build the images:
sudo docker build -t pktgen-docker .
start the container:
sudo docker run -ti --privileged --name=pktgen-docker \
-v /mnt/huge:/mnt/huge -v /usr/local/var/run/openvswitch:/var/run/openswitch \
pktgen-docker:latest
(4) start pktgen app.
app/x86_64-native-linuxapp-gcc/pktgen -l 0-1 -n 3 --file-prefix=pktgen-docker --no-pci --log-level=8\
--vdev 'net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0' \
--vdev 'net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1' \
-- -T -P -m '1.[0-1]'
I simply use 2 lcores, lcore-1 is for rx/tx of 2 virtual port. I set up 2 --vdev using the ovs-dpdk ports, i.e., vhost_user0, vhost_user1.
However, the error come:
Copyright (c) <2010-2017>, Intel Corporation. All rights reserved. Powered by DPDK
sh: 1: lspci: not found
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: Setting up physically contiguous memory...
EAL: Trying to obtain current memory policy.
EAL: Hugepage /mnt/huge/pktgen-dockermap_1 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_2 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_3 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_4 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_5 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_6 is on socket 0
...
EAL: Hugepage /mnt/huge/pktgen-dockermap_167 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_0 is on socket 0
EAL: Ask a virtual area of 0x15000000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x15000000)
EAL: Requesting 168 pages of size 2MB from socket 0
EAL: TSC frequency is ~2594018 KHz
EAL: Master lcore 0 is ready (tid=8f222900;cpuset=[0])
EAL: lcore 1 is ready (tid=8d5fd700;cpuset=[1])
vdev_probe(): failed to initialize net_virtio_user0 device
vdev_probe(): failed to initialize net_virtio_user1 device
EAL: Bus (vdev) probe failed.
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio
Copyright (c) <2010-2017>, Intel Corporation. All rights reserved.
Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
>>> Packet Burst 64, RX Desc 1024, TX Desc 2048, mbufs/port 16384, mbuf cache 2048
!PANIC!: *** Did not find any ports to use ***
PANIC in pktgen_config_ports():
*** Did not find any ports to use ***6: [app/x86_64-native-linuxapp-gcc/pktgen(_start+0x2a) [0x5592bd24d66a]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f5b8de20b97]]
4: [app/x86_64-native-linuxapp-gcc/pktgen(main+0xe67) [0x5592bd24a1e7]]
3: [app/x86_64-native-linuxapp-gcc/pktgen(pktgen_config_ports+0x170b) [0x5592bd277a4b]]
2: [app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xc5) [0x5592bd244bb7]]
1: [app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x2e) [0x5592bd2dcb3e]]
Aborted (core dumped)
I use IGB_UIO not VFIO.
I don't set --socket-mem, and lcore1 and lcore2 are both ready.
--vdev fails to initialize. I have checked /var/run/openvswitch, vhost-user0 and vhost-user1 exists.
Thank you for sharing your idea. Best wishes.

Answer 1

I think /var/run/openswitch should be /var/run/openvswitch.
sudo docker run -ti --privileged --name=pktgen-docker -v /mnt/huge:/mnt/huge -v /usr/local/var/run/openvswitch:/var/run/openswitch pktgen-docker:latest

Answer 2

OVS creates the virtio-user sock in the directory as it is running in server mode. The pktgen is also started with virtio-user mode. This conflicts as both are running in server mode.
The correct way is to start pktgen in client mode using virtio_userx. If the goal is to have packet transfer between Docker-1 Pktgen and Docker-2 that is connected by OVS-DPDK one has to use net_vhost and virtio_user pair.
Docker-1 Pktgen (net_vhost) <==> OVS-DPDK port-1 (virtio_user) {Rule to forward} OVS-DPDK port-2 (virtio_user) <==> Docker-2 Pktgen (net_vhost)
In the current setup, you will have to make the following changes
start DPDK pktgen in Docker-1 by changing from --vdev 'net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0' to --vdev 'net_vhost0,iface=/var/run/openvswitch/vhost-user0'
start DPDK testpmd by changing from --vdev
start DPDK pktgen in Docker-2 by changing from --vdev 'net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0' to --vdev 'net_vhost1,iface=/var/run/openvswitch/vhost-user1'
then start DPDK-OVS with --vdev=virtio_user0,path=/var/run/openvswitch/vhost-user0 and --vdev=virtio_user1,path=/var/run/openvswitch/vhost-user1
add rules to allow the port to port forwarding between Docker-1 pktgen and Docker-2 pktgen
Note:
please update the command line for multiple ports.
explained in comments. This is more of a usage problem