[Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart

Fri Jul 19 01:29:59 UTC 2019

Hi Kunpeng,
You can check the bridge and openflows by the following commands:
ovs-vsctl show
ovs-ofctl dump-flows br-int
ovs-ofctl dump-flows br-phy0

The virtual network used by VMs is based on those openflows. And restarting ovs-vswitchd can’t reinstall the openflows. That’s why when the ovs-vswitchd restart, you will lose the connections to VMs.

I think we need to figure out why ovs-vswitchd is restarted when you restart the VM. Could you please check the below logs to see why ovs-vswitchd is restarted?
       /var/log/openvswitch/ovs-vswitchd.log
/var/log/syslog
neutron log (the log file is specified in /etc/neutron/neutron.conf)

Best Regards,
Xu, Chenjie

From: 张鲲鹏 [mailto:zhang.kunpeng at 99cloud.net]
Sent: Thursday, July 18, 2019 7:09 PM
To: Xu, Chenjie <chenjie.xu at intel.com>
Cc: starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart

I also find that when the ovs-vswitchd is restarted, I will lose the connections to VMs.

Before restart ovs:

controller-0:/home/wrsroot# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.3/24 brd 127.168.204.255 scope host lo
       valid_lft forever preferred_lft forever
    inet 169.254.202.2/24 scope global lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.2/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.5/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.6/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.152/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: enp59s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether f8:f2:1e:64:01:21 brd ff:ff:ff:ff:ff:ff
4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:d9:8f:72:fb:63 brd ff:ff:ff:ff:ff:ff
    inet 172.16.180.154/24 brd 172.16.180.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::4ed9:8fff:fe72:fb63/64 scope link
       valid_lft forever preferred_lft forever
5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:72:fb:64 brd ff:ff:ff:ff:ff:ff
6: enp94s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:5e:ab:88 brd ff:ff:ff:ff:ff:ff
7: enp94s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:5e:ab:89 brd ff:ff:ff:ff:ff:ff
8: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:64:80:60 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:8060/64 scope link
       valid_lft forever preferred_lft forever
9: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:64:80:61 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:8061/64 scope link
       valid_lft forever preferred_lft forever
10: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:34:73:47:50:be brd ff:ff:ff:ff:ff:ff
13: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c6:91:5d:27:b4:43 brd ff:ff:ff:ff:ff:ff
16: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether f8:f2:1e:64:01:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:120/64 scope link
       valid_lft forever preferred_lft forever
17: lldp16ba3755-27: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 7e:3a:87:a3:98:03 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7c3a:87ff:fea3:9803/64 scope link
       valid_lft forever preferred_lft forever

After:

controller-0:/home/wrsroot# systemctl restart ovs-vswitchd
controller-0:/home/wrsroot# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.3/24 brd 127.168.204.255 scope host lo
       valid_lft forever preferred_lft forever
    inet 169.254.202.2/24 scope global lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.2/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.5/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.6/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.152/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: enp59s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether f8:f2:1e:64:01:21 brd ff:ff:ff:ff:ff:ff
4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:d9:8f:72:fb:63 brd ff:ff:ff:ff:ff:ff
    inet 172.16.180.154/24 brd 172.16.180.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::4ed9:8fff:fe72:fb63/64 scope link
       valid_lft forever preferred_lft forever
5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:72:fb:64 brd ff:ff:ff:ff:ff:ff
6: enp94s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:5e:ab:88 brd ff:ff:ff:ff:ff:ff
7: enp94s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:5e:ab:89 brd ff:ff:ff:ff:ff:ff
8: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:64:80:60 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:8060/64 scope link
       valid_lft forever preferred_lft forever
9: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:64:80:61 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:8061/64 scope link
       valid_lft forever preferred_lft forever
10: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:34:73:47:50:be brd ff:ff:ff:ff:ff:ff
13: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c6:91:5d:27:b4:43 brd ff:ff:ff:ff:ff:ff
16: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether f8:f2:1e:64:01:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:120/64 scope link
       valid_lft forever preferred_lft forever
17: lldp16ba3755-27: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 7e:3a:87:a3:98:03 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7c3a:87ff:fea3:9803/64 scope link
       valid_lft forever preferred_lft forever
20: tapfb74713e-cc: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0e:2d:36:ee:2d:85 brd ff:ff:ff:ff:ff:ff
21: tap1a965902-0b: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5e:f6:81:31:54:21 brd ff:ff:ff:ff:ff:ff

One of the VMs:

[cid:image001.png at 01D53E10.A2B5CD00]

On Jul 18, 2019, at 18:09, 张鲲鹏 <zhang.kunpeng at 99cloud.net<mailto:zhang.kunpeng at 99cloud.net>> wrote:

Hi Xu,Chenjie

I have tried to create VM with 2 pci_passthrough network ports without DPDK, there was the same problem when I rebooted it.
Also, it was same when I reboot the VM with 2 SR-IOV VFs.
Do you have any ideas to debug this problem?

Thanks
Kunpeng

On Jul 17, 2019, at 14:59, Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>> wrote:

Hi Kunpeng,
Maybe you can use SR-IOV and passthrough the VF which has similar performance to physical NIC to the VM. And then you can use DPDK inside the VM with the VF.

Sorry, I don’t have easy way to disable DPDK in stx1.0. The following command is used for stx2.0 which is still in progress:
system modify --vswitch_type none

Best Regards,
Xu, Chenjie
From: 张鲲鹏 [mailto:zhang.kunpeng at 99cloud.net]
Sent: Tuesday, July 16, 2019 5:40 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart

Hi Chenjie,

Well, I will try the network topology as you said. But passthrough NIC with DPDK is our customer’s requirement.
And do you have some easy ways to disable dpdk of openvswitch in stx1.0?
I had tried to execute “system modify --vswitch_type none” before “system host-unlock controller-0", but it doesn’t work well.

Thanks
Kunpeng

On Jul 16, 2019, at 16:49, Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>> wrote:

Hi Kunpeng,
When you reboot the VM with two physical pci-passthrough NICs, ovs-vswtichd is restarted and the interfaces and bridges are down. The virtual networks used by the VMs are based on these interfaces and bridges. So other VMs will lost connections.

Typically you should not pass through a PCI device which is bound to DPDK to the VM. Are the 2 network ports which is used to passthrough to the VM bound to DPDK? If so, could you please try OVS-DPDK with the following network topology:
2 network port without DPDK      >      VM
2 network port with DPDK             >      Data Network
1 network port without DPDK       >      OAM

Best Regards,
Xu, Chenjie

From: 张鲲鹏 [mailto:zhang.kunpeng at 99cloud.net]
Sent: Tuesday, July 16, 2019 3:54 PM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart

Hi guys,

Recently I got a strange problem in StarlingX. When I reboot one VM with two physical pci-passthrough NICs, then all of VMs cannot be connected. I lost connections with all VMs and also the VMs lost each others.

Below is the StarlingX environment.

1. stx1.0 version, bootimage[1]
2. Simplex deployment
3. 5 Network ports. Only one don’t support DPDK，and it is used to OAM Network. In the rest, two are used to data network, and another two are used to passthrough to a VM.
4. The VM was attached two more virtual networks. I have tested the case of attaching one virtual net, it was no problem.

When I reboot the VM, something were happened. The interfaces and bridges were down, all the virtual dhcp services were down and ovs-vswitchd was restarted. But when I up the interfaces and dhcp services and reboot the other VMs, I have got the connections with them again.

It’s ok when to reboot the VM without physical NIC. We think it may be caused by ovs-dpdk, so we stop to use ovs-dpdk and start the ovs manually, the problem was gone.

I cannot understand the problem, anybody could give me some comments for it? Thanks a lot.

[1] http://mirror.starlingx.cengn.ca/mirror/starlingx/r2018.10/centos/2018.10.0/outputs/iso/bootimage.iso

Kunpeng
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190719/5a5f27a6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 103649 bytes
Desc: image001.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190719/5a5f27a6/attachment-0001.png>