Hi Kunpeng, You can check the bridge and openflows by the following commands: ovs-vsctl show ovs-ofctl dump-flows br-int ovs-ofctl dump-flows br-phy0 The virtual network used by VMs is based on those openflows. And restarting ovs-vswitchd can’t reinstall the openflows. That’s why when the ovs-vswitchd restart, you will lose the connections to VMs. I think we need to figure out why ovs-vswitchd is restarted when you restart the VM. Could you please check the below logs to see why ovs-vswitchd is restarted? /var/log/openvswitch/ovs-vswitchd.log /var/log/syslog neutron log (the log file is specified in /etc/neutron/neutron.conf) Best Regards, Xu, Chenjie From: 张鲲鹏 [mailto:zhang.kunpeng@99cloud.net] Sent: Thursday, July 18, 2019 7:09 PM To: Xu, Chenjie <chenjie.xu@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart I also find that when the ovs-vswitchd is restarted, I will lose the connections to VMs. Before restart ovs: controller-0:/home/wrsroot# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 127.168.204.3/24 brd 127.168.204.255 scope host lo valid_lft forever preferred_lft forever inet 169.254.202.2/24 scope global lo valid_lft forever preferred_lft forever inet 127.168.204.2/24 scope host secondary lo valid_lft forever preferred_lft forever inet 127.168.204.5/24 scope host secondary lo valid_lft forever preferred_lft forever inet 127.168.204.6/24 scope host secondary lo valid_lft forever preferred_lft forever inet 127.168.204.152/24 scope host secondary lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: enp59s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether f8:f2:1e:64:01:21 brd ff:ff:ff:ff:ff:ff 4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 4c:d9:8f:72:fb:63 brd ff:ff:ff:ff:ff:ff inet 172.16.180.154/24 brd 172.16.180.255 scope global eno1 valid_lft forever preferred_lft forever inet6 fe80::4ed9:8fff:fe72:fb63/64 scope link valid_lft forever preferred_lft forever 5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 4c:d9:8f:72:fb:64 brd ff:ff:ff:ff:ff:ff 6: enp94s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 4c:d9:8f:5e:ab:88 brd ff:ff:ff:ff:ff:ff 7: enp94s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 4c:d9:8f:5e:ab:89 brd ff:ff:ff:ff:ff:ff 8: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether f8:f2:1e:64:80:60 brd ff:ff:ff:ff:ff:ff inet6 fe80::faf2:1eff:fe64:8060/64 scope link valid_lft forever preferred_lft forever 9: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether f8:f2:1e:64:80:61 brd ff:ff:ff:ff:ff:ff inet6 fe80::faf2:1eff:fe64:8061/64 scope link valid_lft forever preferred_lft forever 10: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 62:34:73:47:50:be brd ff:ff:ff:ff:ff:ff 13: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether c6:91:5d:27:b4:43 brd ff:ff:ff:ff:ff:ff 16: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000 link/ether f8:f2:1e:64:01:20 brd ff:ff:ff:ff:ff:ff inet6 fe80::faf2:1eff:fe64:120/64 scope link valid_lft forever preferred_lft forever 17: lldp16ba3755-27: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000 link/ether 7e:3a:87:a3:98:03 brd ff:ff:ff:ff:ff:ff inet6 fe80::7c3a:87ff:fea3:9803/64 scope link valid_lft forever preferred_lft forever After: controller-0:/home/wrsroot# systemctl restart ovs-vswitchd controller-0:/home/wrsroot# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 127.168.204.3/24 brd 127.168.204.255 scope host lo valid_lft forever preferred_lft forever inet 169.254.202.2/24 scope global lo valid_lft forever preferred_lft forever inet 127.168.204.2/24 scope host secondary lo valid_lft forever preferred_lft forever inet 127.168.204.5/24 scope host secondary lo valid_lft forever preferred_lft forever inet 127.168.204.6/24 scope host secondary lo valid_lft forever preferred_lft forever inet 127.168.204.152/24 scope host secondary lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: enp59s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether f8:f2:1e:64:01:21 brd ff:ff:ff:ff:ff:ff 4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 4c:d9:8f:72:fb:63 brd ff:ff:ff:ff:ff:ff inet 172.16.180.154/24 brd 172.16.180.255 scope global eno1 valid_lft forever preferred_lft forever inet6 fe80::4ed9:8fff:fe72:fb63/64 scope link valid_lft forever preferred_lft forever 5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 4c:d9:8f:72:fb:64 brd ff:ff:ff:ff:ff:ff 6: enp94s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 4c:d9:8f:5e:ab:88 brd ff:ff:ff:ff:ff:ff 7: enp94s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 4c:d9:8f:5e:ab:89 brd ff:ff:ff:ff:ff:ff 8: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether f8:f2:1e:64:80:60 brd ff:ff:ff:ff:ff:ff inet6 fe80::faf2:1eff:fe64:8060/64 scope link valid_lft forever preferred_lft forever 9: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether f8:f2:1e:64:80:61 brd ff:ff:ff:ff:ff:ff inet6 fe80::faf2:1eff:fe64:8061/64 scope link valid_lft forever preferred_lft forever 10: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 62:34:73:47:50:be brd ff:ff:ff:ff:ff:ff 13: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether c6:91:5d:27:b4:43 brd ff:ff:ff:ff:ff:ff 16: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether f8:f2:1e:64:01:20 brd ff:ff:ff:ff:ff:ff inet6 fe80::faf2:1eff:fe64:120/64 scope link valid_lft forever preferred_lft forever 17: lldp16ba3755-27: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 7e:3a:87:a3:98:03 brd ff:ff:ff:ff:ff:ff inet6 fe80::7c3a:87ff:fea3:9803/64 scope link valid_lft forever preferred_lft forever 20: tapfb74713e-cc: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0e:2d:36:ee:2d:85 brd ff:ff:ff:ff:ff:ff 21: tap1a965902-0b: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 5e:f6:81:31:54:21 brd ff:ff:ff:ff:ff:ff One of the VMs: [cid:image001.png@01D53E10.A2B5CD00] On Jul 18, 2019, at 18:09, 张鲲鹏 <zhang.kunpeng@99cloud.net<mailto:zhang.kunpeng@99cloud.net>> wrote: Hi Xu,Chenjie I have tried to create VM with 2 pci_passthrough network ports without DPDK, there was the same problem when I rebooted it. Also, it was same when I reboot the VM with 2 SR-IOV VFs. Do you have any ideas to debug this problem? Thanks Kunpeng On Jul 17, 2019, at 14:59, Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> wrote: Hi Kunpeng, Maybe you can use SR-IOV and passthrough the VF which has similar performance to physical NIC to the VM. And then you can use DPDK inside the VM with the VF. Sorry, I don’t have easy way to disable DPDK in stx1.0. The following command is used for stx2.0 which is still in progress: system modify --vswitch_type none Best Regards, Xu, Chenjie From: 张鲲鹏 [mailto:zhang.kunpeng@99cloud.net] Sent: Tuesday, July 16, 2019 5:40 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart Hi Chenjie, Well, I will try the network topology as you said. But passthrough NIC with DPDK is our customer’s requirement. And do you have some easy ways to disable dpdk of openvswitch in stx1.0? I had tried to execute “system modify --vswitch_type none” before “system host-unlock controller-0", but it doesn’t work well. Thanks Kunpeng On Jul 16, 2019, at 16:49, Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> wrote: Hi Kunpeng, When you reboot the VM with two physical pci-passthrough NICs, ovs-vswtichd is restarted and the interfaces and bridges are down. The virtual networks used by the VMs are based on these interfaces and bridges. So other VMs will lost connections. Typically you should not pass through a PCI device which is bound to DPDK to the VM. Are the 2 network ports which is used to passthrough to the VM bound to DPDK? If so, could you please try OVS-DPDK with the following network topology: 2 network port without DPDK > VM 2 network port with DPDK > Data Network 1 network port without DPDK > OAM Best Regards, Xu, Chenjie From: 张鲲鹏 [mailto:zhang.kunpeng@99cloud.net] Sent: Tuesday, July 16, 2019 3:54 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart Hi guys, Recently I got a strange problem in StarlingX. When I reboot one VM with two physical pci-passthrough NICs, then all of VMs cannot be connected. I lost connections with all VMs and also the VMs lost each others. Below is the StarlingX environment. 1. stx1.0 version, bootimage[1] 2. Simplex deployment 3. 5 Network ports. Only one don’t support DPDK，and it is used to OAM Network. In the rest, two are used to data network, and another two are used to passthrough to a VM. 4. The VM was attached two more virtual networks. I have tested the case of attaching one virtual net, it was no problem. When I reboot the VM, something were happened. The interfaces and bridges were down, all the virtual dhcp services were down and ovs-vswitchd was restarted. But when I up the interfaces and dhcp services and reboot the other VMs, I have got the connections with them again. It’s ok when to reboot the VM without physical NIC. We think it may be caused by ovs-dpdk, so we stop to use ovs-dpdk and start the ovs manually, the problem was gone. I cannot understand the problem, anybody could give me some comments for it? Thanks a lot. [1] http://mirror.starlingx.cengn.ca/mirror/starlingx/r2018.10/centos/2018.10.0/... Kunpeng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
