I also find that when the ovs-vswitchd is restarted, I will lose the connections to VMs.

Before restart ovs:

controller-0:/home/wrsroot# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.3/24 brd 127.168.204.255 scope host lo
       valid_lft forever preferred_lft forever
    inet 169.254.202.2/24 scope global lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.2/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.5/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.6/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.152/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: enp59s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether f8:f2:1e:64:01:21 brd ff:ff:ff:ff:ff:ff
4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:d9:8f:72:fb:63 brd ff:ff:ff:ff:ff:ff
    inet 172.16.180.154/24 brd 172.16.180.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::4ed9:8fff:fe72:fb63/64 scope link
       valid_lft forever preferred_lft forever
5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:72:fb:64 brd ff:ff:ff:ff:ff:ff
6: enp94s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:5e:ab:88 brd ff:ff:ff:ff:ff:ff
7: enp94s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:5e:ab:89 brd ff:ff:ff:ff:ff:ff
8: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:64:80:60 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:8060/64 scope link
       valid_lft forever preferred_lft forever
9: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:64:80:61 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:8061/64 scope link
       valid_lft forever preferred_lft forever
10: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:34:73:47:50:be brd ff:ff:ff:ff:ff:ff
13: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c6:91:5d:27:b4:43 brd ff:ff:ff:ff:ff:ff
16: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether f8:f2:1e:64:01:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:120/64 scope link
       valid_lft forever preferred_lft forever
17: lldp16ba3755-27: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 7e:3a:87:a3:98:03 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7c3a:87ff:fea3:9803/64 scope link
       valid_lft forever preferred_lft forever

After:

controller-0:/home/wrsroot# systemctl restart ovs-vswitchd
controller-0:/home/wrsroot# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.3/24 brd 127.168.204.255 scope host lo
       valid_lft forever preferred_lft forever
    inet 169.254.202.2/24 scope global lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.2/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.5/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.6/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.152/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: enp59s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether f8:f2:1e:64:01:21 brd ff:ff:ff:ff:ff:ff
4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:d9:8f:72:fb:63 brd ff:ff:ff:ff:ff:ff
    inet 172.16.180.154/24 brd 172.16.180.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::4ed9:8fff:fe72:fb63/64 scope link
       valid_lft forever preferred_lft forever
5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:72:fb:64 brd ff:ff:ff:ff:ff:ff
6: enp94s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:5e:ab:88 brd ff:ff:ff:ff:ff:ff
7: enp94s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:d9:8f:5e:ab:89 brd ff:ff:ff:ff:ff:ff
8: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:64:80:60 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:8060/64 scope link
       valid_lft forever preferred_lft forever
9: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:64:80:61 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:8061/64 scope link
       valid_lft forever preferred_lft forever
10: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:34:73:47:50:be brd ff:ff:ff:ff:ff:ff
13: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c6:91:5d:27:b4:43 brd ff:ff:ff:ff:ff:ff
16: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether f8:f2:1e:64:01:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:fe64:120/64 scope link
       valid_lft forever preferred_lft forever
17: lldp16ba3755-27: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 7e:3a:87:a3:98:03 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7c3a:87ff:fea3:9803/64 scope link
       valid_lft forever preferred_lft forever
20: tapfb74713e-cc: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0e:2d:36:ee:2d:85 brd ff:ff:ff:ff:ff:ff
21: tap1a965902-0b: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5e:f6:81:31:54:21 brd ff:ff:ff:ff:ff:ff

One of the VMs:


On Jul 18, 2019, at 18:09, 张鲲鹏 <zhang.kunpeng@99cloud.net> wrote:

Hi Xu,Chenjie

I have tried to create VM with 2 pci_passthrough network ports without DPDK, there was the same problem when I rebooted it.
Also, it was same when I reboot the VM with 2 SR-IOV VFs.
Do you have any ideas to debug this problem?

Thanks 
Kunpeng  

On Jul 17, 2019, at 14:59, Xu, Chenjie <chenjie.xu@intel.com> wrote:

Hi Kunpeng,
Maybe you can use SR-IOV and passthrough the VF which has similar performance to physical NIC to the VM. And then you can use DPDK inside the VM with the VF.
 
Sorry, I don’t have easy way to disable DPDK in stx1.0. The following command is used for stx2.0 which is still in progress:
system modify --vswitch_type none
 
Best Regards,
Xu, Chenjie
From: 张鲲鹏 [mailto:zhang.kunpeng@99cloud.net] 
Sent: Tuesday, July 16, 2019 5:40 PM
To: Xu, Chenjie <chenjie.xu@intel.com>
Cc: starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart
 
Hi Chenjie,
 
Well, I will try the network topology as you said. But passthrough NIC with DPDK is our customer’s requirement.
And do you have some easy ways to disable dpdk of openvswitch in stx1.0? 
I had tried to execute “system modify --vswitch_type none” before “system host-unlock controller-0", but it doesn’t work well.
 
Thanks 
Kunpeng
 
On Jul 16, 2019, at 16:49, Xu, Chenjie <chenjie.xu@intel.com> wrote:
 
Hi Kunpeng,
When you reboot the VM with two physical pci-passthrough NICs, ovs-vswtichd is restarted and the interfaces and bridges are down. The virtual networks used by the VMs are based on these interfaces and bridges. So other VMs will lost connections.
 
Typically you should not pass through a PCI device which is bound to DPDK to the VM. Are the 2 network ports which is used to passthrough to the VM bound to DPDK? If so, could you please try OVS-DPDK with the following network topology:
2 network port without DPDK      >      VM
2 network port with DPDK             >      Data Network
1 network port without DPDK       >      OAM
 
Best Regards,
Xu, Chenjie
 
From: 张鲲鹏 [mailto:zhang.kunpeng@99cloud.net] 
Sent: Tuesday, July 16, 2019 3:54 PM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] [Starlingx-disscuss][ovs-dpdk] Reboot instance with physical pci-passthrough NIC causes OVS restart
 
Hi guys,
 
Recently I got a strange problem in StarlingX. When I reboot one VM with two physical pci-passthrough NICs, then all of VMs cannot be connected. I lost connections with all VMs and also the VMs lost each others. 
 
Below is the StarlingX environment.
 
1. stx1.0 version, bootimage[1] 
2. Simplex deployment
3. 5 Network ports. Only one dont support DPDKand it is used to OAM Network. In the rest, two are used to data network, and another two are used to passthrough to a VM.
4. The VM was attached two more virtual networks. I have tested the case of attaching one virtual net, it was no problem. 
 
When I reboot the VM, something were happened. The interfaces and bridges were down, all the virtual dhcp services were down and ovs-vswitchd was restarted. But when I up the interfaces and dhcp services and reboot the other VMs, I have got the connections with them again.
 
Its ok when to reboot the VM without physical NIC. We think it may be caused by ovs-dpdk, so we stop to use ovs-dpdk and start the ovs manually, the problem was gone.
 
I cannot understand the problem, anybody could give me some comments for it? Thanks a lot.
 
 
Kunpeng
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
 
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss