[Starlingx-discuss] OVS-DPDK Upgrade Testing

Xu, Chenjie chenjie.xu at intel.com
Mon Mar 18 14:05:15 UTC 2019 

Hi Bruce/Ghada/Matt,
I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file.

By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts).

I have reported the bug in launchpad and you can find my debugging steps in the following bug report:
https://bugs.launchpad.net/starlingx/+bug/1820378

Best Regards,
Xu, Chenjie

From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Monday, March 18, 2019 8:40 PM
To: Xu, Chenjie <chenjie.xu at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; Xie, Cindy <cindy.xie at intel.com>; Zhao, Forrest <forrest.zhao at intel.com>; Richard, Joseph <Joseph.Richard at windriver.com>; Winnicki, Chris <Chris.Winnicki at windriver.com>; Lin, Shuicheng <shuicheng.lin at intel.com>; Peng, Peng <Peng.Peng at windriver.com>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Qin, Kailun <kailun.qin at intel.com>; Guo, Ruijing <ruijing.guo at intel.com>; Le, Huifeng <huifeng.le at intel.com>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Chenjie,
Sorry for the delayed reply.

For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath?  Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all?  If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP.

You can dump the datapath ports and flows with the following command:


ovs-appctl dpif/show

ovs-appctl dpif/dump-flows br-int


Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present?

Regards, Matt

From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Friday, March 15, 2019 at 2:48 PM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>, "Xie, Cindy" <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>, "Richard, Joseph" <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>, "Winnicki, Chris" <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Peng, Peng" <Peng.Peng at windriver.com<mailto:Peng.Peng at windriver.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Jones, Bruce E" <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>, "Qin, Kailun" <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>, "Guo, Ruijing" <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>, "Le, Huifeng" <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Matt,
We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment?

1.     We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work.
Bring up br-int to verify the connectivity:     sudo ifconfig br-int 192.168.101.1/24 up
                  Check the IP assigned to the VM:                   openstack server list
                  Manually assign IP to VM:                                sudo ifconfig 192.168.101.216/24 up
                  Tcpdump in dhcp namespace:                                  sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device
                  Ping DHCP server in VM:                                            ping 192.168.101.1
                  No packets can be captured.

2.     We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM:
Create vhostuser port:                                      ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1
type=dpdkvhostuserclient
options:vhost-server-path=/var/run/openvswitch/vhost-user-1
Create VM:                                                          virsh create cirros-dpdk-vhostuserclient.xml
Bring up br-int to verify the connectivity:     sudo ifconfig br-int 192.168.101.1/24 up
Manually assign an IP to VM:                                   sudo ifconfig eth0 192.168.101.18/24 up
In created VM, ping br-int:                               ping 192.168.101.1
And the ping succeeds.

Best Regards,
Xu, Chenjie
From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Wednesday, March 13, 2019 9:13 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Winnicki, Chris <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Peng, Peng <Peng.Peng at windriver.com<mailto:Peng.Peng at windriver.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hello,
That message is a normal message from DPDK if you have multiple huge page sizes configured in your system.  Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages.

You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting.

-Matt

From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Wednesday, March 13, 2019 at 8:55 AM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>, "Xie, Cindy" <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>, "Richard, Joseph" <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>, "Winnicki, Chris" <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Peng, Peng" <Peng.Peng at windriver.com<mailto:Peng.Peng at windriver.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Jones, Bruce E" <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>, "Qin, Kailun" <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>, "Guo, Ruijing" <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>, "Le, Huifeng" <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi team,
I installed StarlingX using following the March 5th ISO image and the issue still exists：http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060000Z/outputs/iso/
Are there any steps not been written on the guide? I just followed the following guide:
https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard

I checked the ovs-vswitchd.log and the following line maybe the cause:
23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know.

Peng has been added and hope Peng can provide some information about his environment.
https://bugs.launchpad.net/starlingx/+bug/1819031


Best Regards,
Xu, Chenjie

From: Xu, Chenjie
Sent: Tuesday, March 12, 2019 12:39 AM
To: Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Winnicki, Chris <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi team,
For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0.

For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team.
https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests

Best Regards,
Xu, Chenjie

From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Tuesday, March 12, 2019 12:28 AM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Winnicki, Chris <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected?  I.e. is it only when you are traversing the physical network that things are not pingable?

-Matt

From: Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Date: Monday, March 11, 2019 at 10:51 AM
To: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>, "Xie, Cindy" <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>, "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, "Richard, Joseph" <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>, "Winnicki, Chris" <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>
Cc: "starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Jones, Bruce E" <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>, "Qin, Kailun" <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>, "Guo, Ruijing" <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>, "Le, Huifeng" <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

To answer Forrest’s question:
               Do you know if OVS/DPDK has been tested after cut-over to container?
I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong.

Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng.

Regards,
Ghada


From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Monday, March 11, 2019 10:40 AM
To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Cindy,
Thank you for your information! The following issue is not exactly the same as mine:
https://bugs.launchpad.net/starlingx/+bug/1819031

In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself.

Best Regards,
Xu, Chenjie

From: Xie, Cindy
Sent: Monday, March 11, 2019 10:05 PM
To: Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Is the below issue similar?

https://bugs.launchpad.net/starlingx/+bug/1819031


From: Zhao, Forrest
Sent: Monday, March 11, 2019 9:54 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Ghada, Matt, Joseph and validation team,

Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container.

Thanks,
Forrest

From: Xu, Chenjie
Sent: Monday, March 11, 2019 9:17 PM
To: Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi team,
I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached.

This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas.

Best Regards,
Xu, Chenjie

From: Lin, Shuicheng
Sent: Friday, March 8, 2019 9:49 AM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi all,
I just shared the new test ISO with Chenjie/Elio in another mail thread.
It is built from centos76 feature branch with latest master code rebased.

Best Regards
Shuicheng

From: Xu, Chenjie
Sent: Wednesday, March 6, 2019 11:07 AM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Ghada,
That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide.

Best Regards,
Xu, Chenjie

From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
Sent: Wednesday, March 6, 2019 12:07 AM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Winnicki, Chris <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Chenjie,
I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos.
Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior.

I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow.

Regards,
Ghada

From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Tuesday, March 05, 2019 8:54 AM
To: Richard, Joseph; Khalil, Ghada; Peters, Matt
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Joseph,
About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini:
extensions = qos
Could you please tell which puppet file I should modify?

Best Regards,
Xu, Chenjie

From: Xu, Chenjie
Sent: Monday, March 4, 2019 11:39 PM
To: 'Richard, Joseph' <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; 'Ghada.Khalil at windriver.com' <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; 'Peters, Matt' <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Joseph,
For the following bug sent on last Friday:
system host-port-list compute-1
'pdevice'
I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments?

Best Regards,
Xu, Chenjie

From: Xu, Chenjie
Sent: Monday, March 4, 2019 10:08 PM
To: Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Joseph,
Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948)

Best Regards,
Xu, Chenjie

From: Richard, Joseph [mailto:Joseph.Richard at windriver.com]
Sent: Saturday, March 2, 2019 12:41 AM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

1:
You need to use sysinv to add the address, not add it using ip commands.
usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6
                            address> <prefix length>

2:
Check in /var/log/sysinv.log for the traceback at that time.

From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Friday, March 1, 2019 9:55 AM
To: Richard, Joseph; Khalil, Ghada; Peters, Matt
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Joseph,
I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you:

1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following:
     neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400
neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255  tenant_vxlan
system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789
system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1
system host-unlock compute-0
After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images.

2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command:
system host-port-list compute-1
'pdevice'
The normal output should be:
system host-port-list compute-0
+--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+
| uuid                                 | name     | type     | pci address  | device | processor | accelerated | device type                                    |
+--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+
| e072d576-90a6-4040-87df-66a913076df0 | eno1     | ethernet | 0000:07:00.0 | 0      | 0         | True        | I350 Gigabit Network Connection                |
| 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2     | ethernet | 0000:07:00.1 | 0      | 0         | True        | I350 Gigabit Network Connection                |
| 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0      | 0         | True        | 82599ES 10-Gigabit SFI/SFP+   Network Connection |
| 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0      | 0         | True        | 82599ES 10-Gigabit SFI/SFP+ Network Connection |
+--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+

Best Regards,
Xu, Chenjie

From: Xu, Chenjie
Sent: Friday, March 1, 2019 2:32 PM
To: Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>
Cc: Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: ISO image for OVS/DPDK upgrade

Hi Shuicheng,
Thank you for your information! I will start to do the Basic Functional Test.

Best Regards,
Xu, Chenjie


From: Lin, Shuicheng
Sent: Friday, March 1, 2019 2:03 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Cc: Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>
Subject: ISO image for OVS/DPDK upgrade

Hi Chenjie/Elio,
Here is the ISO image for OVS/DPDK upgrade test.
I have done basic deploy test, and could ping VM from each other successfully.
Please help have a try with it, and notify me if there is any issue.
It is built with recent code of centos76 feature branch.

ec at 10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec at 10.239.154.150:/home/ec/bootimage_ovs.iso>
scp passwd: 1

Here is the patch list for the upgrade in case you want to have a check:
https://review.openstack.org/640244
https://review.openstack.org/640245

Best Regards
Shuicheng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190318/27d44fd0/attachment-0001.html>

More information about the Starlingx-discuss mailing list